PaperlayerView exact revision on arXiv

Source-linked AI summary

RSL-SQL: Robust Schema Linking in Text-to-SQL Generation

Zhenbiao Cao, Yuanlei Zheng, Zhihao Fan, Xiaojin Zhang, Wei Chen, Xiang Bai

arXiv:2411.00073v2cs.CLcs.AIcs.DB

TL;DR

Schema linking can streamline Text-to-SQL prompts, but may omit necessary schema elements or relationships and introduce ambiguity. RSL-SQL combines bidirectional linking with contextual augmentation, binary selection, and self-correction. With GPT-4o, it reports open-source state-of-the-art execution accuracy on BIRD and comparable-to-state-of-the-art accuracy on Spider.

  • Problem

    Schema linking reduces irrelevant schema input in LLM prompts, but can omit necessary elements or relationships and introduce ambiguity that risks SQL accuracy.

    Source: §I · p. 1

  • Method

    RSL-SQL combines bidirectional schema linking, contextual information augmentation, binary selection between SQL candidates, and multi-turn self-correction using execution feedback.

    Source: §I · p. 2

  • Results

    With GPT-4o, RSL-SQL reports 67.21% execution accuracy on the BIRD dev dataset, a new open-source state-of-the-art, and 87.9% on Spider test, comparable to state-of-the-art.

    Source: §I · p. 2

  • Takeaways & Limitations

    With DeepSeek, RSL-SQL outperforms many GPT-4-based methods on BIRD and Spider despite GPT-4's per-token cost being 215 times higher.

    Source: §I · p. 2

Abstract

from arXiv · show

Text-to-SQL generation aims to translate natural language questions into SQL statements. In Text-to-SQL based on large language models, schema linking is a widely adopted strategy to streamline the input for LLMs by selecting only relevant schema elements, therefore reducing noise and computational overhead. However, schema linking faces risks that require caution, including the potential omission of necessary elements and disruption of database structural integrity. To address these challenges, we propose a novel framework called RSL-SQL that combines bidirectional schema linking, contextual information augmentation, binary selection strategy, and multi-turn self-correction. We improve the recall of pattern linking using forward and backward pruning methods, achieving a strict recall of 94% while reducing the number of input columns by 83%. Furthermore, it hedges the risk by voting between a full mode and a simplified mode enhanced with contextual information. Experiments on the BIRD and Spider benchmarks demonstrate that our approach achieves SOTA execution accuracy among open-source solutions, with 67.2% on BIRD and 87.9% on Spider using GPT-4o. Furthermore, our approach outperforms a series of GPT-4 based Text-to-SQL systems when adopting DeepSeek (much cheaper) with same intact prompts. Extensive analysis and ablation studies confirm the effectiveness of each component in our framework. The codes are available at https://github.com/Laqcce-cao/RSL-SQL.

I. INTRODUCTION

Schema linking can reduce irrelevant prompt content, but may also omit necessary schema elements or context and disrupt accurate SQL generation. RSL-SQL combines schema-linking and SQL-selection strategies to address these risks, and reports strong results on BIRD and Spider.

I. INTRODUCTION

  • Text-to-SQL translates natural-language questions into SQL, enabling non-experts to query relational databases without deep technical knowledge.The introduction frames this as a way to democratize data access and extract insights.Source: §I · p. 1
  • Schema linking reduces irrelevant schema input, but can omit necessary tables, columns, or relationships and make names more ambiguous, risking erroneous or degraded SQL generation.Its net benefit depends on corrected queries outweighing initially correct queries made incorrect (y −x > 0).Source: §I · pp. 1–2
  • RSL-SQL aims to maximize schema linking’s positive gains and minimize its negative impact through bidirectional linking, contextual augmentation, binary selection, and multi-turn self-correction.It first generates SQL from the complete schema, then uses a context-enriched simplified schema, selects between the candidates, and refines SQL using execution feedback.Source: §I · p. 2
  • 67.21% execution accuracy on the BIRD dev dataset set a new state-of-the-art for open-source methods, while 87.9% on the Spider test dataset was comparable to state-of-the-art.With GPT-4o, RSL-SQL also outperformed many GPT-4-based methods on both datasets when using DeepSeek, whose per-token cost is 215 times lower.Source: §I · p. 2

II. RELATED WORK

  • Earlier Text-to-SQL work progressed from hand-designed templates to sequence-to-sequence models and LLM methods using decomposition, reasoning, or multiple SQL candidates.Unlike methods generating candidates through multiple prompts, RSL-SQL generates two SQL candidates from complete and simplified schemas.Source: §II · p. 2
  • Prior schema-linking approaches include relationship-aware models, graph methods, database retrieval, and multi-prompt sampling; RSL-SQL instead uses bidirectional linking to relate an initial SQL to required elements.The passage describes its linking approach as using database-provided external knowledge and analyzing the relationship between the initial SQL and elements needed to answer the question.Source: §II · pp. 2–3

III. METHOD

RSL-SQL generates SQL through four stages, beginning with bidirectional schema linking and ending with multi-turn correction. Its prompts combine database information, examples, the user question, and optional context.

III. METHOD

  • RSL-SQL has four stages: bidirectional schema linking, contextual information augmentation, binary selection, and multi-turn self-correction.The stages aim to recall necessary database elements, increase schema-linking gains, reduce negative effects, and refine erroneous SQL.Source: §III · p. 3

A. Prompt Engineering

  • Prompts include database schema, value samples, schema descriptions, few-shot examples, and the user question, with additional context optional.The paper organizes these elements in Fig. 2.Source: A. Prompt Engineering · p. 3, Figure 2 · p. 4

B. Step 1: Bidirectional Schema Linking (BSL)

  • Bidirectional schema linking begins by generating a preliminary SQL query from the complete database schema.The framework then uses forward and backward linking to recall necessary database elements comprehensively.Source: B. Step 1: Bidirectional Schema… · p. 3, §III · p. 3

1) Forward Schema Linking (FSL):

  • Forward Schema Linking prompts an LLM with the full schema, value samples, and question to identify relevant tables and columns; available context can add columns named in it.The resulting list is denoted Lfwd.Source: 1) Forward Schema Linking (FSL) · p. 3

2) Preliminary SQL Generation:

The method generates a preliminary SQL query using the full schema and forward-linking results, then uses backward linking to form a simplified schema. It augments that schema with column descriptions and predicted SQL components for a second generation step.

2) Preliminary SQL Generation:

  • The preliminary SQL query uses the full schema and value samples, few-shot examples, the question, context, and forward-linking results as supplementary context.The authors report that forward-linking results can slightly enhance preliminary-query accuracy.Source: 2) Preliminary SQL Generation · p. 4

3) Backward Schema Linking (BSL):

  • Backward linking uses exact column-name matching rather than SQLGlot, favoring higher recall while potentially recalling more irrelevant columns.The authors note that SQLGlot reduces noise but also reduces recall.Source: 3) Backward Schema Linking (BSL) · p. 4

4) Schema Simplification:

  • Merging forward- and backward-linked elements produces condensed schemas, value samples, and descriptions while retaining high recall and reducing irrelevant information.The reduced input length and decision space support subsequent SQL generation.Source: 4) Schema Simplification · p. 4

C. Step 2: Contextual Information Augmentation (CIA)

2) SQL Generation with Simplified Schema:

RSL-SQL generates SQL from a simplified schema augmented with contextual information, then uses binary selection to choose between queries from full and simplified schemas. It also flags likely errors for iterative refinement.

2) SQL Generation with Simplified Schema:

  • The simplified-schema SQL prompt combines the schema, value samples, augmented context, few-shot examples, question, and optional context.The augmented context includes simplified-schema descriptions and predicted SQL components.Source: 2) SQL Generation with Simplified Schema · p. 5

D. Step 3: Binary Selection Strategy (BSS)

  • The binary selection strategy uses an LLM to choose between independently generated full-schema and simplified-schema SQL queries.The full schema retains structure but includes redundancy; the simplified schema reduces redundancy but risks information loss.Source: D. Step 3: Binary Selection Strategy… · p. 5

E. Step 4: Multi-Turn Self-Correction (MTSC)

1) Iterative Refinement Process:

Multi-turn self-correction feeds a query and error information to the LLM for repeated refinement. The process ends when the SQL executes without syntax errors, returns a non-empty result, or reaches the iteration limit.

1) Iterative Refinement Process:

  • The refinement loop supplies the question, context, simplified schema, value samples, examples, the initial SQL, and error information to the LLM for another SQL statement.The initial SQL is SQL3, the selected query from the preceding step.Source: 1) Iterative Refinement Process · pp. 5–6

2) Termination Condition:

  • Refinement ends when the SQL executes without syntax errors, returns a non-empty result, or reaches the maximum number of iterations.Source: 2) Termination Condition · p. 6
  • The framework's preceding steps pass the simplified schema and value samples to contextual augmentation, which generates SQL2 for selection against SQL1.The algorithm also specifies the backward-linking and schema-simplification operations that produce the simplified inputs.Source: 2) Termination Condition · p. 6

IV. EXPERIMENT

The experiment covers BIRD and Spider, both described as large-scale, cross-domain Text-to-SQL datasets. Their stated challenges include database values and query efficiency for BIRD, and complex queries across multi-table databases for Spider.

1) BIRD Set:

  • BIRD is a large-scale, cross-domain Text-to-SQL dataset emphasizing dirty external knowledge bases, noisy database values, and SQL query efficiency.Its queries are generally more complex than Spider's.Source: 1) BIRD Set · p. 6

2) Spider Set:

  • BIRD queries are generally more complex than Spider queries, while Spider is characterized by its multi-table databases.Source: 1) BIRD Set · p. 6, 2) Spider Set · p. 6
  • Spider is a large-scale, cross-domain Text-to-SQL dataset with complex queries and multi-table databases for testing generalization.Source: 2) Spider Set · p. 6

3) Subsampled Development Set (SDS):

The experiments use a 10%-per-database BIRD development subsample and define measures for SQL execution, valid-query efficiency, and schema-linking recall. Strict recall evaluates whether all required schema elements are recovered.

3) Subsampled Development Set (SDS):

  • The Subsampled Development Set (SDS) uses 10% of each BIRD development-set database to preserve the set’s distribution and reduce experimental costs.The authors use it to facilitate ablation studies.Source: 3) Subsampled Development Set (SDS) · p. 6

1) Execution Accuracy (EX):

2) Valid Efficiency Score (VES):

3) Non-Strict Recall(NSR):

  • Non-Strict Recall (NSR) is the total overlap between linked and ground-truth schema elements across questions, divided by the total number of ground-truth elements.The linked and ground-truth sets are defined for each question.Source: 3) Non-Strict Recall(NSR) · p. 6

4) Strict Recall Rate(SRR):

  • Strict Recall Rate (SRR) is the proportion of examples in which the linked schema contains all elements required for the correct SQL query.The method aims to maximize SRR while keeping the linked schema small to reduce prompt size.Source: 4) Strict Recall Rate(SRR) · pp. 6–7

C. Baseline Methods

RSL-SQL is compared with a range of Text-to-SQL baselines on BIRD and Spider. With GPT-4o, it sets an open-source BIRD execution-accuracy benchmark and achieves Spider accuracy close to the reported MCS-SQL result.

C. Baseline Methods

  • The BIRD comparison includes task-decomposition, schema-enrichment, prompting, fine-tuning, hallucination-mitigation, and multistage Text-to-SQL baselines.Examples include DIN-SQL and MAC-SQL for task decomposition, and CHESS and E-SQL for schema enrichment and linking.Source: C. Baseline Methods · p. 7

1) BIRD Results:

  • 67.21% execution accuracy on BIRD development sets a new open-source SOTA, with higher accuracy and significantly lower cost than E-SQL using GPT-4o.Source: 1) BIRD Results · p. 8

2) Spider Results:

3) Schema Linking Results:

The section reports bidirectional schema-linking results and analyzes prompt refinement and the incremental RSL-SQL framework components. The reported analyses cover schema recall, prompt inputs, contextual augmentation, binary selection, and self-correction.

3) Schema Linking Results:

  • 94% of columns required for SQL generation are fully recalled by bidirectional linking, which achieves SOTA on both NSR and SRR.Backward linking approaches 90% recall; forward and backward linking are complementary, and their union does not substantially increase the number of columns.Source: 3) Schema Linking Results · p. 8

1) Prompt Refinement:

  • Few-shot examples improve execution accuracy by 10% for both models, and data samples add a further 2% after prompt refinement.The refined prompt alone reaches 62.06% with GPT-4o and 57.63% with DeepSeek.Source: 1) Prompt Refinement · p. 8

2) RSL-SQL Framework:

  • Each framework step improves execution accuracy by 1%–3%, with Steps 2 and 3 contributing over 70% of the overall improvement.Source: 2) RSL-SQL Framework · p. 8, Table IV · p. 9
  • Step 1 incorporates forward-linking results into a preliminary SQL generated against the complete schema, while bidirectional linking simplifies the schema for later steps.The preliminary SQL integration yields an approximate 1% performance improvement.Source: 2) RSL-SQL Framework · p. 8
  • Contextual information augmentation (CIA) improves execution accuracy by 2%–3% and increases positive gains by helping the model map queries to database content and match keywords.CIA may slightly increase negative impact, but the reported positive gain outweighs it.Source: 2) RSL-SQL Framework · p. 9, Figure 7 · p. 9, Table VI · p. 10
  • Binary selection reduces negative impact and increases execution accuracy by 1.5%, while its positive-gain effect differs by backbone.It significantly increases positive gain with DeepSeek but only slightly with GPT-4o, whose accuracy had already reached 65.06% after CIA.Source: 2) RSL-SQL Framework · pp. 9–10, Figure 7 · p. 9
  • Multi-turn self-correction (MTSC) uses execution-risk rules to regenerate and adjust high-risk SQL when execution fails or returns empty results.The strategy slightly improves SQL-generation accuracy.Source: 2) RSL-SQL Framework · p. 10

3) Token Consumption and Cost:

The reported cost comparison finds that RSL-SQL with DeepSeek has lower overhead and outperforms some GPT-4 systems. Bidirectional-linking ablations show that recall, noise, and model strength relate differently to execution accuracy.

3) Token Consumption and Cost:

  • With GPT-4o, E-SQL uses three times as many tokens as RSL-SQL while performing about 2% worse.RSL-SQL with DeepSeek also has lower overhead and performs better than some GPT-4 methods.Source: 3) Token Consumption and Cost · p. 10

1) Bidirectional Schema Linking:

  • With GPT-4o, bidirectional linking yields higher downstream execution accuracy than forward or backward linking alone; with DeepSeek, backward-only linking is slightly higher.The authors attribute the DeepSeek result to bidirectional linking’s higher recall also introducing more noise.Source: 1) Bidirectional Schema Linking · p. 10
  • Forward-only linking trends toward lower execution accuracy in the CIA step for both models, while binary selection significantly improves results based on its outputs.The authors connect the forward-only decline to its lower recall and describe its recall contribution as useful despite potential noise.Source: 1) Bidirectional Schema Linking · p. 10
  • The contextual-information components contribute about 2%–3% performance improvement by helping the model map queries to database content and identify keyword matches.Source: 2) Contextual Information Augmentation · p. 10

2) Contextual Information Augmentation:

Contextual information augmentation adds database elements, SQL keywords, and decomposed query conditions to guide SQL generation. The case study illustrates how augmentation and binary selection can correct errors introduced earlier in the pipeline.

2) Contextual Information Augmentation:

  • Fig. 8 presents BIRD development-set execution accuracy after forward, backward, and bidirectional schema linking.Source: Figure 8 · p. 11
  • Column descriptions improved GPT-4o by over 1%, while SQL-component text improved DeepSeek by only 0.46%.The authors attribute DeepSeek’s smaller gain to lower-quality generated component text.Source: 2) Contextual Information Augmentation · p. 11
  • More than 2% execution-accuracy improvement came from SQL Components Generation as a whole on the subsampled development set.Each component significantly contributed on both DeepSeek and GPT-4o.Source: 2) Contextual Information Augmentation · p. 11

VI. CASE STUDY

  • In the case study, contextual information and later steps can correct errors made during earlier SQL generation.Fig. 9 presents four examples of framework robustness.Source: §VI · p. 11, Figure 9 · p. 12

A. Maximize Positive Gain

  • Filtered database elements focus attention, SQL keywords flag overlooked constraints, and decomposed conditions organize query solving.Source: A. Maximize Positive Gain · p. 11
  • For the lowest-potential player’s preferred foot query, adding the MIN keyword and relevant context corrected an initial SQL that missed parallel conditions.Source: A. Maximize Positive Gain · pp. 11–12

B. Minimize Negative Impact

RSL-SQL reports open-source state-of-the-art performance on BIRD development and a 94% strict recall rate for bidirectional schema linking. Its binary selection strategy uses execution information to select between SQL generated from complete and simplified schemas, limiting negative impact.

B. Minimize Negative Impact

  • In the molecule example, execution returned one row for the complete-schema candidate and no rows for the simplified-schema candidate; the first was selected.Source: B. Minimize Negative Impact · p. 12
  • Binary selection compares SQL from complete and simplified schemas, using execution information to help choose the more accurate candidate and minimize negative impact.Source: B. Minimize Negative Impact · p. 12

VII. CONCLUSION

  • 94% strict recall and open-source state-of-the-art performance on BIRD development are reported for RSL-SQL.The conclusion also describes the framework as balancing performance and computational cost.Source: §VII · p. 12

Related papers

Loading 2411.00073v2…