PaperlayerView exact revision on arXiv

Source-linked AI summary

E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL

Hasan Alp Caferoğlu, Özgür Ulusoy

arXiv:2409.16751v2cs.CL

TL;DR

Text-to-SQL systems must map natural-language questions to database structures and generate accurate SQL, yet a substantial gap from human-level accuracy remains. E-SQL enriches questions with database elements and candidate predicates, then refines generated SQL. On BIRD, E-SQL with GPT-4o achieved 66.29% execution accuracy on the test set.

  • Problem

    Text-to-SQL must bridge natural-language questions and database structures, but the best-performing models remain around 20% below human-level accuracy.

    Source: §1 · p. 1

  • Method

    E-SQL enriches questions with relevant database elements and SQL construction steps, augments them with candidate predicates, and uses the enriched input in SQL refinement.

    Source: §1 · p. 2

  • Results

    On BIRD, E-SQL with GPT-4o achieved 65.58% execution accuracy on the development set and 66.29% on the test set.

    Source: §6 · p. 12

  • Takeaways & Limitations

    Question enrichment improves query accuracy, particularly for complex queries, and candidate-predicate augmentation further improves pipeline performance.

    Source: §5 · p. 11, §6 · p. 12

    Due to hardware and cost constraints, almost all experiments used GPT-4o-mini or small open-source LLMs without fine-tuning; the full pipeline was tested with only one small model.

    Source: §5 · p. 11

Abstract

from arXiv · show

Translating Natural Language Queries into Structured Query Language (Text-to-SQL or NLQ-to-SQL) is a critical task extensively studied by both the natural language processing and database communities, aimed at providing a natural language interface to databases (NLIDB) and lowering the barrier for non-experts. Despite recent advancements made through the use of Large Language Models (LLMs), significant challenges remain. These include handling complex database schemas, resolving ambiguity in user queries, and generating SQL queries with intricate structures that accurately reflect the user's intent. In this work, we introduce E-SQL, a novel pipeline specifically designed to address these challenges through direct schema linking and candidate predicate augmentation. E-SQL enhances the natural language query by incorporating relevant database items (i.e., tables, columns, and values) and conditions directly into the question and SQL construction plan, bridging the gap between the query and the database structure. The pipeline leverages candidate predicate augmentation to mitigate erroneous or incomplete predicates in generated SQLs. Comprehensive evaluations on the BIRD benchmark illustrate that E-SQL achieves competitive performance, particularly excelling in complex queries with a 66.29% execution accuracy on the test set. A further observation from our experiments reveals that incorporating schema filtering into the translation pipeline does not have a positive impact on performance when the most advanced proprietary LLMs are used. Additionally, our experiments with small LLMs highlight the importance and positive impact of enriched questions on their performance. Without fine-tuning, single-prompt SQL generation using enriched questions with DeepSeek Coder 7B Instruct 1.5v achieves 56.45% execution accuracy on the BIRD development set.

1 Introduction

E-SQL addresses Text-to-SQL schema linking by enriching questions with database items, possible conditions, and SQL construction steps. The introduction reports gains from question enrichment and candidate predicates, while noting that schema filtering can hurt performance with advanced proprietary LLMs.

1 Introduction

  • The introduction describes an around 20% gap between the best-performing models and human-level accuracy, and says current pipelines are not yet suitable for real-world database interfaces.Source: §1 · p. 1
  • E-SQL directly links schemas by enriching questions with database items and potential conditions, then adds SQL construction steps and candidate-predicate augmentation.The pipeline comprises Candidate SQL Generation, Candidate Predicate Generation, Question Enrichment, and SQL Refinement.Source: §1 · p. 2
  • The pipeline generates candidate SQL, retrieves similar database values to form candidate predicates, enriches the question, executes the draft, then refines or regenerates SQL.Source: §1 · p. 2
  • Schema filtering can hurt performance with advanced proprietary LLMs, while direct schema linking through enrichment and candidate predicates is reported as more reliable, especially for complex cases.The experiments explore adding schema filtering and correcting inconsistencies between filtered and original schemas.Source: §1 · p. 2
  • Nearly 5%: question enrichment increased accuracy on challenging questions in the ablation study.Source: §1 · p. 2
  • Evaluations on Spider and BIRD report competitive performance with state-of-the-art methods, particularly for complex queries involving multiple conditions and joins.Source: §1 · p. 2

2 Related Work

Related work spans neural Text-to-SQL systems, LLM reasoning prompts, schema linking and filtering, prompt data augmentation, and SQL correction. The paper positions question enrichment with embedded reasoning and candidate predicates as additions to these lines of work.

2 Related Work

  • Earlier Text-to-SQL systems used neural encoder-decoder architectures, while recent work increasingly applies proprietary and open-source LLMs.Source: §2 · p. 2

2.1 LLM Reasoning

  • Question refinement with embedded reasoning for Text-to-SQL remains underexplored; E-SQL proposes schema-aware enriched queries to guide SQL construction and address schema linking.Prior reasoning strategies include chain-of-thought prompting, decomposition, self-consistency, and iterative refinement.Source: §2.1 · pp. 2–3

2.2 Schema Linking and Filtering

  • Prior schema-linking systems rank or filter database items, whereas E-SQL extracts possible values from candidate SQL predicates for downstream augmentation instead of filtering with them.Earlier work reports both benefits of filtering irrelevant items and potential performance degradation with advanced LLMs.Source: §2.2 · pp. 3–4

2.3 Data Augmentation

  • Prompt augmentation commonly supplies schema descriptions and similar database values; E-SQL additionally proposes candidate conditions with exact values to reduce predicate errors.The paper also includes candidate-query execution errors as feedback during refinement.Source: §2.3 · p. 4

2.4 SQL Query Generation and Correction

  • SQL correction methods include self-correction and selecting among multiple generations; E-SQL refines SQL after question enrichment and candidate-predicate augmentation.The paper notes that multi-generation approaches can incur high computational costs.Source: §2.4 · p. 4

3 Methodology

E-SQL links questions directly to database schema by adding relevant database items and possible predicates, then uses an SQL construction plan to guide generation. Its pipeline combines candidate SQL and predicate generation, question enrichment, and SQL refinement; experiments found schema filtering could reduce performance with the latest proprietary LLMs.

3.1 Database Description and Value Selection

  • BM25 selects relevant database values and descriptions for prompts, while NULL-containing columns retain NULL among their selected values.The prompt includes the 10 most relevant values per column and 20 relevant database-description sentences.Source: §3.1 · pp. 4–5

3.2 Candidate SQL Generation Module (CSG)

  • Candidate SQL Generation produces an initial query whose potentially useful information, including predicate values, is reused to improve later pipeline steps.Its prompt includes schema code, selected descriptions and values, and few-shot examples from other databases.Source: §3.2 · p. 5

3.3 Candidate Predicate Generation (CPG)

  • Candidate Predicate Generation parses values and operations from an initial SQL query, retrieves matching database values with LIKE, and supplies possible predicates for downstream tasks.The authors target predicate errors involving incomplete values or incorrect schema selections.Source: §3.3 · p. 5

3.4 Schema Filtering and Filtered Schema Correction Module (SF)

  • Schema filtering selects relevant tables and columns, and filtered-schema correction checks and repairs mismatches with the original schema, but filtering reduced performance with the latest proprietary LLMs.The authors therefore exclude schema filtering from the final E-SQL pipeline.Source: §3.4 · p. 6

3.5 Question Enrichment Module (QE)

  • Question enrichment adds relevant tables, columns, values, and conditions to the question, then appends an SQL construction plan to guide SQL generation.The fully enriched question combines the original question, enriched question, and enrichment reasoning; it is generated in a single prompt without iterative refinement.Source: §3.5 · pp. 6–7

3.6 Predicate and Error-Aware SQL Refinement Module (SR)

  • SQL Refinement uses execution errors, candidate predicates, the enriched question, and schema to prompt the LLM to generate a new SQL query or refine its candidate.The candidate SQL is executed before the LLM is prompted to correct or replace it.Source: §3.6 · p. 7

4 Experiments and Results

The experiments evaluate E-SQL on Spider and BIRD using execution accuracy and other BIRD metrics, with proprietary and small open-source LLMs. They standardize prompting and generation settings across tests.

4.1 Experiment Settings

  • Spider evaluation uses 2,147 test pairs, while BIRD contains 12,751 pairs across 95 databases and has a concealed test set.BIRD development and test splits contain 1,534 and 1,789 instances, respectively.Source: §4.1.1 · p. 7

4.1.1 Dataset.

  • The evaluation uses Execution Accuracy, which compares predicted and ground-truth SQL execution results rather than requiring syntactically identical queries.The study also reports Reward-based Valid Efficiency Score and Soft F1-score.Source: §4.1.2 · p. 7

4.1.2 Evaluation Metrics.

  • Reward-based Valid Efficiency Score accounts for accuracy and runtime, while Soft F1-score is designed to tolerate minor output discrepancies such as column reordering or missing values.Soft F1-score complements Execution Accuracy with a more flexible assessment.Source: §4.1.2 · pp. 7–8

4.1.3 Models.

  • Experiments use GPT-4o-mini, GPT-4o, Qwen2.5 Coder, and DeepSeek Coder without fine-tuning, with fixed generation settings and 9-shot examples per module.Prompts also include 10 relevant distinct values and 20 relevant database-description sentences.Source: §4.1.3 · p. 8, §4.1.4 · p. 8

4.2 Results

E-SQL reports competitive BIRD performance, with results across query difficulty levels. The experiments also find that schema filtering reduces performance with advanced LLMs, so it is excluded from the final pipeline.

4.2 Results

  • 66.29 Execution Accuracy (EX) with GPT-4o on the BIRD test set; E-SQL also reports 59.81 EX with GPT-4o-mini on that set.The reported Soft F1-Scores are 67.93 with GPT-4o and 61.59 with GPT-4o-mini.Source: §4.2 · p. 8, Table 1 · p. 8

4.3 Schema Filtering

  • Schema filtering consistently reduced performance across difficulty levels, with drops of 6.38%, 12.28%, and 8.96% for simple, moderate, and challenging BIRD development questions.The pipeline comparison also reports an overall drop of up to 8.21% in EX.Source: Table 4 · p. 9, §4.3 · p. 9
  • Adding schema filtering to E-SQL lowered BIRD development performance by 5.54 EX and 5.68 Soft F1, so it was excluded from the final pipeline.The authors report that advanced LLMs can manage schema linking without explicit filtering, aligning with previous research [31].Source: §4.3 · p. 9, Table 5 · p. 9

4.4 Ablation Study

The ablations examine Question Enrichment (QE), Candidate Predicate Augmentation (CPG), and SQL Refinement (SR). QE improves performance, while CPG and SR have distinct effects across overall and challenging-query results.

4.4 Ablation Study

  • QE improved performance with advanced proprietary and small open-source LLMs, with particularly significant effects on challenging questions.The ablation study evaluates QE, CPG, and SR as individual pipeline modules.Source: §4.4.1 · p. 9, §4.4 · p. 9

4.4.1 Question Enrichment.

  • Removing QE, especially alongside CPG, further decreased overall performance, while question reformulation provides direct schema linking between natural-language queries and the database schema.The authors connect this direct schema linking with more accurate SQL generation.Source: §4.4.1 · p. 10

4.4.2 Possible Predicate Augmentation.

  • Removing CPG caused a nearly 2% overall performance drop, although it slightly improved results on challenging BIRD development questions.The authors describe that challenging-question effect as negligible relative to CPG’s overall contribution.Source: §4.4.2 · p. 10

4.4.3 SQL Refinement.

  • Without SR, BIRD development performance fell by 3.57 EX and 3.73 Soft F1; SR corrected 5.35% of initially incorrect GPT-4o-mini queries and 1.83% of GPT-4o queries.The correction percentages are from the analysis of the SQL Refinement module.Source: §4.4.3 · p. 10, Table 7 · p. 10

4.5 Impact of Enriched Questions on Small Large Language Models

The study tests whether database-integrated enriched questions affect small LLMs in single-prompt SQL generation. The results report competitive performance without task-specific fine-tuning when questions include database items and logical SQL construction steps.

4.5 Impact of Enriched Questions on Small Large Language Models

  • The single-prompt comparison isolates enriched questions by omitting database descriptions, value samples, and few-shot examples from the generation prompt.The enriched questions were produced by the QE module using GPT-4o and included database items and SQL construction plans.Source: §4.5 · p. 10
  • Small LLMs achieved competitive performance without task-specific fine-tuning when given high-quality database-integrated questions containing logical SQL construction steps.The study compares default questions with enriched questions.Source: §4.5 · p. 10

4.6 Computational Cost Analysis

Question enrichment increases question and reasoning token counts, but that increase is relatively small compared with total prompt tokens. E-SQL executes each module once, and execution-error feedback adds an average of 49.936 milliseconds per query.

4.6 Computational Cost Analysis

  • Question enrichment increases question and reasoning token counts, but the added question tokens are relatively insignificant compared with total prompt tokens.The prompts are already token-intensive because of instructions, few-shot examples, database descriptions, and database values.Source: §4.6 · p. 10, Table 9 · p. 11
  • E-SQL executes each module once, which the authors contrast with methods that generate, correct, and select among multiple SQL queries [10] [13] [22] [31] [40].They describe this design as minimizing costs associated with repeated calls.Source: §4.6 · pp. 10–11
  • 49.936 milliseconds per query is the average execution time for the initial candidate SQL query, which supplies execution-error feedback for SQL refinement.Overall response time also varies with question complexity, prompt length, and proprietary-model API response time.Source: §4.6 · p. 11

5 Discussion and Limitations

The discussion attributes gains on challenging queries to question enrichment’s schema linking, while noting unresolved evaluation variability and scope constraints. It also limits exploration of alternative prompts and full-pipeline small-model use.

5 Discussion and Limitations

  • Question enrichment improves accuracy particularly on challenging questions by linking enriched database items, conditions, and SQL steps to the schema.The authors cite an ablation study as evidence for the module’s efficacy.Source: §5 · p. 11
  • Development-to-test performance trends differed between GPT-4o and GPT-4o-mini, and the unavailable BIRD test set prevented analysis of the causes.The authors note that variation across LLM runs might also contribute; the reasons remain unclear.Source: §5 · p. 11
  • Alternative prompt templates were left outside the study’s scope, although prompt design influences model performance.The study focuses on schema linking through question enrichment and data augmentation.Source: §5 · p. 11
  • Hardware and cost constraints limited most experiments to GPT-4o-mini and small open-source LLMs without fine-tuning.The full pipeline was run only with Qwen2.5 Coder 7B Instruct among small models, because other models lacked sufficient context length.Source: §5 · p. 11

6 Conclusions

E-SQL combines question enrichment and candidate predicates for Text-to-SQL, reporting competitive BIRD results while finding that schema filtering can degrade performance with advanced LLMs. The conclusion also notes efficiency trade-offs and directions left for future work.

6 Conclusions

  • Question enrichment links natural-language queries with database items and logical steps; candidate predicate augmentation further improves pipeline performance.The conclusion reports that enriched questions positively affect small open-source LLMs with limited context lengths.Source: §6 · p. 12
  • Schema filtering degraded performance in the text-to-SQL pipeline when used with advanced LLMs.The authors suggest explicit filtering may be redundant in modern architectures using the latest LLMs.Source: §6 · p. 12
  • 65.58% development and 66.29% test execution accuracy were achieved by E-SQL with GPT-4o on BIRD.The authors characterize the results as competitive and highlight handling complex queries.Source: §6 · p. 12
  • Question enrichment adds token overhead, but the authors say it is negligible relative to total usage and outweighed by performance gains.Each module runs once, avoiding repeated query generation and correction demands.Source: §6 · p. 12
  • Fine-tuning, iterative or multiple question refinements, and schema linking optimized for small LLMs with limited context lengths remain future work.Source: §6 · p. 12

A Prompt Templates

The appendix provides full prompt templates for candidate SQL generation, question enrichment, SQL refinement, and schema filtering.

A.1 Full Prompt Template for Candidate SQL Generation (CSG)

  • Candidate SQL Generation prompts combine the question with schema, descriptions, samples, and evidence to generate SQLite SQL and reasoning in JSON.Source: §A.1 · p. 15
  • The CSG template preserves question and database wording, requires backticked identifiers, and specifies safeguards for extrema, arithmetic, and null handling.Source: §A.1 · p. 15

A.2 Full Prompt Template for Quesiton Enrichment (QE)

  • Question Enrichment prompts use schema, descriptions, samples, evidence, and possible conditions to expand the question with relevant database items and SQL steps.Source: §A.2 · p. 16
  • The QE task requests original database terminology and an expanded, clear question incorporating identified components and conditions.Source: §A.2 · p. 16

A.3 Full Prompt Template for SQL Refinement (SR)

  • SQL Refinement prompts ask the model to inspect a candidate query for condition, function, aggregation, syntax, and execution-error problems, then validate the revised SQL.Source: §A.3 · p. 17
  • The SR template emphasizes schema-consistent joins, accurate aggregation and filters, exact reuse of selected conditions, and deliberate extrema and null handling.Source: §A.3 · p. 17

A.4 Full Prompt Template for Schema Filtering (SF)

  • Schema Filtering selects useful tables and columns by matching the question and evidence against schema, descriptions, and sample values.The prompt requests selected columns grouped under their correct database tables.Source: §A.4 · p. 18

B Manually Enriched Questions and Enrichment Reasoning

The appendix presents manually enriched questions and reasoning that connect natural-language requests to database items, joins, conditions, and SQL operations.

B Manually Enriched Questions and Enrichment Reasoning

  • The SAT example enriches a school-count question by linking direct charter funding and Math scores above 560 to a join and COUNT operation.The examples and their enrichment reasoning are available in the project repository.Source: B Manually Enriched Questions and… · p. 19
  • The phone-number example maps charter funding and post-2000 opening date conditions to schools and frpm fields, joined on CDSCode.The enriched question selects the schools table’s Phone column.Source: B Manually Enriched Questions and… · p. 19
  • The charter-school example links school phone numbers, charter indicators, direct funding, and opening dates to a CDSCode join and selection conditions.Source: B Manually Enriched Questions and… · p. 19
  • The creatinine example turns a male-versus-female comparison into a Patient–Laboratory join, a CRE threshold, and CASE WHEN counts returned as True or False.The example defines not-within-normal-range as creatinine at least 1.5.Source: B Manually Enriched Questions and… · p. 19

C E-SQL Execution Flow

Figure 7 presents the E-SQL execution flow for development-set question 1448.

C E-SQL Execution Flow

  • Figure 7 depicts the E-SQL execution flow for question 1448 in the development set.Source: Figure 7 · p. 20
Loading 2409.16751v2…