Source-linked AI summary
The Dawn of Natural Language to SQL: Are We Fully Ready?
Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, Nan Tang
TL;DR
NL2SQL lowers barriers to relational-database access, but existing evaluations do not fully establish which methods suit different domains and SQL characteristics. The paper introduces NL2SQL360 for multi-angle evaluation and design-space exploration, using it to identify SuperSQL, which achieves 87% execution accuracy on Spider and 62.66% on BIRD.
Problem
Existing evaluations often emphasize overall benchmark results and lack direct, comprehensive comparisons, limiting evidence for selecting NL2SQL methods across usage scenarios.
Method
NL2SQL360 systematically evaluates PLM- and LLM-based methods across benchmarks, domains, SQL characteristics, and other scenario-specific criteria, then searches the design space for a tailored solution.
Results
SuperSQL, an interleaved LLM- and PLM-based solution identified through NL2SQL360, achieves 87% execution accuracy on Spider and 62.66% on BIRD.
Takeaways & Limitations
Model selection should be matched to usage scenarios because no single NL2SQL model is a clear winner across domains and SQL characteristics.
Takeaways & Limitations
Existing evaluations commonly report aggregate benchmark results and omit detailed comparisons across specific SQL characteristics and database domains.
Abstract
from arXiv · showhide
Translating users' natural language questions into SQL queries (i.e., NL2SQL) significantly lowers the barriers to accessing relational databases. The emergence of Large Language Models has introduced a novel paradigm in NL2SQL tasks, enhancing capabilities dramatically. However, this raises a critical question: Are we fully prepared to deploy NL2SQL models in production? To address the posed questions, we present a multi-angle NL2SQL evaluation framework, NL2SQL360, to facilitate the design and test of new NL2SQL methods for researchers. Through NL2SQL360, we conduct a detailed comparison of leading NL2SQL methods across a range of application scenarios, such as different data domains and SQL characteristics, offering valuable insights for selecting the most appropriate NL2SQL methods for specific needs. Moreover, we explore the NL2SQL design space, leveraging NL2SQL360 to automate the identification of an optimal NL2SQL solution tailored to user-specific needs. Specifically, NL2SQL360 identifies an effective NL2SQL method, SuperSQL, distinguished under the Spdier dataset using the execution accuracy metric. Remarkably, SuperSQL achieves competitive performance with execution accuracy of 87% and 62.66% on the Spider and BIRD test sets, respectively.
1 INTRODUCTION
NL2SQL lowers barriers to database access, but model performance varies across domains, SQL characteristics, and linguistic query variants, making production deployment and model selection unresolved challenges. NL2SQL360 systematically evaluates these dimensions and uses the findings to identify SuperSQL, which combines LLM- and PLM-based modules.
- NL2SQL converts natural-language questions into SQL, lowering barriers for lay and expert users to access datasets and derive insights.
- Real-world NL2SQL evaluation must account for varied domains, complex SQL operations, and linguistic query variants rather than overall benchmark accuracy alone.Users may express the same intent with different abbreviations, synonyms, and question styles.
- Fine-tuned methods outperform prompting methods in competition-domain execution accuracy, with RESDSQL-3B+NatSQL reaching 83.9% versus DAILSQL at 80.6%.The passage attributes this pattern to fine-tuning improving domain adaptation.
- PLM-based methods lead on JOIN-only queries, whereas LLM-based methods generally outperform PLM-based methods on nested queries.
- Fine-tuned LLMs and PLMs better handle linguistic query variation, and in-domain fine-tuning data is crucial for performance across domains.
- NL2SQL360 evaluates methods across benchmarks and application-specific criteria, then identifies SuperSQL with 87% execution accuracy on Spider and 62.66% on BIRD.The evaluation covered 13 LLM-based and 7 PLM-based solutions across 15 settings.
2 NATURAL LANGUAGE TO SQL
NL2SQL has evolved from rule-based and neural methods to PLM- and LLM-based approaches, but existing evaluations do not fully reveal how methods perform across scenarios or how their components can be combined.
- Evolution of NL2SQL: NL2SQL research evolved through rule-based, neural network-based, PLM-based, and LLM-based methods.Rule-based approaches had limited adaptability, scalability, and generalization; later methods introduced benchmark datasets, pretrained models, and large language models.
- NL2SQL System Design: Recent competitive NL2SQL systems use language models as backbones, including API-based LLMs and tunable models such as T5 and Llama.Systems also differ in modules such as schema linking, database-content alignment, constrained decoding, and decoding strategy.
- Existing Evaluation Gaps: Existing experiments often report overall benchmark results without detailed comparisons across SQL characteristics or database domains.Filtering datasets by query properties or domains could reveal which models work best for particular scenarios.
- Existing Evaluation Gaps: Many LLM- and PLM-based solutions lack systematic direct comparisons on established benchmarks and customized datasets.This limits comprehensive assessment of their relative effectiveness across evaluation settings.
- Existing Evaluation Gaps: Current research has limited exploration of combining LLM- and PLM-based modules within the NL2SQL design space.The paper identifies this as restricting understanding of how their architectural and functional modules can be incorporated together.
3 NL2SQL360: A TESTBED FOR NL2SQL
NL2SQL360 is a testbed for evaluating NL2SQL methods across benchmark subsets, SQL and domain scenarios, and query variations using multiple accuracy and efficiency measures.
- Testbed Components: NL2SQL360 comprises benchmark datasets, a model zoo, dataset filtering, an executor and logs, an evaluator, and visualization tools.The framework supports widely used benchmarks and competitive LLM- and PLM-based models.
- Evaluation Scenarios: Dataset filtering creates focused subsets based on query complexity, SQL characteristics, data domains, and natural-language query variation.These scenarios include JOINs, subqueries, aggregate functions, and domains such as finance, healthcare, and retail.
- Evaluation Metrics: NL2SQL360 supports Execution Accuracy, Exact Match Accuracy, and Valid Efficiency Score for assessing generated-query effectiveness and valid-query generation efficiency.These metrics are applied alongside scenario-specific dataset filtering.
- Evaluation Metrics: Query Variance Testing measures how robustly and flexibly an NL2SQL system adapts to different phrasings and structures of natural-language queries.The metric is designed to assess adaptation to diverse linguistic forms.
- Evaluation Metrics: For each SQL query, Query Variance Testing groups multiple natural-language variants and evaluates model-generated SQL against the corresponding query results.The formulation uses the number of SQL queries, the number of variants per query, generated queries, and an indicator of equal results.
4 EXPERIMENTS
Experiments show that NL2SQL performance depends strongly on SQL characteristics, domains, metrics, and resource constraints rather than a single universally superior model family. Fine-tuning, model reasoning, and domain-specific training each affect different evaluation outcomes.
- Overall accuracy: Fine-tuning improves performance overall: fine-tuned LLMs lead on EX, while PLM-based methods lead on EM.The reported distinction holds across the overall comparison.
- SQL characteristics: Subqueries are the hardest SQL cases, but LLM-based methods—especially GPT-4 systems—outperform PLM-based methods on them.The paper attributes this pattern to the reasoning demands of generating subqueries.
- SQL characteristics: LLM-based methods outperform PLM-based methods when queries require logical connectors or JOIN operations.Without JOIN, performance is inconsistent; NatSQL may reduce JOIN-prediction complexity.
- SQL characteristics: ORDER BY results vary by dataset: LLMs underperform PLMs on Spider but outperform them on BIRD, while LLMs generally show stronger generalization.The paper links the difference partly to BIRD’s greater complexity.
- Domains and robustness: No model family consistently wins across domains or QVT, although in-domain fine-tuning data improves domain performance and can stabilize results against language variation.Fine-tuned LLMs generally achieve higher QVT than prompting LLMs.
- Efficiency: GPT-3.5-turbo prompt-based methods provide higher EX-to-cost ratios, while larger models increase latency and GPU-memory requirements.For example, RESDSQL-Base+NatSQL reaches 80.2% EX with lower latency and memory than RESDSQL-Large at 80.1%.
5 COMBINING THE BEST OF BOTH WORLDS
NL2SQL360 represents solutions as combinations of modular preprocessing, prompting, generation, and post-processing choices, then uses automated search to find stronger combinations. Its selected SuperSQL configuration achieves competitive accuracy and efficiency on Spider and BIRD.
- Design space: The design space covers schema linking, database contents, prompting, SQL generation, and post-processing modules.Generation choices include multi-step strategies, decoding constraints, and intermediate representations such as NatSQL.
- Design space: Post-processing options include self-correction, self-consistency, execution-guided selection, and N-best reranking.These modules respectively repair, vote over, validate, or rank generated SQL candidates.
- Automated architecture search: NL2SQL360-AAS uses a genetic algorithm to search predefined module combinations, treating each valid combination as an individual evaluated by EX, EM, or VES.The process initializes randomized systems, selects higher-scoring individuals, swaps modules, and mutates modules across generations.
- Automated architecture search: NL2SQL360-AAS iteratively improves candidate populations through metric-based selection, module swapping, and module mutation.The evaluation uses randomized initial systems and repeats the evolutionary steps across generations.
- SuperSQL: SuperSQL achieves 87.0% EX on Spider test and 62.66% EX on BIRD test, outperforming DAILSQL(SC) by 5.25% on BIRD.It also attains VES scores of 99.18 on Spider and 61.99 on BIRD development sets.
6 RESEARCH OPPORTUNITIES
The paper identifies trustworthiness, interpretability, cost, and adaptation to unseen databases as key directions for future NL2SQL research. Proposed directions target ambiguity, debugging, efficiency, and training-data coverage.
- Trustworthiness: Incorrect SQL results can arise from ambiguous queries, ambiguous schemas or dirty database contents, and inadequate schema linking.The paper frames these issues as barriers to trustworthy NL2SQL systems.
- Trustworthiness: Query rewriting and auto-completion are proposed to make underspecified user questions clearer and better aligned with database content.Auto-completion suggests candidate tokens for formulating queries.
- Trustworthiness: NL2SQL debuggers and SQL-result interpretation tools could help users identify generation errors and assess whether outputs meet their requirements.The proposed debugger supports stepping through the SQL generation process.
- Cost effectiveness: Reducing token consumption and inference time while preserving accuracy is an open challenge for costly LLM-based methods.The paper points to modularized solutions and multi-agent frameworks as possible directions, especially for complex queries.
- Training data: Dynamically generating natural-language/SQL training pairs from model-evaluation feedback could improve coverage, quality, and adaptation to unseen databases.The proposed approach uses performance insights to create diverse training data.
7 CONCLUSION
NL2SQL360 evaluates NL2SQL methods across SQL and database-domain characteristics, then searches its design space for a user-tailored solution. SuperSQL, an interleaved LLM- and PLM-based method, achieves 87% execution accuracy on Spider and 62.66% on BIRD.
- NL2SQL360 evaluates NL2SQL methods across SQL characteristics and database domains in a fine-grained, multi-angle testbed.
- NL2SQL360 evaluates 13 LLM-based and 7 PLM-based methods across 2 benchmarks and 15 settings.
- SuperSQL interleaves LLM-based and PLM-based modules to provide a solution tailored to user-specific needs.
- 87% and 62.66% execution accuracy are achieved by SuperSQL on the Spider and BIRD test sets, respectively.