Source-linked AI summary
OmniSQL: Synthesizing High-quality Text-to-SQL Data at Scale
Haoyang Li, Shang Wu, Xiaokang Zhang, Xinmei Huang, Jing Zhang, Fuxin Jiang, Shuai Wang, Tieying Zhang, Jianjun Chen, Rui Shi, Hong Chen, Cuiping Li
TL;DR
Text-to-SQL needs training data that generalizes beyond limited public datasets without depending on costly, privacy-sensitive closed-source APIs. The paper proposes an automatic synthesis framework and uses it to train OmniSQL, which achieves state-of-the-art results across nine datasets while remaining open source and smaller than leading models.
Problem
Public text-to-SQL datasets have limited real-world coverage, while prompting methods rely on costly, privacy-sensitive, and less customizable closed-source LLMs.
Method
The paper automatically synthesizes diverse text-to-SQL data through realistic databases, complexity-controlled SQL, stylized questions, and chain-of-thought solutions, then fine-tunes OmniSQL.
Results
Across nine benchmarks, OmniSQL substantially outperforms similarly scaled baseline LLMs and matches state-of-the-art models despite using fewer parameters.
Takeaways & Limitations
SynSQL-2.5M and OmniSQL provide open-source text-to-SQL data and models for further research.
Takeaways & Limitations
The evaluation isolates single-step inference and does not directly compare OmniSQL with multi-stage text-to-SQL frameworks.
Abstract
from arXiv · showhide
Text-to-SQL, the task of translating natural language questions into SQL queries, plays a crucial role in enabling non-experts to interact with databases. While recent advancements in large language models (LLMs) have significantly enhanced text-to-SQL performance, existing approaches face notable limitations in real-world text-to-SQL applications. Prompting-based methods often depend on closed-source LLMs, which are expensive, raise privacy concerns, and lack customization. Fine-tuning-based methods, on the other hand, suffer from poor generalizability due to the limited coverage of publicly available training data. To overcome these challenges, we propose a novel and scalable text-to-SQL data synthesis framework for automatically synthesizing large-scale, high-quality, and diverse datasets without extensive human intervention. Using this framework, we introduce SynSQL-2.5M, the first million-scale text-to-SQL dataset, containing 2.5 million samples spanning over 16,000 synthetic databases. Each sample includes a database, SQL query, natural language question, and chain-of-thought (CoT) solution. Leveraging SynSQL-2.5M, we develop OmniSQL, a powerful open-source text-to-SQL model available in three sizes: 7B, 14B, and 32B. Extensive evaluations across nine datasets demonstrate that OmniSQL achieves state-of-the-art performance, matching or surpassing leading closed-source and open-source LLMs, including GPT-4o and DeepSeek-V3, despite its smaller size. We release all code, datasets, and models to support further research.
1 INTRODUCTION
Existing text-to-SQL methods make strong benchmark progress but remain costly, privacy-sensitive, difficult to customize, or poorly generalizable. OmniSQL addresses these gaps with scalable synthetic data and an open-source multi-scale model.
- Prompting-based methods rely on closed-source APIs, creating high costs, privacy concerns, and limited control over model behavior.
- 43.8% and 31.4% execution accuracy on ScienceBenchmark and EHRSQL show limited out-of-domain generalizability for Qwen2.5-Coder-7B-Instruct fine-tuned on Spider and BIRD.GPT-4-Turbo reaches 59.2% and 43.1% on the same datasets under zero-shot prompting.
- Large-scale, diverse, high-quality training data is proposed as a route to improving open-source text-to-SQL performance and generalizability.Local deployment can also make open-source systems more cost-effective, data-secure, and adaptable.
- The framework generates realistic databases, complexity-aware SQL queries, stylized questions, and chain-of-thought solutions automatically and at scale.Its four-stage design reduces reliance on extensive human intervention.
- OmniSQL is an open-source 7B, 14B, and 32B model trained on SynSQL-2.5M and evaluated across nine datasets.The introduction reports state-of-the-art average performance across the evaluation suite.
2 RELATED WORK
Related work spans encoder-decoder, seq2seq, LLM-based, and synthetic-data approaches. OmniSQL differs by decoupling synthesis into controllable stages and adding chain-of-thought solutions while reducing reliance on expensive proprietary models.
- Early text-to-SQL systems use encoder-decoder architectures that encode schemas and questions before generating SQL.Some improve encoders with graph relations or pretraining, while others constrain decoders with grammars.
- Seq2seq models and later LLMs recast text-to-SQL as generation and decompose it into simpler subtasks.
- SQL templates and grammars constrain diversity, while grammar-based augmentation is labor-intensive and difficult to scale.
- Question-to-SQL augmentation can introduce noise from inaccurate models, whereas template-pair methods produce limited diversity and unnatural questions.
- Unlike Sense, OmniSQL decouples synthesis into simpler controllable steps, enabling less powerful open-source LLMs and adding CoT solutions for interpretability.
3 DATA SYNTHESIS FRAMEWORK
The framework progressively synthesizes databases, SQL, natural-language questions, and reasoning traces, using controls and post-processing to improve realism, complexity, linguistic diversity, and quality. It begins from abundant web tables and enhances generated schemas before filtering and diversifying queries.
- Framework overview: The four-stage pipeline synthesizes web-table-driven databases, complexity-aware SQL queries, stylized questions, and chain-of-thought solutions.LLMs are combined with automated preprocessing and post-processing to reduce extensive human intervention.
- Database synthesis: Web tables seed realistic business scenarios and relational databases, addressing the scarcity of accessible real-world enterprise databases.Each generated database includes relational tables and schema information such as primary and foreign keys.
- Database synthesis: Database enhancement adds relevant columns and completes missing primary and foreign key relationships after initial generation.The step targets overly simplistic tables and incomplete relationships observed in initial outputs.
- SQL synthesis: Four SQL complexity levels guide query generation, while advanced functions and sampled database values support meaningful, realistic queries.
- SQL synthesis: Post-processing removes non-SELECT, syntactically invalid, timed-out, and duplicate-template queries to enforce validity and diversity.Only one query is retained per extracted SQL template.
- Question synthesis: Nine language styles generate questions ranging from formal and concise to vague, metaphorical, and conversational.The styles model varied real-world expressions, including ambiguity, figurative language, and iterative clarification.
- Reasoning synthesis: Chain-of-thought synthesis can produce SQL better aligned with questions than original triplets containing unnecessary columns or incorrect join paths.
4 SYNSQL-2.5M: A MILLION-SCALE DATASET
SynSQL-2.5M is an automatically generated, million-scale text-to-SQL dataset designed for broad domain, question, database, and SQL coverage. Evaluations indicate substantial structural diversity and high synthetic-data quality.
- 2,544,390 samples span 16,583 synthetic databases, with each sample containing a database, question, SQL query, and CoT solution.
- The framework uses multiple LLMs, complexity levels, advanced SQL functions, and sampled database values to diversify synthetic queries and reduce model-specific bias.
- Database and Question Diversity: SynSQL-2.5M’s synthetic databases exceed WikiSQL, Spider, and BIRD in average structural complexity, while its question embeddings encompass human-annotated dataset distributions.
- SQL Statistics: Synthetic SQL averages 1.75 joins per query, compared with 0.94 in BIRD and 0.48 in Spider.
- SQL Statistics: SynSQL-2.5M provides over 2 million unique SQL skeletons and covers 83 database-engine functions.
5 OMNISQL: STATE-OF-THE-ART OPEN-SOURCE TEXT-TO-SQL LLM
OmniSQL is trained from SynSQL-2.5M, Spider, and BIRD using schema- and value-enriched inputs paired with chain-of-thought solutions and final SQL queries. Its objective is conditional next-token prediction over these outputs.
- OmniSQL is available in 7B, 14B, and 32B scales and is trained using SynSQL-2.5M together with Spider and BIRD.
- Input-Output Construction: Each input combines a CREATE TABLE-formatted schema, natural-language question, column descriptions, representative values, and question-relevant values.
- Input-Output Construction: The output is a step-by-step CoT solution followed by the final SQL query; CoT solutions are also synthesized for Spider and BIRD training data.
- Supervised Fine-Tuning: OmniSQL is fine-tuned with conditional next-token prediction to predict the CoT solution from database information and the corresponding question.
6 EXPERIMENTS
The experiments comprehensively evaluate OmniSQL against leading large language models. The supplied passage establishes the comparative evaluation scope but does not report outcomes.
- OmniSQL’s performance is evaluated by comparison with leading LLMs.
6.1 Experimental Setup
The evaluation covers standard cross-domain benchmarks and difficult enterprise or professional-domain settings, using execution-based metrics and comparisons across model families. The setup also examines deterministic and voting-based inference.
- Datasets: Experiments use Spider dev/test, BIRD dev, Spider2.0-SQLite, ScienceBenchmark, and EHRSQL to assess cross-domain, enterprise, and professional-domain text-to-SQL.
- Evaluation Metrics: Execution accuracy measures matching results on one database, while test-suite accuracy evaluates results across multiple test-suite databases.
- Implementation: OmniSQL-7B and 14B are fully fine-tuned, whereas OmniSQL-32B uses LoRA because of limited GPU computational resources.
- Baselines: The comparison includes closed-source and open-source LLMs ranging from 6.7B to 671B parameters and spanning general-purpose and code-specific models.
- Inference: Greedy decoding is deterministic, while sampling generates eight candidates per sample and selects the SQL receiving the most execution-result votes.
6.2 Main Results
OmniSQL delivers leading text-to-SQL performance across standard, domain-specific, and robustness benchmarks, with benefits varying by model scale and decoding strategy.
- OmniSQL-7B improves Spider test accuracy from 87.9% with greedy decoding to 88.9% with majority voting.
- OmniSQL-32B reaches 67.0% majority-voting accuracy on BIRD development, near Distillery + GPT-4o’s 67.2%.
- OmniSQL-32B achieves 46.8% majority-voting accuracy on EHRSQL, exceeding GPT-4o’s 45.5%.
- OmniSQL-7B reaches 10.4% greedy accuracy on Spider2.0-SQLite, compared with 0.7%–3.7% for 7B-scale baselines.
- OmniSQL performs strongly across standard and domain-specific datasets, whereas some open-source models’ advantages diminish outside familiar benchmarks.
- OmniSQL is robust on Spider-Syn and Spider-Realistic, but OmniSQL-14B and OmniSQL-32B underperform their base models on Spider-DK.
- Full fine-tuning benefits OmniSQL-7B more than its LoRA version, while LoRA is sufficient for larger models.
6.3 Ablation studies
Ablations show that SynSQL-2.5M, CoT synthesis, and increased training-data scale each contribute to model performance.
- Fine-tuning with SynSQL-2.5M improves performance across eight datasets, including gains of 9.0% on BIRD development and 12.9% on EHRSQL.
- Removing CoT solutions causes notable performance drops on nearly all datasets except EHRSQL.
- Model performance consistently improves as training data increases from 20% to 100%, with the upward trend persisting at the full dataset.
- The data augmentation comparison evaluates improvement from synthetic data using without-synthetic-data, with-synthetic-data, and Δ columns.
- CoT synthesis is preferred in 4,699 of 5,000 cases, or 93.98%, where SQL execution results changed after synthesis.
6.4 Comparison with Data Augmentation
The proposed data synthesis method outperforms three representative augmentation methods on Spider and BIRD development improvements.
- The proposed method achieves larger synthetic-data accuracy gains than existing augmentation methods on both Spider and BIRD development sets.
7 CONCLUSION
The paper presents a controllable synthesis framework, the SynSQL-2.5M dataset, and OmniSQL, whose evaluations show strong performance with fewer parameters.
- The framework decomposes text-to-SQL data synthesis into four sequential, controllable steps.
- SynSQL-2.5M contains 2.5 million high-quality samples, while OmniSQL is evaluated across nine benchmarks.
- OmniSQL substantially outperforms similarly sized baseline LLMs and matches state-of-the-art models despite using fewer parameters.
- Releasing SynSQL-2.5M and OmniSQL is intended to provide resources for further text-to-SQL research.