Source-linked AI summary
Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, Yongbin Li
TL;DR
Existing text-to-SQL benchmarks underrepresent large, noisy database values, external knowledge, and SQL efficiency, limiting their realism for database interfaces. BIRD addresses this gap with a large cross-domain benchmark and evaluation framework, finding that current models remain substantially below human performance. The benchmark and analyses highlight database value comprehension and efficient SQL generation as central challenges for real-world text-to-SQL.
Problem
Existing benchmarks focus on database schemas with few database rows, leaving large, noisy values, external knowledge reasoning, and SQL efficiency underrepresented.
Method
BIRD provides 12,751 text-to-SQL examples over 95 databases totaling 33.4 GB across 37 professional domains, with external-knowledge evidence, double-blind annotation, and Valid Efficiency Score evaluation.
Results
Current models struggle to generalize on BIRD: GPT-4 achieves 54.89% execution accuracy versus 92.96% for humans, while analyses examine database, SQL, and model performance.
Takeaways & Limitations
BIRD shows that realistic text-to-SQL requires handling large and dirty values, external knowledge evidence, and SQL execution efficiency in addition to semantic parsing.
Takeaways & Limitations
SQLite makes the benchmark user-friendly but complicates precise Query Execution Plan-based efficiency computation and adaptation to different SQL syntaxes.
Abstract
from arXiv · showhide
Text-to-SQL parsing, which aims at converting natural language instructions into executable SQLs, has gained increasing attention in recent years. In particular, Codex and ChatGPT have shown impressive results in this task. However, most of the prevalent benchmarks, i.e., Spider, and WikiSQL, focus on database schema with few rows of database contents leaving the gap between academic study and real-world applications. To mitigate this gap, we present Bird, a big benchmark for large-scale database grounded in text-to-SQL tasks, containing 12,751 pairs of text-to-SQL data and 95 databases with a total size of 33.4 GB, spanning 37 professional domains. Our emphasis on database values highlights the new challenges of dirty database contents, external knowledge between NL questions and database contents, and SQL efficiency, particularly in the context of massive databases. To solve these problems, text-to-SQL models must feature database value comprehension in addition to semantic parsing. The experimental results demonstrate the significance of database values in generating accurate text-to-SQLs for big databases. Furthermore, even the most effective text-to-SQL models, i.e. ChatGPT, only achieves 40.08% in execution accuracy, which is still far from the human result of 92.96%, proving that challenges still stand. Besides, we also provide an efficiency analysis to offer insights into generating text-to-efficient-SQLs that are beneficial to industries. We believe that BIRD will contribute to advancing real-world applications of text-to-SQL research. The leaderboard and source code are available: https://bird-bench.github.io/.
1 Introduction
Existing text-to-SQL benchmarks show strong LLM performance but underrepresent large, noisy databases, external knowledge, and SQL efficiency. BIRD addresses these real-world challenges with a large cross-domain benchmark and finds a substantial gap between current models and humans.
- 85.3% execution accuracy on Spider from the latest SOTA parser motivates asking whether LLMs can already serve as database interfaces.
- BIRD identifies three underrepresented challenges: large and noisy database values, external knowledge reasoning, and efficient SQL execution.Large databases can require context compression, while efficient queries matter in practical applications.
- 12,751 examples over 95 databases totaling 33.4 GB across 37 professional domains form BIRD’s large-scale benchmark.The benchmark includes 80 databases for training and development and 15 additional databases for a hidden test set.
- 54.89% execution accuracy is achieved by GPT-4 on BIRD, compared with 92.96% human performance.The authors report that current models struggle to generalize to BIRD’s realistic settings.
2 Task Formulation & Annotations
BIRD formulates text-to-SQL as generating a SQL query from a natural-language question, database schema, and external knowledge evidence. The formulation represents databases through their columns and tables and allows a model parameterized by θ to produce the query.
- Text-to-SQL converts a natural-language question Q into a SQL query Y that retrieves relevant database information.
- The database is represented as D = ⟨C, T⟩, where C and T denote columns and tables.
- For complex database values, external knowledge evidence K is incorporated to improve models’ understanding of those values.
- The task is formulated as Y = f(Q, D, K | θ), with f representing a model or neural network parameterized by θ.
3 Dataset Construction
BIRD constructs its dataset from real-world relational databases and uses description files, external knowledge evidence, double-blind SQL annotation, and expert examination to improve annotation quality and value comprehension.
- BIRD obtains and processes databases from multiple sources to enrich real-world attributes despite the difficulty of collecting complex schemas and sufficient values.Earlier benchmarks often self-designed schemas and values, which may differ from real-world distributions.
- Database Description Files provide full schema names and value descriptions so annotators can understand abbreviated fields and mismatched question terms.
- External knowledge evidence covers numeric reasoning, domain knowledge, synonym knowledge, and value illustration for mapping natural-language instructions to database values.
- Double-blind SQL annotation has two independent annotators generate SQLs, compares execution results, and uses experts to resolve disagreements.Experts select a semantically equivalent and efficient SQL as ground truth and record external knowledge evidence.
- Expert examination verifies SQL validness and text-knowledge-SQL alignment to ensure each annotated pair is executable and returns a valid result.
4 Data Statistics
BIRD is a large-scale, cross-domain benchmark whose data and SQL statistics emphasize database-value reasoning, diverse domains, and complex queries.
- BIRD covers complex SQL functions, knowledge reasoning, and efficiency evaluation as a large-scale cross-domain benchmark.
- Table 1 compares benchmarks by SQL functions, external-knowledge requirements, and execution-efficiency evaluation.
- Question Statistics: 70.1% of questions require value illustrations, while domain knowledge, numeric computing, and synonym reasoning account for 23.6%, 24.5%, and 7.2%.
- Database Statistics: Database distributions span domains, sizes, and value types, with Donor the largest database at 4.5 GB and date-related values prevalent.
- SQL Statistics: BIRD SQL queries are analyzed by token count, JOIN count, keyword count, and 3-gram count to characterize complexity and diversity.
5 Evaluation Metrics
BIRD evaluates text-to-SQL parsers using execution accuracy and valid efficiency score, combining correctness with the efficiency of correct queries.
- BIRD uses execution accuracy and valid efficiency score to evaluate parsers on large real-world database values.
- Execution Accuracy: Execution accuracy is the proportion of examples where predicted and ground-truth SQLs produce identical executed results.
- Valid Efficiency Score: Valid efficiency score measures the efficiency of predicted SQL queries whose result sets align with the ground-truth results.
- Valid Efficiency Score: VES incorporates both execution-result accuracy and efficiency, rejecting queries that fail to retrieve the correct values.
- Valid Efficiency Score: BIRD primarily measures efficiency through running time while allowing relative comparisons across machine-status uncertainty.
6 Experiments
BIRD evaluates text-to-SQL models on large, value-rich databases and finds substantial challenges in execution accuracy, efficiency, external-knowledge grounding, and fine-grained reasoning.
- Baseline Models: GPT-4 surpasses the baseline language models, while DIN-SQL + GPT-4 achieves a new state-of-the-art result using value sampling, few-shot demonstrations, and self-correction.The models still lag behind human capabilities on BIRD.
- Execution Accuracy Analysis: Models with higher EX can more possibly achieve higher VES because accurate result prediction is prerequisite to attaining a higher efficiency score.VES reflects the practical purpose of producing correct and efficient SQL.
- Efficiency Analysis: 87.3% time-saving is achieved by configuring indexes within the database.BIRD also introduces Chat With Database, which uses global SQL queries to expose data types and distributions.
- Knowledge Evidence Analysis: External knowledge evidence produces clear improvements across difficulty levels, showing that database values are important for realistic text-to-SQL.ChatGPT with chain-of-thought can perform better through multi-step reasoning, but adding external evidence to its chain-of-thought version can yield limited or declining performance.
- More Analysis: GPT-4 performs best across fine-grained categories, but ranking and numerical computing remain notably weaker than domain knowledge, synonym detection, and value illustration.The analysis associates the stronger categories with linguistic training and reasoning capabilities from pretraining.
- More Analysis: Wrong schema linking accounts for 41.6% of ChatGPT errors, followed by misunderstanding database content at 40.8% and knowledge evidence at 17.6%.The analysis covers 500 randomly sampled error cases and identifies schema linking and database understanding as major obstacles.
7 Related Work
BIRD extends text-to-SQL benchmarking toward real-world database values by combining large-scale, cross-domain coverage with knowledge-intensive analysis.
- Related Work: BIRD is the first large-scale benchmark to incorporate real-world features while emphasizing database values.Earlier datasets generally emphasized schema, remained single-domain, or used smaller collections of value-rich databases.
8 Limitation and Future work
The paper identifies resource-intensive annotation and SQLite-specific efficiency limitations, and proposes broader database-system support as future work.
- Limitation and Future work: Double-blind SQL annotation produces high-quality labels but is resource-intensive, motivating human-computer interaction approaches to reduce annotation effort.The proposed direction includes using advanced AI systems such as GPT-4 for part of the annotation process while maintaining data quality.
- Limitation and Future work: SQLite makes BIRD user-friendly but complicates precise query execution-plan analysis and adaptation to other SQL syntaxes.Future versions are planned for PostgreSQL and MySQL.
9 Conclusion
BIRD introduces a large-scale, cross-domain text-to-SQL benchmark centered on database values and three real-world challenges: dirty values, external knowledge, and SQL efficiency.
- BIRD targets large and dirty database values, external knowledge evidence, and SQL execution efficiency in text-to-SQL.
A Datasheet for Datasets
BIRD is a large text-to-SQL dataset designed to assess LLMs on real-world databases containing substantial database values. It documents the dataset’s composition, splits, annotation process, quality controls, licensing, and usage constraints.
- Purpose: BIRD targets the gap between schema-focused benchmarks and real-world text-to-SQL tasks involving large database values and external knowledge.
- Dataset composition: BIRD contains 12,751 natural-language questions, external-knowledge evidence sentences, processed databases, database-description CSV folders, and ground-truth SQL queries.
- Dataset splits: The dataset comprises 9,428 training instances, 1,534 development instances, and 1,789 concealed test instances.Training and development data are public, while the test data remain hidden for evaluation.
- Quality and limitations: The authors report rigorous double-blind annotation but acknowledge that complex datasets cannot be entirely free of errors and will continue to be improved.
- Collection and annotation: Two independent crowdworker teams annotate questions and SQLs, with English-native question annotators and database engineers or database students annotating SQLs.Four PhD students and two MS students create database-description files.
- Licensing and use: BIRD is distributed under CC BY-NC 4.0 because the authors are concerned that extensive database values could enable inappropriate commercial use.The databases themselves are collected under appropriate licenses.
B.4 Experiment Details
The experiments compare fine-tuned and in-context text-to-SQL models, then examine knowledge grounding, question types, efficiency-oriented SQL generation, and ChatGPT error patterns.
- Model settings: Fine-tuned T5 generates SQL end-to-end from questions concatenated with serialized database schemas.The passage characterizes T5 as a strong pre-trained language model for semantic parsing tasks, including text-to-SQL.
- Model settings: Codex and ChatGPT generate SQL through in-context learning from programming-based prompts without additional training.The experiments use the Azure OpenAI API for these model variants.
- Knowledge grounding: Naively concatenating external knowledge evidence with questions and database schemas produces a significant improvement.The evidence sentences are expert-annotated external knowledge associated with expected SQLs.
- Question types: BIRD questions include Fundamental Type questions comparable to existing benchmarks and Reasoning Type questions requiring external-knowledge grounding.
- SQL efficiency: Two-stage optimization and embodied databases are presented as strategies for generating more efficient SQLs.The examples indicate that both strategies can help semantic parsers produce more efficient SQLs.
- Error analysis: Figure 11 analyzes four major types of errors made by ChatGPT, including cases described as shortcuts for better presentation.
B.7 Evaluation Details
The evaluation details define efficiency metrics, control measurement variability, describe annotation and licensing procedures, and characterize SQL functions, keywords, and prior text-to-SQL approaches.
- Evaluation and annotation: Annotation uses HashSet rather than List to compare final results because DISTINCT-related ambiguity can produce mismatches between annotators.The ambiguity concerns whether DISTINCT should denote unique values generally or only when explicitly requested.
- Efficiency metrics: Efficiency E measures execution time, with lower E indicating faster and more efficient SQL execution.E is bounded within (ϵ, 30s), where ϵ prevents floating-point overflow.
- Efficiency metrics: Efficiency ratio R normalizes predicted-query efficiency against human-annotated SQL efficiency, with higher R indicating higher efficiency.The metric averages results over 100 runs after filtering outliers to reduce machine-status effects.
- Measurement procedure: VES measurements average 100 runs on the same CPU and remove outliers using thresholds at mean ± 3×standard_deviation.The reported standard deviations after 10 trials are 0.043 on the development set and 0.025 on the test set.
- Human performance: Human-performance collection divides data into 10 batches, using the first 8 for public training and development data and the remaining 2 for testing.Experts track errors, and early annotation batches serve as a learning process for SQL annotators.
- SQL coverage: BIRD covers window, date, conversion, mathematical, and string SQL functions, alongside joins, clauses, aggregation, scalar, comparison, and computing keywords.Examples include OVER(), JULIANDAY(), CAST(), ROUND(), and SUBSTR().
- Related evaluation context: Cross-domain text-to-SQL parsers generally encode questions and schemas before decoding SQL, while SQL-efficiency research emphasizes optimization and index selection.