Source-linked AI summary
KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers
Chia-Hsuan Lee, Oleksandr Polozov, Matthew Richardson
TL;DR
Text-to-SQL parsers still struggle to generalize robustly to unseen, real-life domains. The paper introduces KaggleDBQA and a few-shot evaluation setting with database documentation, finding that real-world databases remain challenging even when domain knowledge is added.
Problem
Zero-shot text-to-SQL parsers lack the robustness needed for deployment across unseen real-life application domains.
Method
The paper introduces KaggleDBQA and evaluates parsers with unrestricted questions, in-domain examples, and database documentation containing column and value descriptions.
Results
26.77% accuracy: baselines still struggle on KaggleDBQA after adding database documentation, while 13.56% accuracy is achieved on its expressible questions by state-of-the-art parsers.
Takeaways & Limitations
KaggleDBQA supports more realistic evaluation of text-to-SQL parsers and highlights the need for better use of documentation and domain-adaptation techniques.
Takeaways & Limitations
Deployment requires safeguards because parser answers may be incorrect, particularly when used for decision making.
Abstract
from arXiv · showhide
The goal of database question answering is to enable natural language querying of real-life relational databases in diverse application domains. Recently, large-scale datasets such as Spider and WikiSQL facilitated novel modeling techniques for text-to-SQL parsing, improving zero-shot generalization to unseen databases. In this work, we examine the challenges that still prevent these techniques from practical deployment. First, we present KaggleDBQA, a new cross-domain evaluation dataset of real Web databases, with domain-specific data types, original formatting, and unrestricted questions. Second, we re-examine the choice of evaluation tasks for text-to-SQL parsers as applied in real-life settings. Finally, we augment our in-domain evaluation task with database documentation, a naturally occurring source of implicit domain knowledge. We show that KaggleDBQA presents a challenge to state-of-the-art zero-shot parsers but a more realistic evaluation setting and creative use of associated database documentation boosts their accuracy by over 13.2%, doubling their performance.
1 Introduction
Existing text-to-SQL benchmarks do not fully reflect deployment conditions: real databases use obscure schemas, real questions are less schema-aligned, and standard evaluation omits in-domain knowledge. KaggleDBQA addresses these gaps with realistic databases and questions, documentation, and few-shot evaluation, while exposing substantial remaining difficulty.
- Text-to-SQL converts natural-language questions into SQL queries over relational databases, enabling natural-language interfaces for diverse applications.
- Spider and WikiSQL encourage zero-shot generalization, but their preprocessing and question-authoring practices create mismatches with real deployment conditions.Spider and WikiSQL use normalized or human-readable schemas, while their questions often mention database columns directly.
- KaggleDBQA introduces real Kaggle databases with abbreviated or obscure names, domain-specific values, minimal preprocessing, and unrestricted questions.Its annotators were not shown original column names or given task priming.
- 13.56% accuracy was achieved on the 272 collected questions that remained expressible for state-of-the-art parsers.One-third of the 400 collected questions were out of scope for those parsers.
- Database documentation and approximately 10 in-domain question-SQL examples define a more realistic few-shot evaluation setting.Documentation includes column and table descriptions, categorical-value descriptions, and SQL examples; augmenting RAT-SQL raised accuracy from 13.56% to 26.77%.
2 Related Work
Related work established cross-domain and zero-shot text-to-SQL evaluation, while newer research questioned whether benchmark construction supports practical generalization. KaggleDBQA builds on these directions by using real data, more natural questions, domain documentation, and few-shot adaptation.
- WikiSQL and Spider established cross-domain evaluation requiring generalization to unseen database domains, supporting rapid development of domain-conditioned parsers.The related work identifies RAT-SQL and IRNet as examples of parsers designed to work out of the box across domains.
- Recent work questioned whether existing datasets generalize well because questions can artificially overlap with table vocabulary.Spider-Realistic addresses this issue by rewriting questions to avoid overlapping terms.
- Other studies highlighted the gap between academic datasets and practical usability, motivating evaluation with real data.KaggleDBQA uses real data found on Kaggle that has been used for competitions or other analyses.
- Description-based resources have been used for zero-shot learning in dialogue state tracking, entity linking, and task transfer.KaggleDBQA similarly includes documentation written for human consumption as a natural source of domain knowledge.
3 KaggleDBQA: A Real World Dataset
KaggleDBQA is designed to reflect real-world text-to-SQL use through unnormalized databases, natural questions, documentation, and a few-shot evaluation setting. Its questions have low schema-term overlap and its SQL queries are more complex than Spider’s, creating a challenging benchmark for practical parser evaluation.
- Database Collection: KaggleDBQA uses real-world databases from Kaggle, preserving domain-specific content and formatting rather than normalizing schemas.The selected databases contained SQLite data, permissive licenses, and documentation describing tables and columns.
- Question Collection: 400 questions were written by 20 annotators across eight databases, using natural user interests rather than requiring SQL knowledge.Annotators saw descriptions and sampled rows, while independent SQL experts supplied equivalent queries.
- Question Collection: 272 questions remained for SOTA evaluation after one-third of collected questions were excluded as yes/no, percentage, temporal, or unexpressible in SQL.The excluded questions were outside the evaluation scope for current text-to-SQL models.
- Database Documentation: Database documentation supplies table, column, categorical-value, and query descriptions, with column descriptions used to address schema linking.The descriptions can also contain information relevant to value linking, although those uses are left for future work.
- Few-shot Evaluation Setting: The primary evaluation is few-shot: 30% of each domain supports adaptation with examples and documentation, while the remaining 70% is evaluated.The paper also reports standard zero-shot results on the same evaluation portion for comparability.
- Dataset Analysis: KaggleDBQA has low column mention percentage, low question-column-name overlap, and more complex SQL queries than Spider’s dev set.Its queries contain significantly more hard and extra-hard examples, while description matching shows a trade-off between correct matches and false positives as n-gram size changes.
4 Experiments
Experiments evaluate Spider-trained parsers on KaggleDBQA, then test few-shot fine-tuning, database documentation, adaptation, and schema normalization. Documentation and adaptation substantially improve accuracy, while remaining errors center on schema and constraint interpretation.
- Baseline Results: Spider-trained EditSQL and RAT-SQL are evaluated on KaggleDBQA, with RAT-SQL selected for subsequent experiments.RAT-SQL is identified as the best-performing parser for the further KaggleDBQA experiments.
- Baseline Results: Both parsers perform significantly worse on KaggleDBQA than on Spider under zero-shot evaluation.KaggleDBQA also contains fewer column mentions and more complex SQL than Spider.
- Few-Shot Setting: 13.56% to 17.96%: fine-tuning on a small amount of in-domain data dramatically increases overall accuracy.The few-shot setting fine-tunes on 30% in-domain data, while zero-shot results are also reported for comparison.
- Leveraging Database Documentation: 17.96% to 26.77%: adaptation with column descriptions produces a large accuracy gain, with adaptation and descriptions each contributing approximately half.Descriptions provide no benefit without adaptation, consistent with a mismatch between artificial training descriptions and real evaluation descriptions.
- Leveraging Database Documentation: 13.56% to 26.77%: incorporating in-domain examples and database documentation nearly doubles model accuracy on KaggleDBQA.The documentation includes column descriptions, and the experiments show that descriptions are critical to leveraging in-domain knowledge.
- Column Normalization: Normalization and descriptions provide complementary knowledge augmentation, jointly improving accuracy by 5% on schema-normalized KaggleDBQA.Normalization clarifies obscure column names, while descriptions contribute after adaptation mitigates train-test mismatch.
5 Conclusion & Future Work
KaggleDBQA offers a more realistic evaluation regime and dataset for text-to-SQL systems, while remaining challenging even after adding database documentation.
- KaggleDBQA bridges academic and industrial evaluation by combining more realistic database distributions with in-domain knowledge.
- 26.77% accuracy remains the baseline performance after adding database documentation, indicating difficulty on harder questions.
- The authors identify better documentation use, modeling, and domain adaptation as directions for advancing text-to-SQL performance.
Ethical Considerations
The dataset collection followed consent and licensing procedures, limited personal-data retention, and documented scope and deployment safeguards.
- Annotators consented before labeling and received a $20 e-gift card for approximately one hour of work.
- Databases were individually reviewed to ensure proper licensing for redistribution.
- Annotator-identifying information was minimized, with email addresses deleted after compensation and annotation associations removed before analysis or distribution.
- KaggleDBQA contains only English questions and databases, so multilingual evaluation requires translation.
- Deployments should use safeguards because parser answers may be incorrect, particularly in decision-making contexts.
A.1 Evaluation on Full Testing Data
The appendix reports zero-shot testing and out-of-domain adaptation results on the complete testing set, contrasting with an earlier partial-data evaluation.
- The zero-shot testing and out-of-domain adaptation results are reported in Table 10.
- Unlike Table 4, these results use the full testing-data set.
- The appendix therefore distinguishes full-set evaluation from the earlier Table 4 evaluation.
A.2.1 Example Page of User Instructions
The example user-instruction pages present task guidance together with database context and sampled content.
- Each user receives two HTML files containing different task instructions.
- The files include a database overview, table names, and column descriptions.
- Each file also provides ten sampled database rows as content context.
A.2.2 Question Types
KaggleDBQA permits unrestricted natural-language questions while applying annotation guidelines for categorical, free-form, duplicate, and identifier-valued columns. The experiments use RAT-SQL with BERT-Large and report exact-match accuracy averaged across three runs.
- Question distribution: Annotators could write unrestricted questions, including types absent from Spider training data; these difficult types remain in the dataset but are excluded from official evaluation.The authors retain them for future work on such questions.
- Annotation guidelines: Categorical columns use “=” with database values, whereas free-form text uses “LIKE” with a term from the question.The examples illustrate equality filtering for categorical values and substring matching for text such as team names.
- Annotation guidelines: When an ID column has a paired name realization, annotators return the ID unless the user explicitly asks for the name.The guideline applies to pairs such as state_code and state.
- Annotation guidelines: DISTINCT is used when duplicate rows could produce an incorrect result or when the user explicitly requests unique entries.Models cannot generally anticipate this issue without encoding database content.
- Experimental setup: Experiments use the official RAT-SQL implementation with Google’s pre-trained BERT-Large and report exact-match accuracy averaged over three random-seed runs.The implementation also lowers BERT’s learning rate and increases transformer dropout for regularization.