Source-linked AI summary
Improving Text-to-SQL Evaluation Methodology
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, Dragomir Radev
TL;DR
Text-to-SQL evaluations may overstate generalization because traditional splits reuse SQL queries across training and test, while datasets differ substantially from realistic human questions. The paper standardizes and expands datasets, introduces query-based evaluation, and finds that human-generated data requires greater complexity; it recommends evaluating across both split types and multiple datasets.
Problem
Current text-to-SQL evaluations provide an incomplete picture of system strengths and weaknesses, especially for generalization to new queries and realistic data.
Method
The paper compares human- and automatically generated questions, standardizes and corrects datasets, introduces Advising, and proposes query-based splits alongside question-based splits.
Results
Human-generated datasets require greater query complexity, while traditional question-based splits mainly test robustness to alternate wording for known queries and state-of-the-art systems struggle on query-based splits.
Takeaways & Limitations
Future evaluations should use multiple datasets containing joins and nesting and report performance on both question- and query-based splits.
Takeaways & Limitations
The authors could not compare with non-neural text-to-SQL systems because their code was unavailable and many require dataset-specific grammars or templates.
Abstract
from arXiv · showhide
To be informative, an evaluation must measure how well systems generalize to realistic unseen data. We identify limitations of and propose improvements to current evaluations of text-to-SQL systems. First, we compare human-generated and automatically generated questions, characterizing properties of queries necessary for real-world applications. To facilitate evaluation on multiple datasets, we release standardized and improved versions of seven existing datasets and one new text-to-SQL dataset. Second, we show that the current division of data into training and test sets measures robustness to variations in the way questions are asked, but only partially tests how well systems generalize to new queries; therefore, we propose a complementary dataset split for evaluation of future work. Finally, we demonstrate how the common practice of anonymizing variables during evaluation removes an important challenge of the task. Our observations highlight key difficulties, and our methodology enables effective measurement of future development.
1 Introduction
The paper argues that current text-to-SQL evaluations incompletely measure real-world generalization and introduces data, baselines, and evaluation changes to address these gaps.
- Contributions: The paper contributes a new challenging dataset, improved existing datasets, and a simple baseline system for future evaluation.These contributions are presented as improvements to current text-to-SQL methodology.
- Query complexity: Human-written questions require more complex SQL queries than automatically generated questions.The Advising dataset captures student questions that lead to particularly complex queries.
- Data splits: Traditional question-based splits allow test queries to recur in training with different entities or wording, mainly measuring expression robustness rather than composing new queries.The authors propose a template-based slot-filling baseline and a query-based split to expose this distinction.
- Data splits: The paper introduces a query-based split that prevents the same SQL query from appearing in multiple data partitions.It complements question-based evaluation by testing generalization to new queries.
- Dataset preparation: The authors apply extensive effort to standardize datasets and fix a range of errors.This supports consistent evaluation across datasets.
2 Related Work
Related work spans database and NLP approaches to mapping language into structured representations, while recent neural methods increasingly target SQL and address limited annotation resources.
- Database approaches: Database-community systems commonly use pattern matching, grammar-based techniques, or intermediate query representations.Many such systems rely on domain-specific resources and are not publicly available.
- NLP approaches: NLP research has studied semantic parsing to logical representations and has increasingly explored direct mappings from text to SQL.The paper situates its evaluation methodology across both traditions.
- Data resources: Automatically generated datasets use templates to create queries and language, while feedback-based learning alternates prediction with user ratings.These methods address the challenge of annotating sufficiently large question-query datasets.
3 Data
The paper standardizes and audits multiple text-to-SQL datasets, adds Advising, canonicalizes SQL, annotates variables, and removes spurious duplicate-query ambiguity.
- Dataset collection: The study analyzes multiple text-to-SQL datasets using a consistent SQL style.The collection includes ATIS, GeoQuery, Restaurants, Scholar, Academic, Yelp, IMDB, WikiSQL, and Advising.
- Dataset collection: Advising contains questions about fictional University of Michigan course records, collected from students and an EECS department Facebook page.The dataset targets realistic academic-advising questions.
- Quality control: The authors manually assess question-query pairs for answer accuracy and helpfulness, fixing or removing cases with low scores.At least two annotators scored each pair using separate accuracy and helpfulness scales.
- SQL canonicalization: The authors canonicalize SQL by ordering clauses, standardizing aliases, and normalizing capitalization and spacing.Unit tests and manual inspection confirmed that canonicalization preserved query meaning; 30 ATIS queries with ambiguous AND/OR mixing were manually fixed.
- Variable annotation: Variables are annotated through automatic extraction and manual review to support accurate anonymization despite ambiguous mappings.The process also identified missing, extra, inaccurate, and inconsistent constraints.
- Query deduplication: Duplicate queries are manually grouped into sets of equivalent questions, with multiple valid queries retained but only the first used thereafter.A second person inspected every group and ran the queries.
4 Evaluating on Multiple Datasets Is Necessary
Dataset size alone does not capture diversity or realism: human-generated and task-oriented datasets contain more joins and nesting than automatically generated collections, motivating multi-dataset evaluation.
- Motivation: Evaluating on multiple datasets is necessary because datasets differ in collection process, redundancy, and query complexity.Performance on one dataset may not reflect performance on another type of data.
- Dataset diversity: Dataset size is not the best indicator of diversity: WikiSQL has fifteen times as many question-query pairs as ATIS, yet ATIS has significantly more patterns.More than half of WikiSQL uses one dominant pattern.
- Query complexity: Human-generated datasets generally exhibit greater query complexity than automatically generated data, including nesting and multi-table joins.All human-generated datasets except Yelp show some nesting, and their average queries join more than one table.
- Query complexity: Task-oriented ATIS and Advising rank among the top three datasets on multiple complexity measures.Their air-travel and student-advising settings require joins and nesting.
- Implications: The largest available dataset does not provide the desired complexity for predicting performance on human-generated or task-oriented questions.The authors therefore recommend evaluating on datasets that test nesting and joins.
5 Current Data Splits Only Partially Probe Generalizability
Question-based splits can reward systems that retrieve known query templates and fill entity slots, so they only partially test generalization to unseen queries. The proposed query split separates anonymized SQL templates across train, development, and test sets, revealing broader difficulty and differences across datasets.
- 5 Current Data Splits Only Partially Probe Generalizability: Question-based splits can measure robustness to wording while leaving generalization to new SQL queries only partially tested.The standard split may place equivalent queries in training and test with different entities or phrasing.
- 5 Current Data Splits Only Partially Probe Generalizability: The query split assigns anonymized-equivalent queries, with all accompanying questions, exclusively to train, development, or test.This prevents the same SQL query from appearing in multiple sets.
- 5 Current Data Splits Only Partially Probe Generalizability: The template-based slot-filling baseline was competitive on question splits but cannot generalize beyond training queries and therefore fails on query-split data.Its predictions select a training SQL template and identify words to fill its slots.
- 5 Current Data Splits Only Partially Probe Generalizability: 70-100% accuracy on question-based splits could be obtained by selecting a training template and filling the correct slots in the template-based oracle condition.This result demonstrates how question splits can reward template retrieval and slot filling.
- 5 Current Data Splits Only Partially Probe Generalizability: Dataset differences reflect redundancy and question structure: near-1:1 question-to-query ratios make the two splits similar, whereas reducing redundancy does not improve query-split accuracy.Academic’s simple question format and compositional queries may explain its exceptional query-split performance, but the evidence is not conclusive.
- 5 Current Data Splits Only Partially Probe Generalizability: Query-split performance was lower across datasets and systems, and oracle entities did not remove this difficulty, while many seq2seq models retained some ability to generalize.Entity matching remains a nontrivial component, but unseen-query composition introduces additional difficulty.
6 Conclusion
The paper identifies dataset realism and evaluation splits as two central weaknesses in text-to-SQL research. It recommends broader dataset coverage and evaluation on both question- and query-based splits.
- Human-written datasets require query properties absent from large automatically generated query sets, revealing a gap in data realism.
- Traditional data splits overstate system generalizability by not fully testing composition of new queries.
- The authors fixed hundreds of dataset mistakes and homogenized SQL structures to support effective multi-domain experiments.
- Future evaluations should use multiple datasets, include joins and nesting, and report results on both question- and query-based splits.