Source-linked AI summary
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, Dragomir Radev
TL;DR
Existing semantic-parsing datasets often permit memorization through repeated databases or query templates and lack broad coverage of complex, cross-domain SQL. Spider introduces a large human-annotated corpus and a database-disjoint task requiring generalization to unseen queries and schemas; models perform poorly, underscoring the challenge.
Problem
Prior datasets provide limited evaluation of generalization to unseen complex programs and databases because they are often small, database-overlapping, or restricted to simple SQL.
Method
Spider constructs a human-annotated corpus of complex questions and SQL queries over multiple-table databases, then defines database-disjoint semantic-parsing and text-to-SQL evaluation.
Results
12.4% exact matching accuracy is achieved by the best model in the database split setting, and database-split performance is much lower than example-split performance.
Takeaways & Limitations
Spider establishes a challenging benchmark for generalization to new databases and complex queries, with substantial room for improvement.
Takeaways & Limitations
Spider excludes ambiguous questions and questions requiring knowledge outside the database, leaving multi-turn clarification for future work.
Abstract
from arXiv · showhide
We present Spider, a large-scale, complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 college students. It consists of 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables, covering 138 different domains. We define a new complex and cross-domain semantic parsing and text-to-SQL task where different complex SQL queries and databases appear in train and test sets. In this way, the task requires the model to generalize well to both new SQL queries and new database schemas. Spider is distinct from most of the previous semantic parsing tasks because they all use a single database and the exact same programs in the train set and the test set. We experiment with various state-of-the-art models and the best model achieves only 12.4% exact matching accuracy on a database split setting. This shows that Spider presents a strong challenge for future research. Our dataset and task are publicly available at https://yale-lily.github.io/spider
1 Introduction
Spider addresses limitations in prior semantic-parsing datasets by introducing a large, complex, cross-domain corpus and a task requiring generalization to unseen queries and databases. State-of-the-art models achieve low performance, demonstrating the task’s difficulty.
- Motivation: Prior datasets either contain few complex programs with database overlap or many programs restricted to simple, single-table SQL.These limitations make it difficult to evaluate generalization to unseen complex programs and new domains.
- Dataset: Spider contains 200 multi-table databases, 10,181 questions, and 5,693 complex SQL queries created by 11 students.The corpus covers complex clauses such as joins and nested queries and required 1,000 man-hours of annotation.
- Task: The new task requires models to map questions and database schemas to unseen queries on new databases.Its split prevents database overlap between training and test sets, requiring generalization to both new programs and schemas.
- Results: 12.4% exact matching accuracy is achieved by the best model in the database split setting.Several state-of-the-art semantic-parsing models struggle with the task, leaving substantial room for improvement.
2 Related Work and Existing Datasets
Prior semantic-parsing datasets often allow models to exploit repeated database-specific query templates, while WikiSQL scales databases and examples by simplifying SQL. Spider targets both unseen queries and unseen databases with richer schemas and programs.
- Existing datasets: Many earlier datasets use one database for training and testing, so models can omit schemas and exploit database-specific templates.Question-based splits may place identical target queries with paraphrases in both training and test sets, inflating accuracy.
- Existing datasets: Complex-program datasets are small, whereas large datasets such as WikiSQL mainly contain simple SQL queries over single tables.This leaves limited evidence for generalization to unseen complex programs and new domains.
- Spider’s task: Spider requires models to generalize to unseen queries and unseen databases by providing database schemas as inputs.This design directly targets cross-domain semantic parsing rather than memorization of a fixed database.
3 Corpus Construction
Spider was constructed through multi-stage human annotation and review over diverse relational databases. Its protocol emphasizes broad SQL-pattern coverage, consistent labels, and clear questions, while excluding ambiguity and outside-database knowledge.
- Construction process: All questions and SQL queries were written and reviewed by 11 computer science students across five construction and quality-control stages.The process included database creation, annotation, SQL review, question review, paraphrasing, and final review.
- Database collection: The corpus combines databases collected from courses, online resources, DatabaseAnswers, and selected WikiSQL tables.The resulting databases use relational schemas with foreign keys and, where necessary, intersection tables linking related tables.
- Annotation protocol: Annotators created 20–50 natural questions and SQL labels per database without templates or scripts.They were asked to cover common components including joins, aggregations, set operations, predicates, and nested queries.
- Annotation protocol: The protocol standardizes equivalent SQL labels so similar questions receive consistent query patterns.This reduces label variation when multiple SQL queries produce the same result.
- Question design: Questions were designed to be clear and database-answerable, excluding vague questions and those requiring outside knowledge.The authors acknowledge that real-world database interfaces also contain ambiguity and identify multi-turn clarification as future work.
- Quality control: Reviewers checked question clarity, SQL-label correctness, grammatical naturalness, semantic alignment, and SQL execution.A final experienced annotator resolved uncertain cases and a script executed and parsed all SQL labels.
4 Dataset Statistics and Comparison
Spider combines complex SQL, multi-table databases, and broad domain coverage to evaluate generalization beyond prior text-to-SQL datasets.
- Spider includes complex SQL components, including about twice as many nested queries and ten times as many ORDER BY and GROUP BY components as prior datasets combined.
- Spider contains 200 distinct databases covering 138 domains, with each database averaging 27.6 columns and 8.8 foreign keys.
- Spider uses different databases for training and testing, evaluating generalization to new SQL queries, schemas, and domains.
5 Task Definition
Spider defines text-to-SQL evaluation across unseen complex queries, databases, and domains, while making assumptions that focus evaluation on SQL structure and schema understanding.
- The task tests models on different complex SQL queries and databases from new domains, rather than allowing memorization of training examples.
- Models are evaluated on SQL structures and columns, while gold condition values are supplied instead of being generated.
- Queries requiring outside knowledge, such as common-sense inference or mathematical calculation, are excluded from the dataset.
- Table and column names are assumed to be clear and self-contained, with database-specific shortcuts manually expanded.
6 Evaluation Metrics
Spider evaluates predictions through component-level, whole-query, execution-based, and difficulty-sensitive metrics, with safeguards for SQL equivalence.
- The evaluation includes Component Matching, Exact Matching, and Execution Accuracy, plus accuracy analyses by query difficulty.
- Component Matching: Component Matching decomposes SELECT, WHERE, GROUP BY, ORDER BY, and keyword predictions into sets and checks exact correspondence with gold components.
- Exact Matching: Exact Matching marks a query correct only when every SQL component matches, while set comparison avoids penalizing ordering differences.
- Execution Accuracy: Execution Accuracy uses supplied gold values to execute predictions, complementing exact matching because either metric can produce false evaluations in some cases.
- Difficulty: Queries are categorized as easy, medium, hard, or extra hard according to SQL components, selections, conditions, keywords, and nested operations.
7 Methods
The experiments adapt several semantic parsing models to Spider’s schema- and query-complexity demands, then report exact and component-level performance across hardness levels.
- The study adapts state-of-the-art semantic parsing models because Spider differs fundamentally from datasets such as GeoQuery and WikiSQL.
- Models receive database columns as input and have their column-selection space restricted to columns relevant to each question.
- Seq2Seq: The Seq2Seq family is tested with attention and copying variants, while schema-aware decoding masks vocabulary to current database tables and columns.
- Performance is summarized with exact matching in Table 2 and component-matching F1 scores in Table 3.
- SQLNet: SQLNet is extended with ORDER BY and GROUP BY modules, preserving its sketch-based slot-filling approach for SQL conditions.
- TypeSQL: TypeSQL uses extracted entity and number types to improve understanding of entities and numbers.
8 Experimental Results and Discussion
Spider experiments show that current models struggle with complex SQL and generalization to unseen databases, with performance worsening as schema complexity increases.
- Overall Performance: SQLNet and TypeSQL outperform Seq2Seq models by using SQL structure information, but cannot generate nested queries or EXCEPT and INTERSECT queries.Their fixed SQL structures constrain the range of outputs they can produce.
- Component Matching: WHERE clause prediction is the most difficult component, while column prediction accounts for the most errors.WHERE clauses often contain multiple columns and operations.
- Overall Performance: Overall model performance remains low, indicating that Spider is challenging and leaves substantial room for improvement.
- Example Split vs Database Split: All models perform much worse under database split than example split, revealing difficulty generalizing to new databases.TypeSQL drops the most despite strong example-split performance, and column selection degrades substantially under database split.
- Complexity of Database Schema: Exact matching accuracy decreases as the number of foreign keys increases.More foreign keys create more candidate tables and columns and require modeling table relationships for increasingly join-heavy queries.
9 Conclusion
The paper introduces Spider as a large, complex, cross-domain semantic parsing and text-to-SQL dataset and defines a challenging task based on it. Results with state-of-the-art models indicate substantial room for improvement.
- Spider is a large, complex, cross-domain semantic parsing and text-to-SQL dataset intended to benefit NLP and database research.
- Experimental results on several state-of-the-art models suggest substantial room for improvement on the task.