Source-linked AI summary
SParC: Cross-Domain Semantic Parsing in Context
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, Dragomir Radev
TL;DR
SParC addresses the limited evaluation of context-dependent text-to-SQL across complex, cross-domain interactions. It introduces a large expert-labeled dataset and evaluates adapted context-dependent models, whose best accuracy is 20.2% exact set matching across questions and degrades sharply in later turns. The results identify contextual information flow and unseen-domain generalization as central challenges, with SQL-format differences limiting direct comparison to ATIS.
Problem
Context-dependent text-to-SQL requires systems to use interaction history while generalizing across databases and domains, but prior benchmarks provide narrower coverage.
Method
SParC is an expert-labeled dataset of contextual questions and SQL queries across complex databases and domains, evaluated with adapted context-dependent text-to-SQL models.
Results
20.2% exact set matching accuracy was achieved by the best model across all questions, while context-aware models outperformed the context-agnostic baseline and later-turn performance degraded.
Takeaways & Limitations
SParC establishes a challenging benchmark for contextual SQL generation, highlighting the importance of better modeling of information flow across interaction turns.
Takeaways & Limitations
SQL formatting differences between SParC and ATIS make some percentages, including WHERE, JOIN, and Nested, not directly comparable.
Abstract
from arXiv · showhide
We present SParC, a dataset for cross-domainSemanticParsing inContext that consists of 4,298 coherent question sequences (12k+ individual questions annotated with SQL queries). It is obtained from controlled user interactions with 200 complex databases over 138 domains. We provide an in-depth analysis of SParC and show that it introduces new challenges compared to existing datasets. SParC demonstrates complex contextual dependencies, (2) has greater semantic diversity, and (3) requires generalization to unseen domains due to its cross-domain nature and the unseen databases at test time. We experiment with two state-of-the-art text-to-SQL models adapted to the context-dependent, cross-domain setup. The best model obtains an exact match accuracy of 20.2% over all questions and less than10% over all interaction sequences, indicating that the cross-domain setting and the con-textual phenomena of the dataset present significant challenges for future research. The dataset, baselines, and leaderboard are released at https://yale-lily.github.io/sparc.
1 Introduction
SParC addresses context-dependent text-to-SQL for thematically related question sequences, where users refer to, omit, refine, or substitute information from interaction history. It introduces a cross-domain dataset and shows that current models struggle, especially on later questions.
- Dataset: Figure 1 illustrates sequences in which each question queries a database, contributes to an interaction goal, and receives an annotated SQL query with contextual SQL segments underlined.The examples include follow-up questions such as asking for total capacity after identifying dorms with a TV lounge.
- Motivation: Real-world database users ask related question sequences to explore topics or achieve complex goals, reducing cognitive burden while making phrasing dependent on interaction history.Questions may refer to or omit entities and constraints, or introduce refinements, additions, and substitutions.
- Dataset: SParC contains 4,298 coherent question sequences and 12k+ questions paired with SQL queries across 200 complex databases in 138 domains.Its cross-domain design supports evaluation on unseen databases and domains.
- Results: 20.2% exact set matching accuracy was achieved by the best model across all questions, with accuracy falling from 38.6% in turn 1 to 1.1% in turns 4 and higher.The result indicates substantial difficulty in modeling contextual, cross-domain interactions.
2 Related Work
Prior context-dependent semantic-parsing datasets provide limited coverage, while SParC targets complex SQL generation across diverse databases and domains. Its evaluation setting requires generalization beyond the databases seen during training.
- Context-independent semantic parsing: Earlier semantic-parsing datasets were often small and single-domain, whereas newer neural approaches increasingly use large cross-domain text-to-SQL datasets.This shift expands domain coverage but does not by itself address context-dependent SQL generation.
- Context-dependent semantic parsing with SQL labels: ATIS is a prominent context-dependent text-to-SQL benchmark, but it focuses on flight planning and uses only one database, limiting its SQL logic and cross-domain generalization.SParC contrasts with this setting through many complex databases and domains.
- Context-dependent semantic parsing with denotations: SCONE and SequentialQA provide contextual meaning through world states or denotations rather than full logical-form annotations, and SequentialQA uses one Wikipedia table at a time.SParC instead supplies complex SQL labels over larger databases.
- Conversational QA and dialogue systems: Dialogue systems commonly use predefined ontologies and slot-value pairs, while SParC focuses on general SQL queries as a more expressive representation of meaning.The comparison concerns the representation used for language understanding in context.
3 Data Collection
SParC was built through guided, student-created interactions over Spider databases, followed by SQL annotation, execution checks, parsing checks, and review. The process was designed to preserve thematic coherence while encouraging diverse contextual relations and complex queries.
- Collection pipeline: SParC construction comprised four stages: selecting interaction goals, creating questions, annotating SQL representations, and reviewing.The stages organize the dataset-creation workflow.
- Interaction goals: Interaction goals were based on Spider questions, and annotators created inter-related questions to obtain the information required by each goal.The goals supplied thematic guidance for coherent sequences.
- Question creation: 15 college students with SQL experience created question sequences using four thematic relation types: refinement, theme-entity, theme-property, and answer refinement/theme.They explored databases and were encouraged to include related questions that did not directly answer the goal question.
- Question creation: Each sequence contained at least two questions and ended when the student had enough information to answer the interaction goal.Goals were divided among students, with one student annotating each goal.
- SQL annotation: Annotators translated their own questions into SQL, executed every query, and used the Spider protocol to select consistent patterns among equivalent queries.This procedure targeted executable and evaluation-consistent SQL labels.
- Review: Every example received at least one review, followed by executable-query checks, SQL-parser validation, and final expert review; 139 sequences were discarded at the final step.Reviewers corrected grammar, checked semantic alignment, and verified the annotation protocol.
4 Data Statistics and Analysis
SParC broadens context-dependent text-to-SQL evaluation through complex, cross-domain interactions with diverse SQL semantics. Its analysis shows dependencies spanning nearby turns, increasing query complexity across interaction turns, and substantial challenges for generalization.
- Data statistics: SParC contains 4,298 unique question sequences spanning 200 complex databases and 138 domains, with 12k+ questions annotated by SQL queries.The dataset is split by database, so each database appears in only one of the train, development, and test sets.
- Contextual dependencies: Most SParC contextual dependencies span three or fewer turns, while SQL-token sharing is stronger between nearby questions and increases in later turns.Later questions tend to narrow toward more specific needs, making context-reference resolution important.
- Query complexity: Nearly all SQL components become more frequent in later SParC turns, indicating larger logical-structure changes than in ATIS.In ATIS, keyword frequencies remain relatively stable because interactions commonly modify only WHERE conditions.
- Contextual linguistic phenomena: SParC interactions commonly preserve the same entity set while requesting another property, or refine constraints across turns.Theme-entity relations reuse FROM and WHERE clauses with different SELECT properties, whereas refinement changes a search constraint such as recency or rank.
- Semantic coverage: SParC covers all SQL components and has greater semantic coverage than ATIS, whose domain restriction yields fewer templates and a smaller vocabulary.ATIS lacks or rarely uses components including ORDER BY, HAVING, SET, GROUP BY, and aggregation.
- Cross-domain evaluation: Database-disjoint splitting requires models to generalize simultaneously to new SQL queries, databases, and domains.This cross-domain setting distinguishes SParC from ATIS, which contains one flight-booking database and domain.
5 Methods
The paper extends two text-to-SQL architectures to generate SQL from the current question, prior interaction history, and database schema in cross-domain settings.
- At turn i, the models generate SQL from the current question, all preceding questions, and the database schema.
- CD-Seq2Seq: CD-Seq2Seq adds a turn-level history encoder to a cross-domain Seq2Seq text-to-SQL model.It encodes utterances with recurrent networks and propagates contextual information through discourse states.
- CD-Seq2Seq: CD-Seq2Seq represents database columns by averaging embeddings of table-and-column-name sequences.Each header uses the form table name.column name before averaging token embeddings.
- CD-Seq2Seq: Its decoder attends to questions from previous turns and generates either SQL keywords or column headers.The number of attended previous turns is controlled by the hyperparameter η.
- SyntaxSQL-con: SyntaxSQL-con extends SyntaxSQLNet by supplying the decoder with the previous question as contextual information.Both current and previous questions are encoded with bi-LSTMs and injected into each syntax module.
- The authors release implementations of the baseline models for reproducibility.
6 Experiments
Experiments evaluate context-dependent text-to-SQL models using exact set matching over individual questions and complete interactions, revealing strong dependence on context and turn position.
- Evaluation Metrics: Question match requires every predicted SQL clause to be correct, while interaction match additionally requires every question in an interaction to match exactly.Clause-level set matching is used instead of string matching.
- Results: 20.2% question match and 5.2% interaction match are reported for SyntaxSQLNet with access to the previous question.Removing that access reduces performance to 16.9% question match and 1.1% interaction match.
- Results: Context-aware models significantly outperform the context-agnostic SyntaxSQLNet baseline.
- Results: SyntaxSQL-con has higher question match, whereas CD-Seq2Seq has higher interaction match.SyntaxSQL-con benefits from stronger first-turn text-to-SQL performance, while CD-Seq2Seq better incorporates information across the full history.
- Question Position: Both models perform worse as question turn increases, and SyntaxSQL-con's first-turn advantage disappears from the second turn onward.The results are consistent with difficulty modeling information flow and accumulated errors in later turns.
- SQL Difficulty: Questions become harder as interactions proceed, with more hard and extra hard questions appearing in later turns.Both models fail on most hard and extra hard questions.
- Thematic Relations: The models perform best on answer refinement/theme relations but poorly on refinement and theme-property relations.Explicitly stated theme entities can make answer-refinement questions less dependent on prior questions.
7 Conclusion
SParC is a large-scale cross-domain dataset of context-dependent questions paired with SQL representations, designed to test semantic parsing across diverse domains and dependencies.
- SParC contains context-dependent questions over databases in different domains, each annotated with a corresponding SQL representation.
- The dataset combines wide semantic coverage with diverse contextual dependencies and unseen-domain evaluation.
- Both evaluated context-dependent models degrade in later interaction turns, indicating a need for better context modeling.
A.1 Additional Baseline Model Details
The adapted baselines encode utterances and interaction history, attend over recent turns, and decode SQL keywords or column headers. A SQL segment-copying extension was also tested but did not significantly improve performance because of error propagation.
- Turn-level history encoder: The turn-level encoder uses a bi-LSTM for each utterance and a uni-directional LSTM to generate the interaction-level discourse state.The final utterance representation is passed to the interaction-level encoder.
- Schema representation: Column headers are represented by averaging word embeddings of table-name.column-name strings.A special dot token separates the table and column names.
- Context-aware decoding: The decoder attends over tokens from the five previous turns using learned relative position embeddings.Attention scores are normalized with softmax, and the context vector is a weighted sum of token and position representations.
- SQL decoding: The sequential decoder separately scores SQL keywords and column headers before producing the output distribution.Generated SQL elements include keywords such as select, where, group by, and order by.
- SQL segment copying: SQL segment copying extracted 3.9 segments per SQL on average but did not significantly improve performance because of error propagation.The extracted segments included SELECT, FROM, GROUP BY, ORDER BY, and WHERE conditions.
- Context integration: Adding history information to SyntaxSQLNet's column-number and column-value prediction modules incorporates the preceding question context.The previous question's LSTM hidden states are added as inputs to the modules.
A.2 Additional Data Examples
The paper provides additional SParC examples, including examples annotated with different thematic relations. Figure 5 colors entities, properties, constraints, and answers distinctly.
- Additional examples: Figure 4 provides additional examples from SParC.
- Thematic relations: Figure 5 presents a SParC example annotated with different thematic relations.Entities are purple, properties magenta, constraints red, and answers orange.