Source-linked AI summary
Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions
Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, Dragomir Radev
TL;DR
Cross-domain context-dependent text-to-SQL must interpret follow-up questions using interaction history and varying database schemas. The paper edits previous SQL queries at the token level with utterance-table and table-aware components, achieving higher SParC accuracy than prior state-of-the-art systems. Its evaluation also indicates greater robustness to error propagation than segment copying.
Problem
Context-dependent text-to-SQL must handle follow-up questions that refer to interaction history while generalizing across complex database schemas.
Method
The model edits the previous SQL query at the token level and uses an utterance-table encoder with a table-aware decoder to incorporate utterances and schemas.
Results
7% question match accuracy and 11% interaction match accuracy improvements over the previous state of the art were reported on SParC.
Takeaways & Limitations
Editing previous queries is more robust to error propagation than copying segments, and performance increases as basic text-to-SQL generation quality improves.
Takeaways & Limitations
The predicted-query setting remains affected by error propagation from incorrect queries produced by the model itself.
Abstract
from arXiv · showhide
We focus on the cross-domain context-dependent text-to-SQL generation task. Based on the observation that adjacent natural language questions are often linguistically dependent and their corresponding SQL queries tend to overlap, we utilize the interaction history by editing the previous predicted query to improve the generation quality. Our editing mechanism views SQL as sequences and reuses generation results at the token level in a simple manner. It is flexible to change individual tokens and robust to error propagation. Furthermore, to deal with complex table structures in different domains, we employ an utterance-table encoder and a table-aware decoder to incorporate the context of the user utterance and the table schema. We evaluate our approach on the SParC dataset and demonstrate the benefit of editing compared with the state-of-the-art baselines which generate SQL from scratch. Our code is available at https://github.com/ryanzhumich/sparc_atis_pytorch.
1 Introduction
The paper addresses cross-domain, context-dependent text-to-SQL generation by editing previous queries at the token level while incorporating utterances and complex table schemas. On SParC, this approach improves question-match and interaction-match accuracy over the previous state of the art.
- Motivation: Users ask related questions across turns, often referring to history or omitting information they expect the system to retain.This makes interaction history important for interpreting follow-up questions and reproducing parts of earlier queries.
- Limitations of prior work: Prior work copied complete query segments using rule-based extraction, limiting reuse when only a few tokens changed.Its setting also did not explicitly address different database schemas because ATIS covered only flight booking.
- Approach: The proposed decoder edits the previous query token by token, allowing individual changes and improving robustness to error propagation.The model encodes the prior query as a token sequence and computes a switch for changing tokens.
- Approach: An utterance-table encoder jointly represents user utterances and column headers, while a table-aware decoder attends to both during SQL generation.These components target complex schemas across different domains.
- Results: 7% question match accuracy and 11% interaction match accuracy improvements were reported over the previous state of the art on SParC.SParC contains coherent question sequences with SQL annotations spanning 200 databases in 138 domains.
2 Cross-Domain Context-Depencent Semantic Parsing
The task maps each utterance to SQL in a multi-turn interaction while conditioning on interaction history and the schema of the current database. The cross-domain formulation therefore requires modeling sequential context together with tables and column headers.
- Datasets: SParC provides context-dependent cross-domain interactions over 200 databases, whereas ATIS is limited to a particular domain.The paper uses SParC as its main benchmark and reports ATIS results for direct comparison with prior work.
- Task formulation: Context-dependent semantic parsing generates each query from the current utterance and the preceding interaction history.The interaction is represented as a sequence of utterance-query pairs, unlike context-independent parsing of individual pairs.
- Model architecture: The architecture edits the previous query while attending to user utterances, the table schema, and the previously generated query.Figure 1 summarizes this architecture, which combines history-aware generation with schema conditioning.
- Task formulation: In the cross-domain setting, each interaction is grounded in a different relational database with multiple tables and column headers.Each column-header representation includes its table name and column name.
3 Methodology
The model combines utterance–table encoding, interaction history, and table-aware decoding to generate SQL by editing the previous query at token level. It uses attention over relevant utterances, table columns, and prior query tokens to support cross-domain context-dependent generation.
- Utterance-Table Encoder: The encoder jointly represents each user utterance and database schema through co-attention between utterance tokens and column headers.Column representations also model schema structure with self-attention and vary across utterances because the two embeddings depend on each other.
- Interaction Encoder with Turn Attention: An interaction-level LSTM encodes turn history, while turn attention adds relevant previous-utterance information to the current representation.The resulting context summarizes interaction history and the current query for decoder initialization.
- Table-aware Decoder: The table-aware decoder attends to both column headers and utterance tokens while generating SQL keywords or schema columns.Separate scoring layers handle SQL keywords and column headers, which supports changing schemas across cross-domain examples.
- Query Editing: Because consecutive queries overlap, the model edits the previous query using copy and insert operations rather than generating every query from scratch.As interactions progress, queries become longer but the number of new tokens remains small because more tokens overlap with the previous query.
- Query Editing: The editing mechanism encodes the previous query, attends to its tokens, and predicts a switch between copying a prior token and inserting a new token.A separate layer scores previous-query tokens and modifies the output distribution using the editing probability.
4 Related Work
Related work spans semantic parsing from natural language to formal representations, including SQL, and increasingly large cross-domain text-to-SQL datasets. However, most text-to-SQL systems remain context-independent, while context-dependent cross-domain parsing has received less attention and introduces schema and interaction challenges.
- Semantic Parsing: Semantic parsing maps natural language sentences to formal representations such as logical forms, lambda calculus, executable programs, and SQL queries.Early studies commonly used small datasets and a limited number of domains.
- Text-to-SQL: Large cross-domain datasets such as WikiSQL and Spider have supported neural text-to-SQL research, but most systems still map single-turn questions to queries.These approaches generally focus on context-independent semantic parsing.
- Context-dependent Parsing: Context-dependent semantic parsing has received comparatively less effort across datasets including ATIS, SpaceBook, SCONE, SequentialQA, SParC, and CoSQL.Prior work includes mapping utterances to semantic frames and then to SQL on ATIS.
- Context-dependent Parsing: CoSQL introduced a large cross-domain conversational text-to-SQL corpus whose diverse users and frequent interaction changes pose new challenges for text-to-SQL systems.Its dialogues simulate database querying with users and SQL-familiar experts under a Wizard-of-Oz setting.
- Editing-based Generation: Editing-based code-generation work typically edits retrieved training examples, whereas this approach edits the system’s previously predicted query in a context-dependent setting.The distinction concerns the source being edited: retrieved examples in prior work versus the preceding predicted query here.
5 Experimental Results
Experiments evaluate the model on Spider, SParC, and ATIS, showing gains from schema-aware encoding, query editing, and token-level reuse across turns. The model remains competitive as context length and question difficulty increase, while predicted-query editing is most effective with utterance-table BERT representations.
- Spider: The model achieves 36.4% dev and 32.9% test performance on Spider without interaction-level decoding or query editing.Because Spider questions are standalone, the experiment isolates the utterance-table encoder and table-aware decoder for context-independent cross-domain SQL generation.
- SParC: On SParC, the model without the previous query reaches 31.4% question matching accuracy and 14.7% interaction matching accuracy.It outperforms SyntaxSQL-con, while the table-utterance encoder, turn attention, and joint utterance-schema decoding improve results over CD-Seq2Seq by 10% question accuracy and 6% interaction accuracy.
- Turn and difficulty analysis: The model maintains 38%-48% accuracy on turns 2 and 3 and 20% accuracy on turn 4 or beyond, while later turns become harder.The analysis also reports that the model is more competitive on hard and extra hard questions.
- ATIS: On ATIS, the model achieves 36.2% dev and 43.9% test string accuracy, comparable to Suhr et al. (2018).The experiment applies query editing but reuses the prior utterance encoder because ATIS is single domain.
- Effect of query editing: Editing the gold query consistently improves question-match and interaction-match accuracy, confirming the benefit of editing when the previous query is correct.The study separately investigates query attention and sequence editing, including predicted-query editing under realistic error propagation.
- Effect of query editing: With utterance-table BERT embeddings, predicted-query editing increases question matching accuracy by 7% and interaction matching accuracy by 11%.Without BERT, predicted-query editing yields only around 1.5% improvement; BERT improves generation at each turn and reduces error propagation.
- Effect of query editing: Using only query attention reaches 42.7% question and 21.6% interaction matching accuracy, while adding query editing raises these to 47.2% and 29.5%.The ablation supports separate contributions from query attention and query editing.
6 Conclusions
The paper concludes that its simple editing-based encoder-decoder model benefits cross-domain context-dependent text-to-SQL generation. Editing is more robust to error propagation than segment copying, and performance improves with stronger non-editing generation.
- The editing-based encoder-decoder model addresses cross-domain context-dependent text-to-SQL generation.
- Editing is more robust to error propagation than copying segments.
- Performance increases when the basic text-to-SQL generation quality without editing is better.