Source-linked AI summary
CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases
Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter S Lasecki, Dragomir Radev
TL;DR
CoSQL addresses the limited support of prior text-to-SQL datasets for interactive, cross-domain database querying with clarification and unanswerable questions. It introduces a large Wizard-of-Oz corpus and evaluates SQL-grounded state tracking, response generation, and dialogue-act prediction, finding substantial challenges for baseline systems. The dataset also leaves effective clarification of ambiguous and non-SQL questions for future work.
Problem
Existing conversational text-to-SQL corpora assume all user questions map to SQL and omit system responses, limiting coverage of clarification, verification, and unanswerable requests.
Method
The paper collects a cross-domain Wizard-of-Oz corpus and defines tasks for SQL-grounded dialogue state tracking, result-based response generation, and user dialogue-act prediction.
Results
Baseline performance indicates substantial difficulty: dialogue state tracking remains below 16% question-level and 3% interaction-level accuracy, while important INFER SQL and AMBIGUOUS dialogue acts have F-scores around 10%.
Takeaways & Limitations
CoSQL provides a benchmark for general-purpose database dialogue across unseen domains, with language and discourse diversity that exposes open problems for future research.
Takeaways & Limitations
The paper does not evaluate how systems effectively clarify ambiguous questions or guide users toward answerable questions, including cases requiring non-SQL reasoning.
Abstract
from arXiv · showhide
We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simulates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the expert describes the SQL and execution results to the user, hence maintaining a natural interaction flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets:(1) the dialogue states are grounded in SQL, a domain-independent executable representation, instead of domain-specific slot-value pairs, and (2) because testing is done on unseen databases, success requires generalizing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research. The dataset, baselines, and leaderboard will be released at https://yale-lily.github.io/cosql.
1 Introduction
CoSQL addresses the need for conversational database querying that can handle context, clarification, verification, and unanswerable questions. It introduces a cross-domain corpus and three tasks for modeling SQL-grounded dialogue interactions.
- Motivation: Existing NL query corpora assume questions map to SQL and often omit system responses, despite the need for interactive clarification and result verification.Conversational systems must process sequential, interrelated requests and respond to ambiguous or unanswerable questions.
- Motivation: Dialogue-based NLIDB systems must detect SQL-answerable questions, produce executable queries, explain results, and handle unanswerable requests.
- Corpus and Tasks: CoSQL contains 3,007 dialogues, over 30k turns, 10k expert-labeled SQL queries, and 200 complex databases across 138 domains.
- Corpus and Tasks: Its cross-domain split places each database in only one partition, requiring generalization to unseen domains without additional annotation.
- Corpus and Tasks: CoSQL introduces SQL-grounded state tracking, natural-language response generation from executed queries and results, and user dialogue-act prediction.
2 Related Work
CoSQL extends text-to-SQL dialogue research beyond isolated questions and domain-specific task ontologies. Its Wizard-of-Oz interactions capture clarification, unanswerable requests, and natural-language explanations of SQL results.
- Text-to-SQL Generation: Most prior text-to-SQL work converts a single complex question into SQL, while fewer datasets address context-dependent queries.
- Conversational Data: CoSQL uses two-party Wizard-of-Oz interactions to cover diverse conversational database-querying behaviors, including clarification and unanswerable questions.
- Task-Oriented Dialogue: Unlike slot-based task-oriented systems, CoSQL uses SQL as a domain-independent intent representation for general-purpose database exploration.
- Response Generation: CoSQL responses present execution results while describing the SQL interpretation so users can understand and verify system actions.
- Response Generation: Response generation must preserve SQL logic in natural-language descriptions of structured queries and result tables.
3 Data Collection
CoSQL was collected through Wizard-of-Oz dialogues between database users and SQL experts, with guided goals, annotated dialogue acts, executed queries, and expert-written responses. Review and quality-control procedures supported data consistency and interaction quality.
- Collection Setup: The collection paired AMT workers as database users with 25 SQL-proficient students serving as experts in an online chat interface.
- Collection Setup: Reference goals based on medium, hard, and extra-hard Spider questions guided users toward meaningful, interrelated exploration of complex databases.
- User Procedure: Users first examined database tables, then asked related questions guided by a revealed reference goal.
- Expert Procedure: Ambiguous, SQL-unanswerable, or inference-dependent questions prompted clarification or rephrasing rather than direct SQL translation.
- Expert Procedure: Experts labeled user and response dialogue acts, wrote SQL for answerable questions, executed it, and described results in natural language.
- Quality Control: Multi-pass review checked dialogue-act annotations, SQL-question alignment, response completeness, and dialogue diversity before rejecting unsuitable sessions.
4 Data Statistics and Analysis
CoSQL provides a large, diverse benchmark for conversational database querying, spanning many complex databases and including varied dialogue acts, SQL structures, and turn-level semantic patterns. Its analyses emphasize exploratory interactions and differences from context-dependent text-to-SQL data.
- Data statistics: 31,148 total turns span more than 200 complex databases, with approximately 80% of dialogues containing at least 8 turns.The average turn contains 11.21 tokens, and 11,039 user utterances are convertible to SQL.
- Dialogue act distribution: Approximately 40% of non-dominant user dialogue acts are AMBIGUOUS, highlighting the need for clarification during database querying.INFORM SQL and THANK YOU are the most common dialogue acts; about 20% of the ambiguous subgroup is INFER SQL.
- Semantic complexity: CoSQL SQL queries cover common keywords and complicated syntactic structures such as nesting.The analysis counts occurrences of different SQL keywords across queries corresponding to the questions.
- Semantic changes by turns: CoSQL keyword frequencies generally fluctuate little across turns, whereas almost all SParC SQL components increase as question turns progress.WHERE occurs slightly more often in CoSQL, while other keyword frequencies are generally lower than in SParC.
- Semantic changes by turns: The differing keyword trends suggest that CoSQL dialogues are more exploratory, with users switching focus instead of consistently building on previous questions.In SParC, the stronger increase across turns is consistent with greater interdependency between questions.
- Cross domain: The dialogues are split by database into train, development, and test sets using a 7:1:2 ratio.This is the same split ratio used for SParC and Spider.
5 Tasks and Models
CoSQL frames conversational database querying as a pipeline spanning SQL grounding, response generation, and user dialogue-act prediction. The tasks address executable SQL tracking, interpretable result descriptions, and recognition of questions requiring special handling.
- CoSQL systems determine whether user questions can be answered by SQL, translate answerable questions into queries, execute them, and return results.
- CoSQL excludes evaluation of effective system clarification and guidance for questions that are ambiguous, unanswerable, or require non-SQL logical reasoning.Questions answerable through other logical reasoning are excluded from the task design.
- SQL-grounded dialogue state tracking: SQL-grounded dialogue state tracking predicts the SQL query for each INFORM SQL utterance using interaction context and the database schema.Gold SQL queries from previous turns are unavailable, unlike traditional dialogue-state-tracking settings.
- SQL-grounded dialogue state tracking: The tracking task uses context-dependent CD-Seq2Seq and SyntaxSQL-con baselines, both applied without task-specific changes.CD-Seq2Seq copies segments of previous generated SQL and encodes schema information; SyntaxSQL-con uses a SQL-specific syntax-tree model with previous questions.
- Response generation: Response generation produces a natural-language description of the SQL query and execution result while preserving logical consistency, naturalness, and grammatical correctness.The baselines are template-based, vanilla attention Seq2Seq, and pointer-generator models.
- User dialogue act prediction: User dialogue-act prediction identifies whether a question maps to SQL or requires a special action such as clarification or another response.The task evaluates Majority and TBCNN-pair baselines; system dialogue acts are not separately predicted because they are defined by user acts.
6 Results and Discussion
CoSQL is difficult across its evaluated tasks: dialogue-state tracking performs poorly, response-generation metrics disagree, and important ambiguous or inferential dialogue acts remain hard to predict. These results leave substantial room for improvement.
- SQL-grounded dialogue state tracking: Less than 16% question-level accuracy and less than 3% interaction-level accuracy were achieved by both DST baselines on CoSQL.Both CD-Seq2Seq and SyntaxSQL-con performed significantly worse on CoSQL than on SParC.
- SQL-grounded dialogue state tracking: CoSQL DST is harder than SParC, potentially because of more diverse users, ambiguous questions, and more complex intent switches.
- Response generation: Human evaluation and BLEU scores do not completely agree for response generation.The template-based method performs better in human evaluation, whereas neural models achieve better BLEU scores.
- Response generation: Template-based responses have logic correctness rates of only around 40%, while neural models struggle to preserve SQL logic despite stronger BLEU scores.Pointer-generator outperforms basic Seq2Seq in both BLEU and human evaluation, but all methods have low logic-correctness performance.
- User dialogue act prediction: About 40% of user questions cannot be directly converted into SQL, while TBCNN predicts around 85% of user intents but achieves around 10% F-scores on INFER SQL and AMBIGUOUS.Most correct TBCNN predictions concern simpler acts such as INFORM SQL, THANK YOU, and GOODBYE.
7 Conclusion and Future Work
CoSQL is presented as a large-scale cross-domain conversational text-to-SQL corpus whose diverse language and discourse expose open research problems. Its future-work agenda includes handling ambiguity, unanswerable questions, and answers requiring non-SQL reasoning.
- CoSQL is a large-scale cross-domain conversational text-to-SQL corpus collected under a Wizard-of-Oz setup.The authors characterize its language and discourse diversity and cross-domain setting as open problems for future research.
- The dataset includes ambiguous and unanswerable questions, but the work does not study effective clarification or guidance toward answerable questions.
- Questions requiring logical reasoning beyond direct SQL answers are left for future research.
A Appendices
The appendices provide supplementary descriptions, implementation details, response guides, dialogue examples, and annotation interfaces.
- The appendices describe dialogue actions, baseline modifications and hyperparameters, system response guides, additional dialogue examples, and annotation interfaces.The interfaces cover both database users and SQL experts.
A.1 Description of Dialog Acts
CoSQL defines dialogue acts for both database users and SQL experts, distinguishing answerable SQL requests from inference, ambiguity, clarification, and unanswerable questions.
- User dialogue acts: User acts include INFORM SQL for requests answerable by SQL, INFER SQL for SQL plus human inference, and AMBIGUOUS for unclear intent.INFER SQL covers questions such as yes/no judgments or ordinal answers that SQL cannot directly return.
- User dialogue acts: Users can also affirm or negate system statements, indicate unrelated or incomprehensible questions, and signal routine greetings, thanks, or farewells.
- System dialogue acts: CANNOT ANSWER marks questions that cannot be easily answered by SQL, while CANNOT UNDERSTAND asks users to rephrase questions the system cannot understand.
- System dialogue acts: System acts include CONFIRM SQL, which describes the SQL and result table while asking users to verify the system’s interpretation.
- System dialogue acts: For ambiguity or unsupported requests, the system uses CLARIFY, REJECT, or REQUEST MORE, alongside routine greeting, apology, welcome, and farewell acts.
A.2 Modifications and Hyperparameters for Baselines
The baselines combine unchanged models, task-specific modifications, template matching, and recurrent generation architectures to address CoSQL’s prediction and response tasks.
- Baseline settings: CD-Seq2Seq and SyntaxSQL-con use the same settings as their SParC versions without changes.
- Baseline settings: The template-based system matches SQL patterns without values, column names, and table names, then fills slots shared by SQL queries and responses.Templates cover frequent training-set cases and preserve one-to-one mappings between table, column, and value slots.
- Baseline settings: Seq2Seq uses 128-dimensional embeddings, 256-dimensional hidden states, bidirectional encoding, attention, greedy decoding, and validation-based early stopping.Training uses batch size 32, gradient-norm clipping at 2.0, and patience of 5.
- Baseline settings: Pointer-generator follows the Seq2Seq settings and adds a copy mechanism during training and testing.The mechanism addresses frequent copying of column and table names from SQL into natural-language responses.
- Baseline settings: TBCNN-pair replaces one sentence encoder with a database-column-name encoder and takes the user utterance plus related column names as input.
A.3 System Response Guide
CoSQL’s response guide standardizes professional system outputs by describing SQL and results directly, combining them when concise and separating them when results are large.
- Response principles: System responses follow standardized, professional rules for each dialogue action type.
- Response principles: CLARIFY requests clarification, REJECT states that the question cannot be answered, and REQUEST MORE asks whether the user wants additional information.
- SQL and result descriptions: CONFORM SQL converts the written SQL back into natural language using its column names and values, independently describing all SQL information.
- SQL and result descriptions: When results can be combined with the SQL description, the response states the query meaning and returned value together.For example, an aggregate query can be rendered as a sentence reporting the average salary.
- SQL and result descriptions: When returned results are too large to combine with the SQL description, the system describes the query and result table separately.
- SQL and result descriptions: A worked example describes average salary by department and illustrates how result-table content is explained to users.