Source-linked AI summary
Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
Victor Zhong, Caiming Xiong, Richard Socher
TL;DR
Users often struggle to retrieve information from relational databases because SQL is difficult to master. Seq2SQL translates natural-language questions into SQL using structured generation and execution-based reinforcement learning, and WikiSQL provides a large training and evaluation corpus. On WikiSQL, it improves execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.
Problem
SQL’s difficulty limits users’ ability to retrieve facts from relational databases, motivating translation from natural-language questions to SQL queries.
Method
Seq2SQL uses SQL structure to reduce its output space, combines supervised losses with policy-based reinforcement learning, and trains unordered conditions using database-execution rewards.
Results
On WikiSQL, Seq2SQL improves execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.
Takeaways & Limitations
WikiSQL supplies 80654 hand-annotated examples across 24241 Wikipedia tables, while Seq2SQL achieves state-of-the-art query-generation performance on the dataset.
Takeaways & Limitations
Cross-entropy supervision is unsuitable for unordered WHERE conditions because equivalent condition orderings can produce the same query result.
Abstract
from arXiv · showhide
A significant amount of the world's knowledge is stored in relational databases. However, the ability for users to retrieve facts from a database is limited due to a lack of understanding of query languages such as SQL. We propose Seq2SQL, a deep neural network for translating natural language questions to corresponding SQL queries. Our model leverages the structure of SQL queries to significantly reduce the output space of generated queries. Moreover, we use rewards from in-the-loop query execution over the database to learn a policy to generate unordered parts of the query, which we show are less suitable for optimization via cross entropy loss. In addition, we will publish WikiSQL, a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables from Wikipedia. This dataset is required to train our model and is an order of magnitude larger than comparable datasets. By applying policy-based reinforcement learning with a query execution environment to WikiSQL, our model Seq2SQL outperforms attentional sequence to sequence models, improving execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.
1 INTRODUCTION
Seq2SQL addresses the difficulty of querying relational databases by translating natural-language questions into SQL and using WikiSQL to evaluate the approach. Its training pipeline executes generated queries during learning, and it achieves 59.4% execution accuracy on WikiSQL.
- Motivation: Relational databases hold substantial information, but SQL is difficult for many users to master, motivating natural-language interfaces for database access.The paper focuses specifically on translating natural-language questions into SQL queries.
- Seq2SQL: Seq2SQL is a deep neural network that uses SQL structure to prune the output space and policy-based reinforcement learning for unordered query conditions.The model combines cross-entropy losses with rewards from in-the-loop database execution.
- WikiSQL: WikiSQL contains 80654 hand-annotated questions, SQL queries, and tables extracted from 24241 Wikipedia HTML tables.The corpus is described as an order of magnitude larger than previous semantic-parsing datasets with logical forms.
- Training: During training, Seq2SQL executes generated SQL queries against a database and uses the execution result as the reinforcement-learning reward.The execution environment is part of the model’s training pipeline.
- Dataset format: WikiSQL examples pair a table and question as inputs with a ground-truth SQL query and its execution result as outputs.This example format connects natural-language questions, structured queries, and database answers.
- Results: 59.4% execution accuracy is achieved by Seq2SQL on WikiSQL, compared with 35.9% for the Dong and Lapata semantic parser and 53.3% for an augmented pointer network.The reported comparison is for execution accuracy on WikiSQL.
2 MODEL
Seq2SQL exploits SQL structure by separately predicting aggregation, SELECT columns, and WHERE conditions. It uses pointer-based representations and policy-gradient rewards from query execution to handle the unordered WHERE component.
- Augmented Pointer Network: The augmented pointer network restricts SQL generation to tokens from column names, the question, and a limited SQL vocabulary.It encodes this concatenated input with a bidirectional LSTM and selects query tokens from the input sequence.
- Seq2SQL: Seq2SQL decomposes queries into aggregation, SELECT-column, and WHERE-clause components, aligning its architecture with SQL structure.The model predicts aggregation and SELECT columns before generating WHERE conditions with a pointer network.
- Aggregation Operation: The aggregation component predicts COUNT, MIN, MAX, or NULL from an attention-based input representation and is trained with cross entropy loss.NULL represents queries without an aggregation operation.
- SELECT Column: The SELECT component matches a question representation against LSTM-encoded column representations and is trained with cross entropy loss.A softmax produces a distribution over possible table columns.
- WHERE Clause: Cross entropy can wrongly penalize equivalent WHERE clauses when their conditions appear in different orders.Seq2SQL therefore uses reinforcement learning to optimize expected execution correctness, with rewards assigned after executing generated queries.
- Mixed Objective Function: The mixed training objective equally combines aggregation cross entropy, SELECT-column cross entropy, and WHERE-clause policy-learning gradients.The objective is L = Lagg + Lsel + Lwhe.
3 WIKISQL
WikiSQL is a crowdsourced dataset of natural-language questions, SQL queries, and Wikipedia tables, released with executable database resources and evaluation metrics.
- Dataset construction: WikiSQL is collected through question paraphrasing and independent verification by Amazon Mechanical Turk workers.Workers paraphrase template-generated questions, while two additional workers verify semantic equivalence; invalid or insufficiently varied paraphrases are discarded.
- Dataset construction: Tables, paraphrases, and queries are split so that each table appears in exactly one train, development, or test split.The release also includes raw tables, queries, results, natural utterances, a SQL database, and a query execution engine.
- Evaluation: Execution accuracy counts queries producing the correct result, whereas logical-form accuracy requires exact string match with the ground-truth query.Both metrics are used because equivalent WHERE clauses can differ in condition order while producing the same result.
4 EXPERIMENTS
Experiments evaluate Seq2SQL and related baselines using WikiSQL, showing gains from constrained generation, SQL structure, and reinforcement learning with query-execution rewards.
- Experimental setup: The experiments use tokenized inputs, fixed GloVe and character n-gram embeddings, dropout, ADAM optimization, and early stopping on development execution accuracy.Recurrent layers use 200 hidden units and dropout of 0.3, with training capped at 300 epochs.
- Results: SQL structure adds 3.8% improvement, while reinforcement learning from in-the-loop query execution adds another 2.3%.These gains compare Seq2SQL without reinforcement learning with the augmented pointer network, then compare full Seq2SQL with its non-RL variant.
- Analysis: Pointer-based generation produces more accurate conditions containing rare words by selecting exclusively from the input.Examples include recovering “1850” in a date and “doug battaglia” in a player name, where the baseline generated shorter values.
- Analysis: Structured prediction reduces invalid SQL queries from 7.9% to 4.8% by constraining selection and aggregation decisions.The analysis attributes many invalid queries to nonexistent selection columns and reports improved COUNT precision, recall, and F1 with an aggregation classifier.
- Analysis: Policy-based reinforcement learning handles correct WHERE clauses whose condition order differs from the ground-truth query.The RL model can produce equivalent reordered conditions and avoids some incorrect clauses generated by the non-RL model.
5 RELATED WORK
Prior work covers semantic parsing, benchmark datasets, sequence-generation models, and natural-language database interfaces, often within fixed schemas or grammar-based systems.
- Semantic parsing: Semantic parsing maps natural-language questions to executable logical forms, with some approaches learning from logs, demonstrations, distant supervision, or question-answer pairs.Traditional systems are often constrained to a single schema and may require hand-curated grammars.
- Semantic parsing datasets: Earlier datasets target complex questions over closed-domain fixed-schema resources such as GeoQuery and ATIS, or subsets of knowledge graphs.Overnight uses crowdsourcing for question-logical-form pairs but covers only eight domains.
- Representation learning: Attentional sequence-to-sequence semantic parsing achieves strong results without hand-engineered grammar, while pointer-based generation is used to improve handling of rare words and column names.Seq2SQL uses pointer-based generation in contrast to the cited baseline.
- Natural-language database interfaces: Grammar-based database interfaces can translate questions to SQL or rank grammar-generated candidates, but they rely on high-quality grammars and are unsuitable for generalizing to new schemas.PRECISE also identifies questions for which it lacks confidence.
6 CONCLUSION
The paper concludes that Seq2SQL combines SQL-structure-aware generation with in-the-loop execution training, alongside the WikiSQL dataset. It reports improved execution and logical form accuracy over a state-of-the-art semantic parser.
- Seq2SQL translates questions to SQL while reducing the model’s output space through SQL structure.
- In-the-loop query execution trains a policy for generating unordered SQL conditions that are unsuitable for cross entropy optimization.
- WikiSQL is an order of magnitude larger than comparable datasets of questions and SQL queries.
- 35.9% to 59.4% execution accuracy and 23.4% to 48.3% logical form accuracy were achieved against a state-of-the-art semantic parser.
- The paper defines table schema as the names of the columns in a table.
A COLLECTION OF WIKISQL
WikiSQL is constructed by filtering Wikipedia tables, generating constrained SQL queries, crowdsourcing paraphrases, and verifying semantic equivalence. The resulting collection uses non-empty, succinct queries and excludes problematic table structures.
- WikiSQL begins with Wikipedia tables and removes tables with malformed or unsuitable structures.Excluded cases include inconsistent row widths, overly long cell content, empty header cells, small tables, and highly repetitive rows.
- The final row is removed because HTML tables often place summary statistics there rather than values matching the header-defined schema.
- Six SQL queries are randomly generated per qualifying table using a SELECT-WHERE format with aggregation and multiple conditions.
- Aggregation is limited to empty or COUNT, with MAX and MIN additionally allowed for numeric aggregation columns.
- Condition operators are restricted to equality, with greater-than and less-than additionally allowed for numeric condition columns.
- Queries must return non-empty results, and redundant conditions are removed when their removal preserves the execution result.
- Template-generated questions receive human paraphrases from Mechanical Turk workers who see the table’s first four rows.
- Two additional workers verify semantic equivalence, and accepted paraphrases must pass correctness and edit-distance filters.A paraphrase is retained when at least one verifier deems it correct and its character-level edit distance exceeds 10.
B ATTENTIONAL SEQ2SEQ NEURAL SEMANTIC PARSER BASELINE
The baseline is an attentional sequence-to-sequence semantic parser with a bidirectional stacked LSTM encoder and attention-based decoder. Training uses teacher forcing, while inference uses beam search and attention-based unknown-word replacement.
- The baseline uses an attentional sequence-to-sequence model implemented with a global-attention encoder-decoder architecture.
- Its encoder is a two-layer bidirectional stacked LSTM, and its decoder differs from the referenced formulation through input feeding.
- At each decoding step, attention context is combined with the decoder state to produce a target-vocabulary distribution.
- Training uses teacher forcing, whereas inference uses beam size 5 and replaces unknown words using the highest-attention input words.
C PREDICTIONS BY SEQ2SQL
Table 4 presents development-set prediction examples from the Augmented Pointer Network, Seq2SQL without reinforcement learning, and full Seq2SQL. The displayed queries omit their FROM table components for brevity.
- Table 4 compares model predictions on the development split against corresponding ground-truth queries.
- P, S’, and S denote predictions from the Augmented Pointer Network, Seq2SQL without reinforcement learning, and Seq2SQL, respectively.
- The examples omit each query’s FROM table portion to keep the displayed predictions succinct.