PaperlayerView exact revision on arXiv

Source-linked AI summary

SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning

Xiaojun Xu, Chang Liu, Dawn Song

arXiv:1711.04436v1cs.CLcs.AIcs.DB

TL;DR

NL2SQL systems often serialize SQL with sequence-to-sequence models, making training sensitive to equivalent orderings and yielding limited gains from reinforcement learning. SQLNet uses dependency-aware sketches, sequence-to-set prediction, and column attention instead. It improves over Seq2SQL by 9 to 13 points on WikiSQL metrics, while the task remains bounded by restrictive single-table and question-derived-value assumptions.

  • Problem

    Equivalent SQL queries can have different serializations, making sequence-to-sequence NL2SQL training sensitive to order; reinforcement learning provides only limited improvement.

  • Method

    SQLNet uses a sketch with slot dependencies, sequence-to-set prediction, and column attention to generate SQL without sequence-to-sequence decoding where order does not matter.

  • Results

    9 to 13 points: SQLNet improves over Seq2SQL across various WikiSQL metrics and achieves the new state of the art.

  • Takeaways & Limitations

    SQLNet demonstrates that avoiding sequence ordering can substantially improve NL2SQL performance on WikiSQL.

  • Takeaways & Limitations

    WikiSQL restricts evaluation to tractable single-table queries with meaningful column names and question-derived output values; more complex queries remain future work.

Abstract

from arXiv · show

Synthesizing SQL queries from natural language is a long-standing open problem and has been attracting considerable interest recently. Toward solving the problem, the de facto approach is to employ a sequence-to-sequence-style model. Such an approach will necessarily require the SQL queries to be serialized. Since the same SQL query may have multiple equivalent serializations, training a sequence-to-sequence-style model is sensitive to the choice from one of them. This phenomenon is documented as the "order-matters" problem. Existing state-of-the-art approaches rely on reinforcement learning to reward the decoder when it generates any of the equivalent serializations. However, we observe that the improvement from reinforcement learning is limited. In this paper, we propose a novel approach, i.e., SQLNet, to fundamentally solve this problem by avoiding the sequence-to-sequence structure when the order does not matter. In particular, we employ a sketch-based approach where the sketch contains a dependency graph so that one prediction can be done by taking into consideration only the previous predictions that it depends on. In addition, we propose a sequence-to-set model as well as the column attention mechanism to synthesize the query based on the sketch. By combining all these novel techniques, we show that SQLNet can outperform the prior art by 9% to 13% on the WikiSQL task.

1 INTRODUCTION

NL2SQL systems commonly serialize queries with sequence-to-sequence models, but equivalent SQL orderings create an order-matters problem. SQLNet avoids this issue with sketch-based, dependency-aware sequence-to-set generation and outperforms prior methods on WikiSQL.

  • Motivation: Equivalent SQL queries can have different constraint orderings, making sequence-to-sequence training sensitive to serialization choice.The WHERE constraints may execute identically while remaining syntactically different queries.
  • Motivation: Reinforcement learning addresses alternative serializations, but improves WikiSQL performance by only 2%.The approach fine-tunes a supervised sequence-to-sequence model using policy-gradient rewards for well-formed, correct queries.
  • SQLNet: SQLNet avoids sequence-to-sequence decoding when ordering does not matter by generating SQL from a sketch whose slots encode dependencies.Each slot prediction uses only predictions from the slots on which it depends.
  • Results: 70.1% execution accuracy: SQLNet’s advantage over Seq2SQL enlarges by 2 points when training tables overlap with deployment tables.This evaluates a reorganized WikiSQL setting in which most tables have at least one query observed during training.
  • SQLNet: SQLNet combines sequence-to-set prediction with column attention to synthesize query contents from the sketch.Column attention is introduced as a separate mechanism that further boosts the raw sequence-to-set model.
  • Results: 9 to 13 points: SQLNet surpasses prior art on various WikiSQL metrics and establishes a new state of the art.On the WikiSQL test set, it reaches 61.5% exact query-match accuracy and 68.3% result-match accuracy.

2 SQL QUERY SYNTHESIS FROM NATURAL LANGUAGE QUESTIONS AND TABLE SCHEMA

WikiSQL maps a natural-language question and table schema to a SQL query, using assumptions that make large-scale single-table synthesis tractable. Despite these constraints, the task remains challenging and is intended to test generalization to unseen schemas.

  • Dataset: The dataset is large-scale, crowd-sourced, and split across disjoint tables to evaluate generalization to unseen schemas.These properties support neural training, reduce reliance on template-synthesized descriptions, and reflect schema generalization.
  • Task: WikiSQL inputs contain a natural-language question and table schema, while the output is a SQL query for that table.The schema specifies each column’s name and type.
  • Task: WikiSQL restricts synthesis to one table, so only SELECT and WHERE clauses need prediction and FROM may be omitted.The paper illustrates this task with an example in Figure 1.
  • Assumptions: WikiSQL assumes meaningful column names, question-derived output tokens, and WHERE constraints of the form COLUMN OP VALUE.Operators are drawn from a restricted set, and VALUE must be a substring of the natural-language question.
  • Difficulty and scope: Around 60% execution accuracy: the task-specific state-of-the-art model substantially exceeds a task-agnostic semantic parser’s 37%.The authors describe WikiSQL as challenging despite its tractability assumptions and identify more complex queries as future work.

3 SQLNet

SQLNet uses a sketch aligned with SQL grammar and dependency structure, filling slots instead of serializing the entire query. Sequence-to-set prediction and column attention further tailor predictions to relevant columns and question tokens.

  • SQLNet: SQLNet fills slots in a SQL-aligned sketch rather than predicting both SQL grammar and query content.The sketch is designed to express all WikiSQL queries of interest.
  • SQLNet: The sketch conditions each slot only on predictions it depends on, avoiding sequence-to-sequence order sensitivity.This dependency structure is intended for slots whose values do not depend on one another.
  • Sequence-to-set: Sequence-to-set prediction independently identifies which table columns belong in the WHERE clause instead of generating them as an ordered sequence.The model computes Pwherecol(col|Q) for each column and uses these decisions to form the relevant subset.
  • Column attention: Column attention conditions the question representation on column names so different question tokens can inform different column predictions.The mechanism computes EQ|col from token-level question states before replacing EQ in the prediction model.
  • Evaluation: Adding an affine-transformation layer improves WHERE-column prediction performance by around 1.5%, while column attention improves over sequence-to-set by around 3 points.These improvements are reported for the corresponding prediction models.
  • Full model: SQLNet separates SELECT and WHERE predictions, first selecting WHERE columns and then generating each column’s operator and value.The model combines these components and trains without reinforcement learning.

4 EVALUATION

SQLNet is evaluated against Seq2SQL on WikiSQL using logical-form, query-match, and execution accuracy, with additional breakdowns and a table-overlap variant. It achieves substantial gains, especially for WHERE-clause prediction, and remains advantageous when test tables are seen during training.

  • Overall evaluation: 10.5 points on the test set: SQLNet’s logical-form accuracy exceeds the re-implemented Seq2SQL baseline by that margin.The corresponding dev-set improvement is 10.7 points; comparison with the original reported Seq2SQL results gives larger gaps.
  • Breakdown analysis: Around 90%: SELECT-clause prediction is less challenging than WHERE-clause prediction, and SQLNet improves SELECT-column prediction over Seq2SQL.The authors attribute the SELECT-column improvement to column attention.
  • Breakdown analysis: Around 11 to 12 points: SQLNet’s largest advantage over Seq2SQL is in WHERE-clause prediction accuracy.Canonicalizing constraint order improves Seq2SQL by only 1 point, while the sequence-to-set architecture contributes around 6 points.
  • Table-overlap variant: 70.1% execution accuracy: SQLNet reaches this score on a WikiSQL variant where all test tables appear at least once during training.SQLNet’s advantage over the re-implemented Seq2SQL remains the same across metrics in this setting.

5 RELATED WORK

Related work spans database-specific semantic parsing, user-guided and table-aware methods, sketch-based systems, and sequence-to-sequence SQL generation. SQLNet differs by combining neural sketch generation with sequence-to-set prediction and eliminating reinforcement learning for order-insensitive structures.

  • Natural-language-to-SQL approaches: Earlier natural-language-to-SQL systems focused on specific databases and required customization for each new database.This contrasts with approaches designed to generalize across schemas.
  • Natural-language-to-SQL approaches: Some recent methods incorporate user guidance or table data, whereas SQLNet does not rely on a human in the loop.The authors note potential scalability and privacy issues for approaches that use table data as input.
  • Sketch-based approaches: SQLizer is also sketch-based and database-general, but uses an off-the-shelf semantic parser to produce sketches before programmatic completion and repair.SQLNet instead presents a neural-network alternative to traditional sketch-based program synthesis.
  • Sequence-based approaches: SQLNet improves over Seq2SQL by using sequence-to-set generation when order does not matter, removing the need for reinforcement learning.The approach retains Seq2SQL’s generalizability to unseen schemas while addressing the order-matters problem.
  • Sequence-based approaches: A generic sequence-to-tree semantic-parsing model was evaluated as less effective than Seq2SQL and therefore was excluded from the comparison.Seq2SQL is identified as the most relevant prior work and the baseline for SQLNet.

6 CONCLUSION

SQLNet addresses order-insensitive SQL generation with a sequence-to-set model and column attention rather than reinforcement learning. On WikiSQL, it improves over Seq2SQL by 9 to 13 points across metrics, supporting this strategy for structural generation when order does not matter.

  • Conclusion: Existing sequence-to-sequence NL2SQL approaches suffer from the order-matters problem when equivalent SQL orderings are possible.Prior reinforcement-learning attempts brought only a small improvement, around 2 points.
  • Conclusion: SQLNet uses sequence-to-set generation to fundamentally solve order sensitivity when SQL element order does not matter.The system also introduces column attention to further improve the sequence-to-set model.
  • Conclusion: 9 points to 13 points: SQLNet improves over Seq2SQL by this range across various WikiSQL metrics.The authors present this as evidence that SQLNet effectively addresses the order-matters problem.
  • Conclusion: SQLNet’s approach suggests a direction for structural generation problems in which ordering is not semantically meaningful.This conclusion is stated within the scope of the authors’ WikiSQL findings.
Loading 1711.04436v1…