PaperlayerView exact revision on arXiv

Source-linked AI summary

SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Task

Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, Dragomir Radev

arXiv:1810.05237v2cs.CLcs.AI

TL;DR

Text-to-SQL systems have struggled with complex, nested queries and generalization to unseen databases. SyntaxSQLNet addresses this with SQL-specific recursive syntax-tree decoding, SQL path history, and table-aware column encoders. On Spider, it improves exact matching over prior work, with further gains from cross-domain augmentation.

  • Problem

    Existing text-to-SQL datasets and methods provide limited evidence for handling complex queries and generalizing to unseen databases.

  • Method

    SyntaxSQLNet combines a SQL-specific syntax tree decoder using generation path history with table-aware column attention encoders and cross-domain data augmentation.

  • Results

    27.2% exact matching accuracy is reported, a 14.8% total improvement over the previous best model after cross-domain augmentation.

  • Takeaways & Limitations

    The model handles more complex, including nested, SQL queries on new databases than prior work, according to the reported error analysis and conclusion.

  • Takeaways & Limitations

    Column prediction is the most common error, alongside incorrect SQL skeletons, aggregators, operators, and table joins.

Abstract

from arXiv · show

Most existing studies in text-to-SQL tasks do not require generating complex SQL queries with multiple clauses or sub-queries, and generalizing to new, unseen databases. In this paper we propose SyntaxSQLNet, a syntax tree network to address the complex and cross-domain text-to-SQL generation task. SyntaxSQLNet employs a SQL specific syntax tree-based decoder with SQL generation path history and table-aware column attention encoders. We evaluate SyntaxSQLNet on the Spider text-to-SQL task, which contains databases with multiple tables and complex SQL queries with multiple SQL clauses and nested queries. We use a database split setting where databases in the test set are unseen during training. Experimental results show that SyntaxSQLNet can handle a significantly greater number of complex SQL examples than prior work, outperforming the previous state-of-the-art model by 7.3% in exact matching accuracy. We also show that SyntaxSQLNet can further improve the performance by an additional 7.5% using a cross-domain augmentation method, resulting in a 14.8% improvement in total. To our knowledge, we are the first to study this complex and cross-domain text-to-SQL task.

1 Introduction

SyntaxSQLNet addresses complex, cross-domain text-to-SQL generation, where models must produce nested, multi-clause queries and generalize across unseen databases. It combines SQL-specific syntax-tree decoding with path history and table-aware column encoding, improving exact-match accuracy over prior work.

  • Motivation: Traditional text-to-SQL benchmarks often use few SQL labels, paraphrased questions, and overlapping train-test queries, limiting tests of semantic understanding and generalization.WikiSQL separates databases but restricts queries and schemas to simple SQL.
  • Task: Spider defines a complex, cross-domain task requiring generalization to new SQL queries and unseen databases.Its database-based split makes correct prediction depend on understanding questions under the target database rather than memorization.
  • Approach: SyntaxSQLNet uses a SQL-specific syntax tree decoder with SQL generation path history and table-aware column attention encoders.The decoder targets complex queries with multiple clauses, selections, and sub-queries.
  • Results: 7.3% boost in exact matching accuracy is reported over previous best models without hyperparameter tuning.Error analysis indicates stronger handling of complex, including nested, queries in new databases.
  • Results: 7.5% further exact-matching improvement comes from cross-domain augmentation, yielding 27.2% accuracy and a 14.8% total improvement over the previous best model.The augmentation method generates more diverse training examples across databases.

2 Related Work

Related work includes semantic parsing and syntax-aware code generation, but SyntaxSQLNet targets complex, cross-domain SQL generation with SQL-specific recursive decoding rather than general abstract syntax trees.

  • Semantic Parsing: Text-to-SQL is a semantic-parsing subtask that maps natural language to formal meaning representations such as logic forms or executable programs.The cited prior work spans several semantic-parsing representations and tasks.
  • Syntax-Aware Generation: Prior syntax-aware code-generation models transduce natural language into abstract syntax trees, whereas SyntaxSQLNet recursively generates SQL syntax trees through sequence-to-set modules.The sequence-to-set design avoids ordering issues among SQL components.
  • Distinction: SyntaxSQLNet uses a SQL-specific grammar to guide complex SQL generation and passes preorder decoding history to modules as dependence information.Its design differs from general AST-based code-generation models in grammar choice, ordering treatment, and history representation.

3 Problem Formulation

The paper formulates text-to-SQL as generating complex queries over multiple-table databases while generalizing to both unseen SQL queries and unseen databases. Spider operationalizes this setting with database-level train, development, and test splits.

  • Problem Setting: The task involves multiple tables, multiple SQL clauses, nested queries, and separate databases for training and testing.The split is intended to evaluate generalization to new databases.
  • Dataset: Spider contains 10,181 questions, 5,693 unique complex SQL queries, and 200 databases with multiple tables.The dataset is used as the paper’s main evaluation benchmark.
  • Dataset: Spider includes more complex SQL labels than prior datasets such as WikiSQL, which existing WikiSQL models cannot handle.Its examples involve more tables, SQL clauses, and nested queries.
  • Cross-Domain Split: Approximately 138 domains are represented, with different databases used for training and testing.This database split requires models to generalize to unseen databases.
  • Evaluation Goal: The split aims to ensure correct predictions reflect understanding of question meaning under the given database rather than mere memorization.Training and testing use different complex SQL queries from different databases.

4 Methodology

SyntaxSQLNet decodes SQL through recursive modules selected by a SQL grammar and organized with a stack. Its modules predict distinct SQL components while using current-token context and SQL decoding history.

  • Decoder: The decoder is a collection of recursive modules with separate parameters, guided by SQL-specific grammar.The grammar determines which module is invoked at each recursive decoding step.
  • Module Design: The model decomposes SQL decoding into nine modules handling components such as keywords, operators, aggregators, and columns.The overview introduces the module decomposition before giving detailed module descriptions.
  • Module Design: The modules predict SQL constructs including set operators, keywords, columns, comparison operators, aggregators, and AND/OR conditions.ORDER BY and HAVING modules are invoked only under relevant grammar conditions.
  • Nested Queries: The Root/Terminal module chooses a new subquery root or terminal value, enabling nested-query generation.IUEN and Root/Terminal can recursively activate generation of nested queries and multiple set-operation subqueries.
  • Decoding Process: At each decoding step, the grammar and previously decoded token determine the next module and SQL token.Some modules also inspect the preceding token type, such as GROUP for HAVING or WHERE/HAVING for OP.

4.3 Input Encoder

SyntaxSQLNet encodes the question, table schema, and current SQL decoding path so its modules can predict tokens while generalizing across databases with multiple tables. Its table-aware column representations combine table names, column names, and type information, while path history conditions recursive decoding on prior states.

  • Input representations: Each module receives the question, table schema, and current SQL decoding history as input.Questions are encoded with a bidirectional LSTM, while schema and history are encoded separately.
  • Table-aware column encoding: Table-aware column encoding is designed to help generalize to new databases by using schema information.The paper identifies schema information as important for testing on unseen databases.
  • Table-aware column encoding: Unlike methods designed for single-table questions, SyntaxSQLNet uses both table and column names because Spider databases contain multiple tables.The representation captures global table context and local column context.
  • Table-aware column encoding: For each column, the encoder combines its table name, column name, and type information before applying a bidirectional LSTM.Type information includes string or number types and primary or foreign-key status.
  • Table-aware column encoding: The tested hierarchical table-and-column encoding performed worse than the proposed table-aware column encoding.The paper reports this as an experimental comparison without giving the numerical difference.
  • SQL path history: SQL path history lets modules use previous decoding states to predict different tokens during recursive generation.During training, gold query trees are traversed in preorder to create path histories; predicted histories are used during test decoding.

4.4 Module Details

SyntaxSQLNet decomposes SQL generation into grammar-guided modules that predict unordered SQL components and recursively distinguish nested subqueries from terminal values. Modules use sketch-based sequence-to-set prediction, SQL path history, and module-specific outputs to construct complex queries.

  • Sketch-based decoding: The sketch-based sequence-to-set framework predicts SQL components without penalizing equivalent outputs that differ only in ordering.For example, SELECT salary, dept name and SELECT dept name, salary represent the same set of predictions.
  • Grammar-guided decoding: SyntaxSQLNet recursively calls modules according to a SQL grammar and passes generated-token history between them.This addresses SQLNet’s fixed sketches and lack of information sharing across modules.
  • Module computation: The decoder computes conditional embeddings with trainable parameters and converts module score matrices into probability distributions.The paper denotes the trainable parameter in the conditional embedding as W and the parameter for probability computation as V.
  • Module details: The keyword module first predicts the number of SQL keywords, then selects them from SELECT, WHERE, GROUP BY, and ORDER BY.The IUEN module separately predicts one choice from INTERSECT, UNION, EXCEPT, and NONE.
  • Module details: The column, operator, and aggregator modules first predict how many items apply, then select the corresponding columns, operators, or aggregators.Operators include comparison and membership operators, while aggregators include MAX, MIN, SUM, COUNT, AVG, and NONE.
  • Nested-query and clause modules: The root/terminal module recursively predicts whether decoding continues with a ROOT node or ends at a value terminal after an operator.Additional modules predict conjunctions, ordering and limits, and whether grouped columns appear in HAVING.

4.5 Recursive SQL Generation

SyntaxSQLNet generates SQL recursively through a stack-driven process, activating grammar-specific modules to predict and organize SQL token instances until decoding completes.

  • 4.5 Recursive SQL Generation: A stack organizes recursive decoding by repeatedly popping a SQL token instance, invoking its grammar-selected module, and pushing the predicted instance.Decoding continues until the stack is empty.
  • 4.5 Recursive SQL Generation: ROOT first activates the IUEN module to decide whether the query contains EXCEPT, INTERSECT, or UNION subqueries.If NONE is predicted, it is pushed back onto the stack for subsequent decoding.
  • 4.5 Recursive SQL Generation: When SELECT is popped as a keyword instance, the COL module predicts a column name and pushes it onto the stack.

4.6 Data Augmentation

The paper uses cross-domain augmentation to address limited examples of complex SQL components by abstracting question-SQL patterns and refilling typed schema slots across databases.

  • 4.6 Data Augmentation: Cross-domain augmentation expands complex-query training data across databases, although domain-specific words and phrases make it harder than in-domain augmentation.The method targets the limited training examples available for some complex SQL components.
  • 4.6 Data Augmentation: The method creates universal question-SQL patterns by removing table, column, and value tokens from labeled pairs and grouping identical SQL query patterns.
  • 4.6 Data Augmentation: About 280 complex SQL query patterns were identified from over 4,000 training SQL labels, after filtering by SQL length and keyword count.The authors selected the 50 most frequent patterns containing multiple SQL components and nested subqueries.
  • 4.6 Data Augmentation: Manual editing aligns question slots one-to-one with SQL slots and adds column-type or table information for type-consistent slot refilling.This prevents, for example, string-type columns from filling integer-type slots.
  • 4.6 Data Augmentation: New question-SQL examples are generated by filling pattern slots with schema-matched columns, tables, and values from new databases.Column sampling uses the slot type and database schema information.
  • 4.6 Data Augmentation: 98,000 question-SQL pairs were obtained by applying the augmentation method to some single-table WikiSQL databases.

5 Experiments

Experiments evaluate SyntaxSQLNet on Spider, a cross-domain benchmark with complex SQL and multiple-table databases, using component and exact matching across query difficulty levels.

  • 5 Experiments: Spider contains 11,840 questions, 6,445 unique complex SQL queries, and 206 databases with multiple tables.The experiments use 146 databases for training, 20 for development, and 40 for testing.
  • 5 Experiments: The database split assigns 146, 20, and 40 databases to train, development, and test sets, respectively.The test databases are therefore distinct from the training databases under this split.
  • 5 Experiments: Evaluation uses SQL Component Matching and Exact Matching, decomposing predicted and gold queries into clause-level component sets for comparison.The evaluated clauses include SELECT, WHERE, GROUP BY, ORDER BY, and keywords.
  • 5 Experiments: Spider queries are categorized as easy, medium, hard, or extra hard according to their numbers of SQL components, selections, and conditions.
  • 5 Experiments: The model is implemented in PyTorch with fixed pre-trained GloVe embeddings, 120-dimensional hidden layers, dropout 0.3, Adam optimization, and batch size 64.

6 Results and Discussion

SyntaxSQLNet substantially improves exact and component matching on Spider, especially for nested queries, by combining syntax-aware decoding, SQL path history, table-aware column encoding, and cross-domain augmentation. Ablations and error analysis identify the contributions and remaining weaknesses of these components.

  • Overall Results: 7.3% exact-matching improvement over the previous best is obtained without hyperparameter tuning.The syntax-aware decoder recursively generates complex SQL according to grammar while using SQL decoding history.
  • Hardness Analysis: 24.3% and 4.6% exact matching are reported for Hard and Extra Hard queries containing nested queries.The tree-based decoder enables systematic nested-query generation and syntax enforcement avoids ungrammatical SQL.
  • Component Matching: More than 10% F1 improvement is reported for most SQL components in component matching.The comparison uses SQL clauses decomposed into sub-components and evaluates their F1 scores.
  • Ablation Study: Removing data augmentation reduces exact matching by 7.5%, especially for GROUP BY and ORDER BY components.The augmentation supplies more cross-domain column-prediction examples where the original Spider training data is sparse.
  • Ablation Study: Removing table information from column encoding lowers test performance by 3.3%, particularly for Medium and Hard queries.The drop is especially large for GROUP BY because identical column names can occur across tables.
  • Ablation Study: Removing SQL decoding history lowers exact matching by about 9.6% and reduces Hard and Extra Hard performance to 0%.The ablation indicates that decoding history supports nested-query prediction.
  • Error Analysis: The main remaining errors involve column prediction, SQL skeleton structures, aggregators, and operators.The authors suggest stronger schema encoders for relationships among columns and foreign keys.

7 Conclusion

The paper presents a syntax tree-based model for complex, cross-domain text-to-SQL generation. Its SQL-specific decoder, path history, and table-aware column attention outperform prior work and support nested queries on unseen databases.

  • Conclusion: SyntaxSQLNet addresses complex and cross-domain text-to-SQL generation with a syntax tree-based model.The model combines a SQL-specific syntax decoder, SQL path history, and table-aware column attention encoders.
  • Conclusion: The ablation study shows that the proposed techniques predict nested, complex SQL queries correctly even for unseen databases.The conclusion attributes this capability to the model’s syntax decoder, SQL path history, and table-aware column encoders.
Loading 1810.05237v2…