Source-linked AI summary
RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases
DongHyun Choi, Myeong Cheol Shin, EungGyun Kim, Dong Ryeol Shin
TL;DR
Text-to-SQL must generate SQL from natural-language questions and databases, but complex cross-domain settings include unseen schemas and nested, multi-table queries. RYANSQL addresses this with recursive Statement Position Codes and sketch-based slot filling, supplemented by two input manipulation methods. It achieves state-of-the-art Spider performance, including a 3.2%p exact-match improvement over the prior state of the art with BERT.
Problem
Text-to-SQL must generate SQL from natural-language questions and databases, while Spider targets complex cross-domain queries that challenge earlier systems.
Method
RYANSQL recursively predicts nested queries through Statement Position Code and fills detailed SELECT-statement sketches, with two input manipulation methods.
Results
3.2%p improvement in hidden-test exact matching accuracy over the current state of the art is reported for RYANSQL with BERT.
Takeaways & Limitations
RYANSQL achieves state-of-the-art performance on the challenging Spider benchmark.
Takeaways & Limitations
The main reported limitation is slot prediction errors, especially column selection and table-number classification, motivating future slot-value updating.
Abstract
from arXiv · showhide
Text-to-SQL is the problem of converting a user question into an SQL query, when the question and database are given. In this paper, we present a neural network approach called RYANSQL (Recursively Yielding Annotation Network for SQL) to solve complex Text-to-SQL tasks for cross-domain databases. State-ment Position Code (SPC) is defined to trans-form a nested SQL query into a set of non-nested SELECT statements; a sketch-based slot filling approach is proposed to synthesize each SELECT statement for its corresponding SPC. Additionally, two input manipulation methods are presented to improve generation performance further. RYANSQL achieved 58.2% accuracy on the challenging Spider benchmark, which is a 3.2%p improvement over previous state-of-the-art approaches. At the time of writing, RYANSQL achieves the first position on the Spider leaderboard.
1 Introduction
RYANSQL addresses complex Text-to-SQL over cross-domain databases, where prior benchmarks and systems do not adequately cover unseen schemas and nested, multi-table queries. It combines sketch-based slot filling, Statement Position Code, and input manipulation, improving exact-match performance.
- Motivation: Spider evaluates complex SQL over cross-domain databases, unlike benchmarks limited to shared databases or single-table, unitary SELECT queries.Its queries include nested queries, multiple JOINed tables, and ORDERBY, GROUPBY, and HAVING clauses.
- Approach: RYANSQL recursively yields component SELECT statements to handle complex, cross-domain Text-to-SQL.The architecture is designed for nested query generation.
- Results: 3.2%p improvement over the current state-of-the-art was achieved on hidden-test exact matching accuracy with BERT.The paper reports this comparison as the proposed system’s overall performance improvement.
- Approach: Its sketch-based slot-filling approach uses a detailed SELECT sketch and fills its slots to predict complex SQL statements.The sketch is paired with a network architecture for slot prediction.
- Approach: Statement Position Code recursively predicts nested queries by associating each SELECT statement with its position in the query structure.SPC supports recursive prediction of nested queries using sketch-based slot filling.
- Results: Two simple input manipulation methods significantly improve overall system performance.The methods are presented as easy to apply.
2 Related Works
Prior Text-to-SQL work includes direct sequence-to-sequence, grammar-based, and sketch-based slot-filling approaches. RYANSQL extends sketch-based slot filling to complex Spider queries with detailed SELECT sketches and Statement Position Code for nested queries.
- Direct sequence-to-sequence: Direct sequence-to-sequence methods generate SQL tokens directly, but their risk of grammatically incorrect SQL has made them uncommon in recent work.These approaches output query tokens rather than grammar rules or slot values.
- Grammar-based approaches: Grammar-based methods generate grammar rules sequentially and apply them to construct the resulting SQL query.Examples include structural representations, SemQL trees, and global reasoning over database constraints.
- Sketch-based approaches: Sketch-based slot filling defines SQL sketches with slots whose values are classified by the decoder.Earlier sketch-based systems performed strongly on WikiSQL but relatively poorly on complex Spider queries.
- RYANSQL: RYANSQL introduces a detailed sketch for complex SELECT statements together with Statement Position Code to predict nested queries.This extends sketch-based slot filling to the complex Text-to-SQL setting.
3 Task Definition
The task maps a natural-language question and database schema to an SQL query. RYANSQL instead constructs a non-nested representation: SELECT statements paired with Statement Position Codes that encode their structural positions.
- Task inputs: The input consists of a tokenized question and a database schema containing tables, columns, and foreign-key relations.Tables and columns are represented through their names, while columns also carry primary-key markers.
- Non-nested representation: For an SQL query S, the non-nested form N(S) is a set of pairs consisting of each statement’s SPC and corresponding SELECT statement.The paper uses N(S) as the target representation for query construction.
- Statement Position Code: An SPC is a sequence of position-code elements such as NONE, UNION, INTERSECT, EXCEPT, WHERE, HAVING, and PARALLEL.NONE denotes the outermost statement, PARALLEL denotes parallel elements within one clause, and other elements identify SQL clauses.
- Recursive construction: The system initializes the outermost position as P1 = [NONE] and predicts its corresponding SELECT statement before recursively handling nested positions.Constructing the SQL query from N(S) is described as straightforward.
- Input encoding: The input encoder includes embedding, embedding-encoder, question-column alignment, table-encoder, and question-table alignment layers.The architecture also uses concatenation, max-pooling, and self-attention operations.
4 Generating a SELECT Statement
RYANSQL encodes the question, schema, and statement position, then fills a detailed SELECT sketch through staged classification and slot prediction. Its decoder recursively handles nested clauses by creating additional SELECT statements for set-operation positions.
- Input Encoder: The input encoder produces contextual representations for question words, columns, tables, the SPC, the question, and the database schema.It combines word and character embeddings, convolutional encoding, question-column alignment, table encoding, and question-table alignment; BERT is also supported as an alternative encoder.
- Base Structure: The decoder classifies a SELECT statement’s base structure by predicting clause existence, condition counts, and whether INTERSECT, UNION, or EXCEPT is present.The base structure includes GROUPBY, ORDERBY, LIMIT, WHERE, and HAVING; FROM and SELECT are required.
- Sketch-based Slot-Filling Decoder: The proposed SELECT sketch exposes slots for tables, columns, aggregations, arithmetic, conditions, boolean keywords, ordering, values, and nested SELECT statements.The decoder predicts values for these slots rather than generating SQL tokens directly.
- Recursive Generation: When a set-operation clause is predicted, RYANSQL creates the corresponding SPC and recursively generates the associated SELECT statement.The system represents nested queries as non-nested SELECT statements paired with Statement Position Codes, then reconstructs the SQL query.
- Slot Filling: The decoder predicts FROM tables and SELECT conditions using table probabilities, attended question vectors, and column-slot probability matrices.It selects up to six candidate tables, generates multiple SELECT conditions, and updates question representations with selected column information.
5 Two Input Manipulation Methods
RYANSQL uses two input manipulations to reduce schema ambiguity and alignment noise: supplementing column names with table names and filtering link tables during training.
- Purpose: The paper presents both input manipulations as simple methods intended to improve overall system performance.The two methods are introduced as additional performance improvements for the proposed system.
- JOIN Table Filtering: JOIN table filtering removes tables used only to connect other tables during training and recovers them at inference through foreign-key relations.These link tables are treated as alignment noise because they lack corresponding question tokens.
- Supplemented Column Names: Supplemented Column Names concatenate table names with column names to distinguish identically named columns from different tables.The table prefix is omitted when its stemmed form is wholly contained in the stemmed column name.
6 Experiment
Experiments evaluate RYANSQL on Spider using exact matching and compare it with prior systems and ablations. Results show gains from BERT, SPC, and input manipulation, while error analysis identifies column and table-selection failures.
- Experimental setup: Spider evaluation uses the established data split, exact matching accuracy, and comparisons with grammar-based and sketch-based systems.The split contains 146 training, 20 development, and 40 test databases; the hidden test set is evaluated through leaderboard submission.
- Comparison results: 3.2%p improvement over the current state-of-the-art is achieved on the hidden test dataset with BERT, measured by exact matching accuracy.RYANSQL also improves RCSQL by 15%p on the development dataset.
- Ablation results: SPC significantly improves performance, especially for Hard and Extra Hard queries, suggesting better handling of nested queries.The ablation studies use the development dataset because Spider’s test dataset is not publicly available.
- Ablation results: JTF improves Medium and Hard queries, indicating effectiveness for statements involving multiple tables and clauses.The ablation defines JTF as removing JOIN Table Filtering from the proposed system.
- Ablation results: SCN produces the most significant improvement among the three proposed features and improves performance across all hardness levels.The results suggest that SCN integrates table names into encoding vectors without modifying the network architecture.
- Error analysis: 34.9% of analyzed failures are column-selection errors, often occurring when the correct column name is not mentioned in the question.The analysis examined 195 of 345 failed RYANSQL(BERT) development examples.
- Error analysis: 25.2% of analyzed failures are table-number classification errors, while 11.3% are condition-number classification errors.The decoder may select too many tables, causing unnecessary JOINs; the authors propose jointly observing and updating extracted slot values as future work.
- Error analysis: Some remaining errors reflect different SQL representations with the same meaning, such as aggregation versus ordering with a limit.These errors were among the 150 cases not classified into a single main category.
7 Conclusion
RYANSQL combines sketch-based slot filling, detailed SELECT sketches, and Statement Position Code to handle nested queries in complex, cross-domain Text-to-SQL. The system achieved state-of-the-art performance on Spider, while error analysis identifies slot-value updating as future work.
- RYANSQL uses sketch-based slot filling for complex, cross-domain Text-to-SQL.
- A detailed sketch supports complex SELECT statement prediction, while Statement Position Code handles nested queries.
- Two input manipulation methods are proposed to further enhance overall system performance.
- The system achieved state-of-the-art performance on the challenging Spider benchmark dataset.
- Future work will focus on updating slot values using other slots’ prediction results.