Source-linked AI summary
TypeSQL: Knowledge-based Type-Aware Neural Text-to-SQL Generation
Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, Dragomir Radev
TL;DR
Text-to-SQL systems must convert natural-language questions about tables into executable SQL, including questions containing rare entities and numbers. TypeSQL addresses this with a type-aware, sketch-based slot-filling approach, outperforming prior work on WikiSQL and benefiting from database-content access.
Problem
WikiSQL requires systems to generate SQL from natural-language questions and table schemas, while rare entities, numbers, and non-exact references challenge query generation.
Method
TypeSQL uses a sketch-based slot-filling design with type recognition, typed question encoding, and grouped models for related SQL slots.
Results
5.5% improvement over prior state of the art is reported without database content, while database access yields 82.6% execute accuracy and a 17.5% improvement over the previous content-aware system.
Takeaways & Limitations
TypeSQL improves WikiSQL text-to-SQL performance and can use database content to better interpret queries that are not well-formed.
Takeaways & Limitations
WikiSQL omits important operators such as JOIN and GROUP BY, limiting generalization to other SQL components.
Abstract
from arXiv · showhide
Interacting with relational databases through natural language helps users of any background easily query and analyze a vast amount of data. This requires a system that understands users' questions and converts them to SQL queries automatically. In this paper we present a novel approach, TypeSQL, which views this problem as a slot filling task. Additionally, TypeSQL utilizes type information to better understand rare entities and numbers in natural language questions. We test this idea on the WikiSQL dataset and outperform the prior state-of-the-art by 5.5% in much less time. We also show that accessing the content of databases can significantly improve the performance when users' queries are not well-formed. TypeSQL gets 82.6% accuracy, a 17.5% absolute improvement compared to the previous content-sensitive model.
1 Introduction
TypeSQL frames WikiSQL text-to-SQL generation as slot filling and adds type information to handle rare entities and numbers. It improves over prior work while reducing training time.
- WikiSQL requires generating a SQL query from a natural-language question, table, and schema.
- TypeSQL views text-to-SQL generation as a slot-filling task built on a sketch-based approach.
- 3.5% improvement over SQLNet is achieved in half the original training time.
- Type information labels entities, columns, and numbers to improve handling of rare words in questions.Examples include PERSON, COLUMN, and FLOAT labels.
- 5.5% total improvement on WikiSQL follows from the sketch-based design and type information.
2 Related Work
The paper situates TypeSQL within semantic parsing and code generation, then represents SQL generation with a slot-filled sketch. Its related work includes neural text-to-SQL systems and sequence-to-set modeling.
- Semantic parsing maps natural language to executable programs, while code generation maps descriptions into programming languages such as Python.
- TypeSQL uses three slot-filling models, with MODEL COL shown as the representative pipeline.
- Text-to-SQL is treated as a semantic-parsing subtask requiring models to account for different table schemas.
- Prior neural approaches include sequence-to-sequence and SQL-sketch methods, including SQLNet's sequence-to-set model.
- The SQL sketch exposes dollar-prefixed tokens as slots and permits zero or more AND clauses.
3 Methodology
TypeSQL frames text-to-SQL generation as slot filling over a SQL sketch, combining typed question representations with column encodings and shared slot models. It also supports database-content matching when content is available.
- Slot-Filling Formulation: TypeSQL predicts SQL-sketch slots using a slot-filling formulation based on SQLNet.Its sketch includes slots for selected columns, conditions, aggregators, operators, and condition values.
- Input Encoder: Two bidirectional LSTMs separately encode typed question words and column names before hidden states predict SQL-sketch slot values.Question word and type embeddings are concatenated before encoding.
- Type Recognition: Type recognition labels schema mentions, numeric expressions, dates, years, and selected named-entity categories in question inputs.The entity categories include PERSON, PLACE, COUNTRY, ORGANIZATION, and SPORT.
- Slot Models: TypeSQL groups related slot predictions into MODEL COL, MODEL AGG, and MODEL OPVAL, with shared encoder parameters within each model.MODEL COL handles selected columns and condition-related column slots; MODEL AGG handles aggregators; MODEL OPVAL handles operators and condition values.
- Attention: Column attention computes weighted question-and-type representations for predicting column slots and other slot decisions.The attention scores are normalized with softmax, producing HQT/COL as the weighted representation.
- Value Generation: The model generates each condition-value substring with a pointer network until the END token is most probable.The decoder selects question tokens sequentially, conditioned on the hidden state of the previously generated token.
4 Experiments
TypeSQL is evaluated on WikiSQL, where tables are separated across splits to test generalization to new schemas. It improves execution and clause-level accuracy without database content, and gains further when database content is available.
- Dataset: WikiSQL contains 87,673 examples across 26,521 tables, with each table appearing in only one train, development, or test split.This split requires generalization to new questions and new table schemas.
- Implementation: The implementation uses PyTorch, concatenated GloVe and paraphrase embeddings, 120-dimensional hidden layers, dropout 0.3, Adam, and batch size 64.The model is implemented based on SQLNet and uses the same loss functions.
- Content-Insensitive Results: 5.5% higher execute accuracy than the previous best work is achieved without database content.Clause-level gains are 1.3% for SELECT and 5.9% for WHERE; integrating word types improves representation of rare entities and numbers.
- Content-Sensitive Results: 82.6% execute accuracy is achieved with complete database access, improving the previous content-aware system by 17.5%.The paper links content access to handling questions lacking column-name and entity indicators.
5 Conclusion and Future Work
TypeSQL frames text-to-SQL generation as slot filling and uses type information to understand rare entities and numbers. On WikiSQL, it improves over prior state of the art, while the dataset’s limited SQL operator coverage constrains generalization.
- TypeSQL views text-to-SQL generation as a slot filling task and uses type information to understand rare entities and numbers.
- TypeSQL significantly improves upon the previous state of the art on the WikiSQL dataset.
- WikiSQL omits important SQL operators such as JOIN and GROUP BY, limiting generalization to other SQL components.The authors plan to evaluate more complex datasets with database splits and broader SQL coverage.