Source-linked AI summary
Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing
Ben Bogin, Matt Gardner, Jonathan Berant
TL;DR
Text-to-SQL systems have often ignored database schema structure, although SPIDER introduces unseen complex databases where that structure can inform query prediction. The paper encodes schemas with graph neural networks and uses the resulting representations during encoding and decoding, improving accuracy from 33.8% to 39.4% versus 19.7% state of the art.
Problem
Prior text-to-SQL work largely ignored schema structure, while SPIDER evaluates queries on unseen complex databases where schema information can inform prediction.
Method
The parser converts each database schema into a graph, uses a graph neural network to encode global structure, and incorporates the representation during question encoding and SQL decoding.
Results
39.4% accuracy is achieved on SPIDER, compared with 33.8% without the GNN and 19.7% for the reported state of the art.
Takeaways & Limitations
Encoding schema structure improves parsing performance, especially on questions involving multiple tables, where accuracy rises from 14.6% to 26.8%.
Abstract
from arXiv · showhide
Research on parsing language to SQL has largely ignored the structure of the database (DB) schema, either because the DB was very simple, or because it was observed at both training and test time. In Spider, a recently-released text-to-SQL dataset, new and complex DBs are given at test time, and so the structure of the DB schema can inform the predicted SQL query. In this paper, we present an encoder-decoder semantic parser, where the structure of the DB schema is encoded with a graph neural network, and this representation is later used at both encoding and decoding time. Evaluation shows that encoding the schema structure improves our parser accuracy from 33.8% to 39.4%, dramatically above the current state of the art, which is at 19.7%.
1 Introduction
Text-to-SQL research has often underused database schema structure, but SPIDER tests on unseen complex databases where schema information helps determine the SQL query. The paper proposes a schema-structured parser and reports higher accuracy.
- Prior text-to-SQL work often used one-table databases or a single database observed during training and testing.
- SPIDER evaluates questions against unseen and complex databases, making an informative representation of schema structure important.
- Similar questions can require different SQL because schema-dependent information may be distributed across multiple tables, requiring a join in one case but not another.
- The proposed semantic parser represents schema structure as a graph and uses graph neural networks within an encoder-decoder parser.
- 33.8% to 39.4% accuracy is achieved by encoding schema structure, exceeding the reported 19.7% state-of-the-art result.
2 Problem Setup
The task is to learn a function that maps a question and its database schema to the correct SQL query, even when the schema is unseen during training. The setup represents questions, queries, and schemas as linked training examples.
- The training data consists of question-query-schema triples, where each question is translated into a SQL query executed on its associated database schema.
- The goal is to map an unseen question-schema pair to its correct SQL query.
- The target schema is not observed during training, so it differs from every training-example schema.
- A database schema contains tables, columns for each table, and foreign-key/primary-key column pairs linking columns across tables.
3 A Neural Semantic Parser for SQL
The baseline parser uses a grammar-based attentive LSTM decoder and linking mechanisms to generate SQL and unseen schema items. Its encoder augments question words with linked schema representations, while decoding attends to question words and legal grammar choices.
- The parser combines the Krishnamurthy et al. model with an AllenNLP SQL grammar covering 98.3% of SPIDER examples.
- Linking schema items: A learned linking score connects question words to unseen schema items using word embeddings and manually crafted features.
- Encoder: The bidirectional LSTM encoder concatenates each word embedding with a schema representation weighted by linking probabilities.
- Decoder: The grammar-based LSTM decoder expands typed nonterminals using schema-independent rules for SQL structure or schema-specific rules for schema items.
- Decoder: At each decoding step, attention over question words helps compute distributions over legal grammar rules and schema items.
- Linking schema items: The parser uses schema-item linking to decode unseen schema items by attending first to question words linked to those items.
4 Modeling Schemas with GNNs
The model represents database schemas as question-conditioned graphs, then uses GNN-derived node representations in both the encoder and decoder. Decoder self-attention further scores legal schema items according to their similarity to previously decoded items.
- Schema-to-graph: The schema is converted into a graph whose nodes are tables and columns, with directed edges encoding table-column and foreign-primary-key relations.Different edge types let the GNN capture distinct ways that tables and columns relate.
- Question-conditioned relevance: Schema relevance is conditioned on the question by assigning each schema item the maximum linking probability from any question word.Relevant items are retained more strongly in the graph representation than items unrelated to the question.
- Neural graph representation: Each node starts with an embedding scaled by its relevance score, then gated GNN updates recompute node representations from neighboring nodes over L steps.The resulting representations incorporate both question relevance and global schema structure.
- Encoder: The encoder augments each question word with a weighted average of GNN representations for linked schema items, exposing surrounding graph structure to the question representation.The weighted average replaces the earlier schema-item representations with the final graph-aware representations.
- Decoder: The decoder replaces a decoded schema item's type embedding with its graph-aware representation, so subsequent decisions can use the item's surrounding schema structure.This change is part of a decoder that also applies self-attention over previously decoded schema-item steps.
- Decoder: Self-attention scores legal schema items by similarity to previously decoded schema items, increasing scores for items structurally similar to attended items.Similarity is computed from transformed GNN representations of the candidate and previously decoded schema items.
5 Experiments and Results
On SPIDER, the schema-aware GNN parser improves text-to-SQL accuracy over its no-GNN counterpart and prior state of the art, with especially large gains on multi-table queries. Ablations and analyses show that relevance modeling, self-attention, and graph representations contribute to performance and better join predictions.
- 39.4% test accuracy substantially exceeds SYNTAXSQLNET’s 19.7%, while removing the GNN reduces accuracy to 33.8%.
- 40.7% development accuracy improves over NO GNN’s 34.9%, including a rise from 14.6% to 26.8% on multi-table questions.The evaluation partitions results into the full dataset, single-table queries, and multi-table queries.
- Removing self-attention causes performance to drop by 2 points, although single-table performance slightly improves.
- 35.9% accuracy after exposing GNN representations only through self-attention shows that other GNN uses also matter; removing relevance modeling reduces accuracy to 37.0%.
- 54.3% oracle performance with gold relevance scores indicates substantial headroom for improving schema-item relevance.
- 15.6% of GNN predictions contain bad joins, compared with 83.4% for NO GNN, and a worked example shows self-attention favoring connected tables.
6 Conclusion
The paper presents a semantic parser that encodes database-schema structure with a graph neural network and uses it for schema-aware decisions during encoding and decoding. On SPIDER’s unseen complex schemas, this approach substantially improves over the current state of the art.
- The parser represents database-schema structure with a graph neural network for schema-aware decisions at encoding and decoding time.