Source-linked AI summary
Global Reasoning over Database Structures for Text-to-SQL Parsing
Ben Bogin, Matt Gardner, Jonathan Berant
TL;DR
Zero-shot semantic parsing must choose database constants in complex databases whose schemas were unseen during training, while existing autoregressive decoding makes those choices locally. The paper combines global, question-conditioned constant selection with query reranking, raising Spider accuracy from 39.4% to 47.4%.
Problem
In zero-shot parsing, the model must map new lexical items to database constants in an unseen database, while local autoregressive decoding selects constants one at a time.
Source: §1 · p. 1Method
Results
47.4% accuracy on Spider exceeds the current state-of-the-art 39.4%.
Source: §1 · p. 2Takeaways & Limitations
The two contributions are applicable to any zero-shot semantic parser and improve performance on Spider.
Source: §5 · p. 5The setup assumes the database schema was not seen during training.
Source: §2 · p. 2
Abstract
from arXiv · showhide
State-of-the-art semantic parsers rely on auto-regressive decoding, emitting one symbol at a time. When tested against complex databases that are unobserved at training time (zero-shot), the parser often struggles to select the correct set of database constants in the new database, due to the local nature of decoding. In this work, we propose a semantic parser that globally reasons about the structure of the output query to make a more contextually-informed selection of database constants. We use message-passing through a graph neural network to softly select a subset of database constants for the output query, conditioned on the question. Moreover, we train a model to rank queries based on the global alignment of database constants to question words. We apply our techniques to the current state-of-the-art model for Spider, a zero-shot semantic parsing dataset with complex databases, increasing accuracy from 39.4% to 47.4%.
1 Introduction
Zero-shot parsing must map new lexical items to database constants in an unseen database, but local, one-at-a-time decoding misses relationships across the question and schema. The paper adds global constant selection and query reranking, improving Spider accuracy over the prior state of the art.
1 Introduction
- Zero-shot parsing maps utterances to executable programs in a new database, where lexical items must be linked to constants unseen during training.Complex databases make selecting the correct set of constants a central challenge.Source: §1 · p. 1
- Local similarity and autoregressive decoding select constants one at a time, leaving informative global considerations unused.The example contrasts the similar columns singer.name and song.name; later question words and a cell value can help resolve the choice.Source: §1 · p. 1
- The parser globally reasons over the database structure and question to decide which database constants should appear in a query.It uses schema-graph message passing to softly select likely constants and reranks the autoregressive decoder’s top-K queries by global database–question matching.Source: §1 · pp. 1–2
- 47.4% accuracy on Spider exceeds the current state-of-the-art 39.4%.The authors report that both contributions improve performance.Source: §1 · p. 2
2 Schema-augmented Semantic Parser
The section describes a grammar-based parser that selects database constants using local word–constant similarities, alongside schema-graph representations used to encode database structure. Its existing relevance gate is based on local information rather than the full question or schema.
Schema-augmented Semantic Parser
- The base parser encodes question words with a BiLSTM and decodes SQL top-down with an LSTM and grammar rules.At steps requiring a table or column, it attends over question words and selects a database constant.Source: §2 · p. 2
- The decoder scores each constant by attention-weighted local similarity, using learned embeddings and string features such as edit distance and overlap.Because the similarity is computed independently for each word and constant, dependencies between constant decisions are weak.Source: §2 · p. 2
- A database schema graph connects table and column nodes with table-column and primary–foreign-key edges, and a GCN learns node representations.Those representations are used by the parser during decoding.Source: §2 · p. 2
- The existing relevance probability gates GCN inputs but is computed from local information, so it does not consider the full question or database structure.The probability is the maximum local-link probability over question words.Source: §2 · p. 2
3 Global Reasoning over DB Structures
The section adds a gating GCN to estimate relevant database constants from global question–schema context, then reranks decoder candidates using the selected constants’ structure and their alignment with question words.
3 Global Reasoning over DB Structures
- A learned gating GCN predicts each database constant’s question-conditioned relevance from global context and supplies those probabilities to the encoder GCN.Its graph adds a global node connected to all other nodes; node inputs combine database-constant and question representations.Source: §3 · p. 3
- The gating GCN is trained with relevance supervision from the gold query’s database constants alongside the usual decoding loss.The gold subset is extracted from constants appearing in the query.Source: §3 · p. 3
- A discriminative model reranks the decoder’s top-K queries by scoring each candidate against the question and schema, focusing on its database-constant set.The re-ranker is trained to assign probability to the correct query.Source: §3 · pp. 3–4
- The reranking GCN processes the subgraph induced by a candidate’s selected constants plus a global node to represent the question-conditioned subgraph.Its global-node representation summarizes selected-node properties.Source: §3 · p. 4
- The reranker supplements selected-node structure with an alignment representation because selected constants may omit nodes relevant to question words.The alignment representation captures whether question words align with selected database constants.Source: §3 · p. 4
- For each question word, the model pools representations of aligned database constants, concatenates that summary to the word representation, then attends over question words.This alignment term lets the model recognize attended words aligned with constants absent from the selected set.Source: §3 · p. 4
4 Experiments and Results
Experiments on SPIDER evaluate GLOBAL-GNN against prior work and through component ablations, with additional oracle scores estimating headroom. The model improves test accuracy and performs especially better on multi-table queries.
Experimental setup
- The SPIDER setup uses 7,000/1,034/2,147 train/development/test examples; re-ranker training draws K = 40 decoder-beam candidates.At test time, the re-ranker selects among the best K = 10 candidates, with decoder scores breaking ties.Source: §4 · p. 4
Results
- 47.4% test accuracy for GLOBAL-GNN exceeds the 39.4% state-of-the-art comparison on SPIDER.Table 1 reports test-set accuracy; evaluation uses the official script for loose exact match of queries.Source: Table 1 · p. 4, §4 · p. 4
- 52.1% development-set accuracy is reported for GLOBAL-GNN, with gains coming mostly from queries requiring more than one table.The paper characterizes multi-table queries as usually more complex.Source: §4 · p. 4
Ablations
- Removing either global gating or re-ranking leads to a 4% performance drop; removing the relevance loss degrades accuracy by 2%.Omitting the re-ranker’s ealign representation also reduces performance, reflecting the importance of identifying unaligned question words.Source: §4 · pp. 4–5
Re-ranking analysis
- A full-query re-ranker performs worse; the authors report that it excels at choosing database constants, while the decoder better determines SQL structure and logical constants.Many errors introduced by the full-query re-ranker are minor mistakes such as min instead of max.Source: §4 · p. 5
Oracle scores and error analysis
- Perfect global gating raises accuracy to 63.2%, and adding a perfect re-ranker raises it to 73.5%; these oracle scores estimate future headroom.In 44.4% of errors where the correct query was in the beam, the selected set of database constants was correct but the query was wrong.Source: §4 · p. 5
5 Conclusion
The conclusion presents global decision-making over database constants as important for zero-shot semantic parsing. It combines schema-aware gating with global re-ranking and reports improved SPIDER performance.
5 Conclusion
- The paper identifies global decision-making over relevant database constants as important for zero-shot semantic parsing.Selecting the relevant set of constants is described as challenging.Source: §5 · p. 5
- A gating GCN attends globally to the question and schema to softly select constants, and re-ranking globally scores the selected set.The re-ranker scores the output of a generative semantic parser.Source: §5 · p. 5
- The contributions can be applied to any zero-shot semantic parser with minimal modifications, and the paper reports substantial improvement over SPIDER state of the art.The conclusion attributes the observed SPIDER improvement to the effectiveness of both contributions.Source: §5 · p. 5
A Re-implementation
The re-implementation adds matching cell values from database tables to the schema graphs.
A Re-implementation
- The implementation extracts cells from the first 5000 rows of every schema table during preprocessing.The added cell values are described as a simple modification to the implementation of Bogin et al. (2019).Source: A Re-implementation · p. 7
- A cell is considered when it has a partial match with any question word and belongs to a database column.The passage denotes a cell by q and its column by c.Source: A Re-implementation · p. 7
- Each qualifying cell becomes a graph node, connected to its column by edges in both directions.Source: A Re-implementation · p. 7
B Selected examples
Table 3 presents selected cases where the baseline is wrong and the proposed parser is correct.
B Selected examples
- The examples are selected cases, not a complete accounting of model outcomes.The passage identifies them as selected examples.Source: B Selected examples · p. 7, Table 3 · p. 8
- Table 3 contains selected examples in which the proposed parser returns a correct result where the baseline does not.Source: Table 3 · p. 8
- The table’s stated comparison is baseline errors versus corrected outputs from the proposed parser.Source: Table 3 · p. 8