Source-linked AI summary
GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, Caiming Xiong
TL;DR
Table semantic parsing requires models to connect natural-language questions with structured tables and SQL while generalizing across domains. GraPPa pre-trains a RoBERTa-based model on grammar-generated question-SQL pairs with schema-semantic supervision and on human table utterances with MLM. Across four supervised and weakly supervised tasks, it achieves new state-of-the-art results and consistently outperforms RoBERTa.
Problem
Pre-trained language models still perform limitedly on complex table semantic parsing because they face domain shift and do not explicitly model compositional utterance-schema relations.
Method
GraPPa induces an SCFG from text-to-SQL data, generates synthetic question-SQL pairs over tables, trains with a schema-linking SQL objective, and regularizes with MLM on table-language data.
Results
GraPPa achieves new state-of-the-art results across four semantic parsing tasks and outperforms RoBERTa by 3.7% on SPIDER, 1.8% on WIKITABLEQUESTIONS, and 2.4% on weakly supervised WIKISQL.
Takeaways & Limitations
Synthetic examples can inject a compositional inductive bias during language-model pre-training when MLM regularization helps balance synthetic and real table-language data.
Takeaways & Limitations
The method relies on a hand-crafted SCFG that often generates unnatural questions and may have limited logical-form coverage.
Abstract
from arXiv · showhide
We present GraPPa, an effective pre-training approach for table semantic parsing that learns a compositional inductive bias in the joint representations of textual and tabular data. We construct synthetic question-SQL pairs over high-quality tables via a synchronous context-free grammar (SCFG) induced from existing text-to-SQL datasets. We pre-train our model on the synthetic data using a novel text-schema linking objective that predicts the syntactic role of a table field in the SQL for each question-SQL pair. To maintain the model's ability to represent real-world data, we also include masked language modeling (MLM) over several existing table-and-language datasets to regularize the pre-training process. On four popular fully supervised and weakly supervised table semantic parsing benchmarks, GraPPa significantly outperforms RoBERTa-large as the feature representation layers and establishes new state-of-the-art results on all of them.
1 INTRODUCTION
GraPPa addresses limited cross-domain table semantic parsing by augmenting language-model pre-training with synthetic question-SQL pairs and table-language data. Its grammar-based linking objective targets the compositional relationship between utterances, schemas, and SQL.
- Table semantic parsing makes tabular information accessible to non-technical users by mapping natural-language queries to formal programs.
- Despite pre-trained language-model representations, complex benchmarks remain limited because models must generalize across unseen tables and databases and produce complex programs.The paper attributes this gap partly to domain shift and pre-training objectives that overlook compositional structure.
- GraPPa induces an SCFG from text-to-SQL examples, abstracts schema mentions, values, and SQL operations, and executes templates on random tables to generate synthetic pairs.
- Its text-schema linking objective predicts each table column’s syntactic role in SQL, encouraging representations that ground schema components to logical-form constituents.
- GraPPa combines SQL semantic loss on synthetic data with MLM on table-related utterances to preserve natural-language representations while injecting compositional bias.The MLM component is included to prevent overfitting to synthetic data.
2 METHODOLOGY
GraPPa uses grammar-based synthesis to create question-SQL training pairs and pre-trains a language model with complementary schema-semantic and natural-language objectives. The method combines synthetic and human-written table utterances to address overfitting while learning column-to-SQL links.
- 2.1 MOTIVATION: Semantic-parsing data is compositional because utterances correspond to formal representations such as logical forms and SQL queries.
- 2.1 MOTIVATION: Augmented data can be less beneficial in random domains because generated utterances may differ substantially from the original distribution, encouraging overfitting.
- 2.1 MOTIVATION: GraPPa uses synthetic examples during language-model pre-training rather than directly adding augmented pairs to end-task training, injecting a compositional inductive bias.The paper describes this as addressing overfitting through a separate pre-training framework.
- 2.2 DATA SYNTHESIS WITH SYNCHRONOUS CONTEXT-FREE GRAMMAR: The SCFG replaces column, table, value, and SQL-logic phrases with substitutable non-terminals, preserving an underlying SQL logic template while generating new examples.Production rules include categories such as TABLE, COLUMN, VALUE, AGG, and OP.
- 2.2 DATA SYNTHESIS WITH SYNCHRONOUS CONTEXT-FREE GRAMMAR: Applying templates to table content lets pre-training expose the model to links between values and their columns.
- 2.2 DATA SYNTHESIS WITH SYNCHRONOUS CONTEXT-FREE GRAMMAR: The synthetic corpus contains 475k question-SQL examples generated from roughly 340k filtered Wikipedia tables plus tables and databases from SPIDER and WIKISQL.
- 2.4 PRE-TRAINING GRAPPA: For each column, SSP predicts whether it appears in the SQL query and which operation it triggers, using column representations derived from the sequence encoding.
- 2.4 PRE-TRAINING GRAPPA: GRAPPA also uses human-written table utterances with MLM, while synthetic examples receive SSP, and the two groups are randomly mixed during pre-training.The final mixture contains 391k natural-language utterances and 475k synthetic examples.
3 EXPERIMENTS
The experiments evaluate table semantic parsing across four cross-domain tasks, using both fully supervised and weakly supervised settings with unseen tables or databases at test time. Results compare GRAPPA- or RoBERTa-augmented base models under task-specific evaluation protocols.
- Experimental setup: Four cross-domain tasks require models to generalize to unseen tables or databases at test time.The evaluation includes fully supervised and weakly supervised semantic parsing settings.
- Supervised semantic parsing: In supervised semantic parsing, models generate a program from a question and table or database schema.The supervised experiments cover SPIDER and fully supervised WIKISQL.
- Supervised semantic parsing: SPIDER contains 10k complex question-query pairs across 200 databases, including queries that join multiple tables through foreign keys.Performance is reported using set match accuracy, with RAT-SQL + BERT as the baseline model.
- Weakly supervised semantic parsing: Weakly supervised semantic parsing provides questions and answers while leaving the underlying meaning representations unknown.WIKITABLEQUESTIONS uses question-denotation pairs over single Wikipedia tables, with comparisons, superlatives, and aggregations among its operations.
- Implementation: The implementation fine-tunes GRAPPA for 300k steps on eight 16GB Nvidia V100 GPUs, completing pre-training in less than 10 hours.Downstream experiments use a BERT-specific optimizer for GRAPPA and RoBERTa.
4 EXPERIMENTAL RESULTS
GraPPa improves table semantic parsing across supervised and weakly supervised benchmarks, with combined MLM+SSP generally performing best. Gains are especially visible on SPIDER, WIKITABLEQUESTIONS, and low-resource settings.
- Overall results: 3.7% on SPIDER, 1.8% on WIKITABLEQUESTIONS, and 2.4% on weakly supervised WIKISQL separate GRAPPA from RoBERTa, establishing state-of-the-art results across four tasks.The comparison concerns base parsers augmented with GRAPPA versus RoBERTa.
- SPIDER: 4% over RAT-SQL+BERT-large is achieved by GRAPPA with MLM+SSP on SPIDER, producing new state-of-the-art performance.
- Fully supervised WIKISQL: 3.0% improvement over RoBERTa appears for GRAPPA in low-resource fully supervised WIKISQL.The setting uses only 10k training examples.
- WIKITABLEQUESTIONS: 1.8% over RoBERTa is obtained by GRAPPA with MLM+SSP on WIKITABLEQUESTIONS, yielding a new state-of-the-art result.The same model also performs much better than RoBERTa with only 10% of the training data.
- Weakly supervised WIKISQL: 84.7% is achieved by GRAPPA with MLM+SSP on weakly supervised WIKISQL, surpassing other baselines and reaching a new state of the art.The setting provides only answer execution results rather than gold programs.
5 ANALYSIS
The analysis finds that combining MLM and SSP is more robust than either objective alone and that SPIDER-derived grammar pre-training generalizes to other datasets. It also identifies risks from excessive pre-training and synthetic-data augmentation.
- Pre-training objectives: MLM+SSP consistently outperforms MLM-only and SSP-only pre-training, balancing preservation of language representations with compositional inductive bias.The reported gains are about +2% on SPIDER, +1.5% on WikiTableQuestions, and +1.2% on weakly supervised WikiSQL.
- Pre-training objectives: 3.9% improvement on SPIDER dev is obtained with SSP, although the grammar is described as overfitted to SPIDER.
- Generalization: GRAPPA improves WIKISQL and WIKITABLEQUESTIONS despite using an SCFG designed solely from SPIDER's data distribution.The authors interpret this as evidence of generalization to different underlying program distributions.
- Generalization: GRAPPA can benefit general table understanding even when pre-trained on SQL-specific semantics and evaluated where programs are latent.
- Pre-training time and data: Longer pre-training can hurt performance, and the best SPIDER result uses only five epochs on relatively small pre-training data.
- Pre-training versus training-data augmentation: Combining extra synthetic data with original training data has often failed to improve or has harmed semantic parsing performance.
6 RELATED WORK
Related work situates GraPPa within joint textual-tabular understanding, language-model pre-training, and grammar-based semantic-parsing data augmentation. These lines of work motivate cross-modal representations and compositional generalization.
- Textual-tabular data understanding: Joint textual-tabular understanding spans table semantic parsing, question answering, retrieval, fact-checking, and summarization.
- Textual-tabular data understanding: Better representations must contextualize input text with different table components across structured and unstructured data.
- Pre-training for NLP tasks: Pre-training with conditional language-modeling objectives has improved textual representations across downstream NLP tasks.
- Data augmentation for semantic parsing: Context-free grammar augmentation exploits the compositional relationship between utterances and formal representations to support generalization beyond training examples.
7 CONCLUSION AND FUTURE WORK
GraPPa is presented as an effective pre-training approach for table semantic parsing, combining grammar-generated question-SQL pairs with table-related language modeling. The authors report significant gains over RoBERTa while identifying several directions for future improvement.
- GraPPa uses a context-free grammar to generate question-SQL pairs and pre-trains an LM with SQL semantic loss.
- Adding MLM loss on a small amount of table-related utterances helps leverage the augmented data.
- Results on four semantic parsing tasks show that GraPPa significantly outperforms RoBERTa.
- The hand-crafted grammar often generates unnatural questions, motivating more sophisticated data augmentation techniques.
- Future work could vary MLM and SSP data proportions and expand grammar coverage and compositional semantic supervision.
A APPENDICES
The appendix documents the annotation regimes used across four semantic parsing tasks and aggregates datasets for table-and-language tasks.
- SPIDER and Fully-sup. WIKISQL require full annotation of SQL programs.
- WIKITABLEQUESTIONS and Weakly-sup. WIKISQL require annotation of answers or denotations rather than full SQL programs.
- Table 8 aggregates datasets used for table-and-language tasks.
A.1 ADDITIONAL ANALYSIS
Additional analysis examines GraPPa's training dynamics on SPIDER and relates its pre-training behavior to task-specific alignment objectives and attention visualizations.
- Training coverage: About 14%: RAT-SQL initialized with GraPPa outperforms RAT-SQL using RoBERTa during SPIDER's early training stage.
- Training coverage: 4%: GraPPa maintains a competitive edge over the baseline later in training.
- A related RAT-SQL study found that adding a similar alignment loss on SPIDER training data did not make a statistically significant difference.
- RAT-SQL initialized with GraPPa converges to higher development exact-match scores in fewer training steps than RAT-SQL with BERT.
- Figures 3 and 4 provide attention visualizations from the last self-attention layer.