PaperlayerView exact revision on arXiv

Source-linked AI summary

GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing

Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, Caiming Xiong

arXiv:2009.13845v2cs.CLcs.AI

TL;DR

Table semantic parsing requires models to connect natural-language questions with structured tables and SQL while generalizing across domains. GraPPa pre-trains a RoBERTa-based model on grammar-generated question-SQL pairs with schema-semantic supervision and on human table utterances with MLM. Across four supervised and weakly supervised tasks, it achieves new state-of-the-art results and consistently outperforms RoBERTa.

  • Problem

    Pre-trained language models still perform limitedly on complex table semantic parsing because they face domain shift and do not explicitly model compositional utterance-schema relations.

    Source: §1 · p. 1

  • Method

    GraPPa induces an SCFG from text-to-SQL data, generates synthetic question-SQL pairs over tables, trains with a schema-linking SQL objective, and regularizes with MLM on table-language data.

    Source: §1 · p. 2

  • Results

    GraPPa achieves new state-of-the-art results across four semantic parsing tasks and outperforms RoBERTa by 3.7% on SPIDER, 1.8% on WIKITABLEQUESTIONS, and 2.4% on weakly supervised WIKISQL.

    Source: §4 · p. 6

  • Takeaways & Limitations

    Synthetic examples can inject a compositional inductive bias during language-model pre-training when MLM regularization helps balance synthetic and real table-language data.

    Source: §1 · p. 2

    The method relies on a hand-crafted SCFG that often generates unnatural questions and may have limited logical-form coverage.

    Source: §2.2 · p. 4, §7 · p. 9

Abstract

from arXiv · show

We present GraPPa, an effective pre-training approach for table semantic parsing that learns a compositional inductive bias in the joint representations of textual and tabular data. We construct synthetic question-SQL pairs over high-quality tables via a synchronous context-free grammar (SCFG) induced from existing text-to-SQL datasets. We pre-train our model on the synthetic data using a novel text-schema linking objective that predicts the syntactic role of a table field in the SQL for each question-SQL pair. To maintain the model's ability to represent real-world data, we also include masked language modeling (MLM) over several existing table-and-language datasets to regularize the pre-training process. On four popular fully supervised and weakly supervised table semantic parsing benchmarks, GraPPa significantly outperforms RoBERTa-large as the feature representation layers and establishes new state-of-the-art results on all of them.

1 INTRODUCTION

GraPPa addresses limited cross-domain table semantic parsing by augmenting language-model pre-training with synthetic question-SQL pairs and table-language data. Its grammar-based linking objective targets the compositional relationship between utterances, schemas, and SQL.

  • Table semantic parsing makes tabular information accessible to non-technical users by mapping natural-language queries to formal programs.Source: §1 · p. 1
  • Despite pre-trained language-model representations, complex benchmarks remain limited because models must generalize across unseen tables and databases and produce complex programs.The paper attributes this gap partly to domain shift and pre-training objectives that overlook compositional structure.Source: §1 · p. 1
  • GraPPa induces an SCFG from text-to-SQL examples, abstracts schema mentions, values, and SQL operations, and executes templates on random tables to generate synthetic pairs.Source: §1 · p. 2
  • Its text-schema linking objective predicts each table column’s syntactic role in SQL, encouraging representations that ground schema components to logical-form constituents.Source: §1 · p. 2
  • GraPPa combines SQL semantic loss on synthetic data with MLM on table-related utterances to preserve natural-language representations while injecting compositional bias.The MLM component is included to prevent overfitting to synthetic data.Source: §1 · p. 2

2 METHODOLOGY

GraPPa uses grammar-based synthesis to create question-SQL training pairs and pre-trains a language model with complementary schema-semantic and natural-language objectives. The method combines synthetic and human-written table utterances to address overfitting while learning column-to-SQL links.

2.1 MOTIVATION

  • Semantic-parsing data is compositional because utterances correspond to formal representations such as logical forms and SQL queries.Source: §2.1 · p. 2
  • Augmented data can be less beneficial in random domains because generated utterances may differ substantially from the original distribution, encouraging overfitting.Source: §2.1 · pp. 2–3
  • GraPPa uses synthetic examples during language-model pre-training rather than directly adding augmented pairs to end-task training, injecting a compositional inductive bias.The paper describes this as addressing overfitting through a separate pre-training framework.Source: §2.1 · p. 3

2.2 DATA SYNTHESIS WITH SYNCHRONOUS CONTEXT-FREE GRAMMAR

  • The SCFG replaces column, table, value, and SQL-logic phrases with substitutable non-terminals, preserving an underlying SQL logic template while generating new examples.Production rules include categories such as TABLE, COLUMN, VALUE, AGG, and OP.Source: §2.2 · p. 3, Table 1 · p. 3
  • Applying templates to table content lets pre-training expose the model to links between values and their columns.Source: §2.2 · p. 4
  • The synthetic corpus contains 475k question-SQL examples generated from roughly 340k filtered Wikipedia tables plus tables and databases from SPIDER and WIKISQL.Source: §2.2 · p. 4

2.4 PRE-TRAINING GRAPPA

  • For each column, SSP predicts whether it appears in the SQL query and which operation it triggers, using column representations derived from the sequence encoding.Source: §2.4 · p. 4
  • GRAPPA also uses human-written table utterances with MLM, while synthetic examples receive SSP, and the two groups are randomly mixed during pre-training.The final mixture contains 391k natural-language utterances and 475k synthetic examples.Source: §2.3 · p. 4, §2.4 · p. 5

3 EXPERIMENTS

The experiments evaluate table semantic parsing across four cross-domain tasks, using both fully supervised and weakly supervised settings with unseen tables or databases at test time. Results compare GRAPPA- or RoBERTa-augmented base models under task-specific evaluation protocols.

Experimental setup

  • Four cross-domain tasks require models to generalize to unseen tables or databases at test time.The evaluation includes fully supervised and weakly supervised semantic parsing settings.Source: §3 · p. 5

Supervised semantic parsing

  • In supervised semantic parsing, models generate a program from a question and table or database schema.The supervised experiments cover SPIDER and fully supervised WIKISQL.Source: §3.1 · p. 5
  • SPIDER contains 10k complex question-query pairs across 200 databases, including queries that join multiple tables through foreign keys.Performance is reported using set match accuracy, with RAT-SQL + BERT as the baseline model.Source: §3.1 · p. 5

Weakly supervised semantic parsing

  • Weakly supervised semantic parsing provides questions and answers while leaving the underlying meaning representations unknown.WIKITABLEQUESTIONS uses question-denotation pairs over single Wikipedia tables, with comparisons, superlatives, and aggregations among its operations.Source: §3.2 · p. 5

Implementation

  • The implementation fine-tunes GRAPPA for 300k steps on eight 16GB Nvidia V100 GPUs, completing pre-training in less than 10 hours.Downstream experiments use a BERT-specific optimizer for GRAPPA and RoBERTa.Source: §3.3 · p. 6

4 EXPERIMENTAL RESULTS

GraPPa improves table semantic parsing across supervised and weakly supervised benchmarks, with combined MLM+SSP generally performing best. Gains are especially visible on SPIDER, WIKITABLEQUESTIONS, and low-resource settings.

Overall results

  • 3.7% on SPIDER, 1.8% on WIKITABLEQUESTIONS, and 2.4% on weakly supervised WIKISQL separate GRAPPA from RoBERTa, establishing state-of-the-art results across four tasks.The comparison concerns base parsers augmented with GRAPPA versus RoBERTa.Source: §4 · p. 6

SPIDER

  • 4% over RAT-SQL+BERT-large is achieved by GRAPPA with MLM+SSP on SPIDER, producing new state-of-the-art performance.Source: §4 · p. 7

Fully supervised WIKISQL

  • 3.0% improvement over RoBERTa appears for GRAPPA in low-resource fully supervised WIKISQL.The setting uses only 10k training examples.Source: §4 · p. 7

WIKITABLEQUESTIONS

  • 1.8% over RoBERTa is obtained by GRAPPA with MLM+SSP on WIKITABLEQUESTIONS, yielding a new state-of-the-art result.The same model also performs much better than RoBERTa with only 10% of the training data.Source: §4 · p. 7

Weakly supervised WIKISQL

  • 84.7% is achieved by GRAPPA with MLM+SSP on weakly supervised WIKISQL, surpassing other baselines and reaching a new state of the art.The setting provides only answer execution results rather than gold programs.Source: §4 · p. 7

5 ANALYSIS

The analysis finds that combining MLM and SSP is more robust than either objective alone and that SPIDER-derived grammar pre-training generalizes to other datasets. It also identifies risks from excessive pre-training and synthetic-data augmentation.

Pre-training objectives

  • MLM+SSP consistently outperforms MLM-only and SSP-only pre-training, balancing preservation of language representations with compositional inductive bias.The reported gains are about +2% on SPIDER, +1.5% on WikiTableQuestions, and +1.2% on weakly supervised WikiSQL.Source: §5 · pp. 7–8
  • 3.9% improvement on SPIDER dev is obtained with SSP, although the grammar is described as overfitted to SPIDER.Source: §5 · p. 7

Generalization

  • GRAPPA improves WIKISQL and WIKITABLEQUESTIONS despite using an SCFG designed solely from SPIDER's data distribution.The authors interpret this as evidence of generalization to different underlying program distributions.Source: §5 · p. 8
  • GRAPPA can benefit general table understanding even when pre-trained on SQL-specific semantics and evaluated where programs are latent.Source: §5 · p. 8

Pre-training time and data

  • Longer pre-training can hurt performance, and the best SPIDER result uses only five epochs on relatively small pre-training data.Source: §5 · p. 8

Pre-training versus training-data augmentation

  • Combining extra synthetic data with original training data has often failed to improve or has harmed semantic parsing performance.Source: §5 · p. 8

6 RELATED WORK

Related work situates GraPPa within joint textual-tabular understanding, language-model pre-training, and grammar-based semantic-parsing data augmentation. These lines of work motivate cross-modal representations and compositional generalization.

Textual-tabular data understanding

  • Joint textual-tabular understanding spans table semantic parsing, question answering, retrieval, fact-checking, and summarization.Source: §6 · p. 8
  • Better representations must contextualize input text with different table components across structured and unstructured data.Source: §6 · p. 9

Pre-training for NLP tasks

  • Pre-training with conditional language-modeling objectives has improved textual representations across downstream NLP tasks.Source: §6 · p. 9

Data augmentation for semantic parsing

  • Context-free grammar augmentation exploits the compositional relationship between utterances and formal representations to support generalization beyond training examples.Source: §6 · p. 9

7 CONCLUSION AND FUTURE WORK

GraPPa is presented as an effective pre-training approach for table semantic parsing, combining grammar-generated question-SQL pairs with table-related language modeling. The authors report significant gains over RoBERTa while identifying several directions for future improvement.

  • GraPPa uses a context-free grammar to generate question-SQL pairs and pre-trains an LM with SQL semantic loss.Source: §7 · p. 9
  • Adding MLM loss on a small amount of table-related utterances helps leverage the augmented data.Source: §7 · p. 9
  • Results on four semantic parsing tasks show that GraPPa significantly outperforms RoBERTa.Source: §7 · p. 9
  • The hand-crafted grammar often generates unnatural questions, motivating more sophisticated data augmentation techniques.Source: §7 · p. 9
  • Future work could vary MLM and SSP data proportions and expand grammar coverage and compositional semantic supervision.Source: §7 · p. 9

A APPENDICES

The appendix documents the annotation regimes used across four semantic parsing tasks and aggregates datasets for table-and-language tasks.

  • SPIDER and Fully-sup. WIKISQL require full annotation of SQL programs.Source: Table 7 · p. 13
  • WIKITABLEQUESTIONS and Weakly-sup. WIKISQL require annotation of answers or denotations rather than full SQL programs.Source: Table 7 · p. 13
  • Table 8 aggregates datasets used for table-and-language tasks.Source: Table 8 · p. 13

A.1 ADDITIONAL ANALYSIS

Additional analysis examines GraPPa's training dynamics on SPIDER and relates its pre-training behavior to task-specific alignment objectives and attention visualizations.

Training coverage

  • About 14%: RAT-SQL initialized with GraPPa outperforms RAT-SQL using RoBERTa during SPIDER's early training stage.Source: §A.1 · p. 13
  • 4%: GraPPa maintains a competitive edge over the baseline later in training.Source: §A.1 · p. 13
  • A related RAT-SQL study found that adding a similar alignment loss on SPIDER training data did not make a statistically significant difference.Source: §A.1 · p. 13
  • RAT-SQL initialized with GraPPa converges to higher development exact-match scores in fewer training steps than RAT-SQL with BERT.Source: Figure 2 · p. 14
  • Figures 3 and 4 provide attention visualizations from the last self-attention layer.Source: Figure 3 · p. 15, Figure 4 · p. 16

Related papers

Loading 2009.13845v2…