PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training

Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Cicero Nogueira dos Santos, Bing Xiang

arXiv:2012.10309v1cs.CL

TL;DR

Text-to-SQL language models can miss explicit column mentions, fail to infer columns from cell values, and struggle with complex queries, while MLM captures utterance-schema alignment only implicitly. GAP combines multiple pre-training objectives with generative models that produce synthetic utterances for schemas and SQL. GAP-augmented parsers achieve state-of-the-art results on SPIDER and CRITERIA-TO-SQL.

  • Problem

    Text-to-SQL models can miss explicit column mentions, fail to infer columns from cell values, and struggle to compose complex queries; MLM models utterance-schema alignment only implicitly.

    Source: Introduction · pp. 1–2

  • Method

    GAP jointly pre-trains utterance and schema representations using multiple objectives and synthetic data generated by SQL-to-text and table-to-text models.

    Source: Introduction · p. 2, Pre-training Model · p. 3, Pre-training Data Generation · p. 4

  • Results

    GAP-augmented semantic parsers achieve state-of-the-art results on both SPIDER and CRITERIA-TO-SQL.

    Source: Introduction · p. 2, Conclusion · p. 8

  • Takeaways & Limitations

    GAP provides a pre-trained representation encoder that can be plugged into neural semantic parsers to represent utterances and schemas.

    Source: Introduction · p. 2

    The column-labeling setup assumes the generation model uses all sampled columns and does not hallucinate columns or cell values absent from its input.

    Source: Pre-training Data Generation · p. 4

Abstract

from arXiv · show

Most recently, there has been significant interest in learning contextual representations for various NLP tasks, by leveraging large scale text corpora to train large neural language models with self-supervised learning objectives, such as Masked Language Model (MLM). However, based on a pilot study, we observe three issues of existing general-purpose language models when they are applied to text-to-SQL semantic parsers: fail to detect column mentions in the utterances, fail to infer column mentions from cell values, and fail to compose complex SQL queries. To mitigate these issues, we present a model pre-training framework, Generation-Augmented Pre-training (GAP), that jointly learns representations of natural language utterances and table schemas by leveraging generation models to generate pre-train data. GAP MODEL is trained on 2M utterance-schema pairs and 30K utterance-schema-SQL triples, whose utterances are produced by generative models. Based on experimental results, neural semantic parsers that leverage GAP MODEL as a representation encoder obtain new state-of-the-art results on both SPIDER and CRITERIA-TO-SQL benchmarks.

Introduction

Text-to-SQL parsers must jointly reason over utterances and structured schemas, but general-purpose language models can miss explicit and implicit column mentions and struggle with complex queries. GAP combines multiple pre-training objectives with synthetic data generation to learn contextual representations of utterances and schemas; GAP-augmented parsers outperform state-of-the-art systems on SPIDER and CRITERIA-TO-SQL.

Introduction

  • Existing text-to-SQL models miss columns stated in utterances, fail to infer columns from cell values, and struggle to compose complex or nested queries.Examples include failing to infer that “republics” indicates GovernmentForm or selecting the wrong column in a nested query.Source: Introduction · pp. 1–2
  • MLM can model utterance-schema alignment only implicitly, motivating objectives that more directly capture alignment and domain knowledge.The authors hypothesize that these objectives are needed to address the three identified pain points.Source: Introduction · p. 2
  • GAP jointly learns utterance and schema representations through column prediction, column recovery, and SQL generation objectives.The objectives target column detection, cell-value-based column inference, and complex-query composition, respectively.Source: Introduction · p. 2
  • GAP uses SQL-to-text and table-to-text generators to create pre-training data because crawled tables and SQL lack suitable associated utterances.Surrounding table text can be noisy and differs from downstream user utterances in length, composition, and content.Source: Introduction · p. 2
  • GAP-augmented parsers outperform state-of-the-art semantic parsers on SPIDER and CRITERIA-TO-SQL.GAP produces a pre-trained model that plugs into neural parsers to represent utterances and schemas.Source: Introduction · p. 2

Models

The semantic parser encodes an utterance and table schema, then decodes a SQL program. GAP pre-trains contextual utterance-schema representations with prediction, recovery, generation, and language-modeling objectives, using synthetic utterances generated from SQLs or tables.

Text-to-SQL Semantic Parser

  • The parser encodes an utterance and table schema, then decodes a distribution over SQL programs, with targets that may be raw SQL or intermediate representations.The decoder targets include SemQL and AST trees as well as raw SQL tokens.Source: Text-to-SQL Semantic Parser · pp. 2–3

Pre-training Model

  • GAP encodes utterances with schema column names and trains representations using column prediction, column recovery, SQL generation, and masked language modeling.Column prediction labels whether a column is mentioned; column recovery reconstructs a column name from a cell value, and SQL generation trains the model to produce SQL token by token.Source: Pre-training Model · pp. 3–4

Pre-training Data Generation

  • SQL-to-Text generation creates utterances from crawled SQLs for utterance-schema-SQL triples, while Table-to-Text generation creates utterances directly from table content for utterance-schema pairs.The SQL-to-Text generator achieved 0.1934 BLEU on the development set; the Table-to-Text generator achieved 0.1821 BLEU.Source: Pre-training Data Generation · p. 4
  • Table-to-Text inputs can include sampled column names and cell values, with aggregator, structure, and order-based control codes used to shape generated utterances.The controls include aggregation operations, set-structure operators, and a LIMIT number.Source: Pre-training Data Generation · p. 4
  • The data-generation labeling scheme assumes every sampled column is used to synthesize the utterance and that the generator does not hallucinate absent column names or cell values.This assumption is used to assign positive labels to sampled columns.Source: Pre-training Data Generation · p. 4

Experiments

Experiments evaluate GAP as a contextual encoder on SPIDER and CRITERIA-TO-SQL, then use IRNet ablations and probing to examine objectives, pre-training inputs, and column-value alignment.

Configurations

  • GAP is pre-trained with BART-initialized transformers, and semantic parsers use its 12-layer encoder as their contextual encoder.The encoder replaces the original contextual encoder during downstream fine-tuning.Source: Configurations · p. 4

Tasks, Datasets and Baseline Systems

  • SPIDER contains 10,181 annotated utterance-database-SQL triples and evaluates exact set match; CRITERIA-TO-SQL has 2,003 examples and reports SQL and execution accuracy.The experiments use RAT-SQL as the SPIDER baseline and a BERT-base YXJ slot-filling system as the CRITERIA-TO-SQL baseline.Source: Tasks, Datasets and Baseline Systems · p. 5

Spider Results

  • SPIDER exact-set accuracy reaches 0.718 on development and 0.697 on hidden test, while CRITERIA-TO-SQL GAP gains 2.0% SQL and 3.6% execution accuracy over BART.The SPIDER result is reported as state of the art on the hidden test set at the time of writing.Source: Table 2 · p. 5, Spider Results · p. 5, Table 5 · p. 6
  • GAP improves over RAT-SQL+BART at every SPIDER hardness level and alleviates all three sampled error types, especially column selection and SQL composition.The error analysis sampled 40 development examples where RAT-SQL+BART failed and focused on column selection, column inference, and SQL composition.Source: Table 3 · p. 5, Spider Results · p. 6, Table 4 · p. 6

Impact of Learning Objectives

  • Without GenSQL, MLM, CPred, and CRec increase SPIDER development accuracy by 1.7%, 1.9%, and 2.5%; combining all three reaches 0.715, up 3.5%.With GenSQL, the baseline accuracy is 0.699; MLM+CRec reaches 0.728, and all three objectives improve accuracy by 2.4% over baseline.Source: Table 6 · p. 6, Impact of Learning Objectives · pp. 6–7

Analysis of Pre-Training Inputs

  • Removing utterances drops MLM accuracy from 0.697 to 0.678 and CRec from 0.705 to 0.688; removing schema lowers MLM accuracy to 0.679.Generated text outperforms surrounding table text by 1.6% for MLM and 0.8% for CRec; CRec is less affected by this substitution.Source: Table 7 · p. 7, Analysis of Pre-Training Inputs · p. 7

Analysis of Pre-trained Model

  • On Column-Value Matching, GAP MLM reaches 32.72% accuracy versus BART’s 23.17%; MLM+CRec and MLM+CPred reach 36.78% and 44.51%.The encoders are fixed during probe training, which tests whether cell values in utterances align with their columns.Source: Analysis of Pre-trained Model · pp. 7–8, Table 8 · p. 8

Related Work

Prior semantic-parsing work improves decoding, targets, encoding, and reranking, while GAP advances encoder-focused work through generation and new learning objectives. The paper also distinguishes its table-question generation approach from related generation and pre-training methods.

Semantic Parsing

  • GAP advances encoder-focused semantic parsing by using generation models and three novel learning objectives to enhance utterance-schema representations.Prior work also improves decoding mechanisms and targets or reranks generated candidates.Source: Related Work · p. 8

Question Generation and Table-to-Text Generation

  • The paper’s Table-to-Text model composes questions from sampled columns and cell values without specifying what to ask.This combines question generation and table-to-text generation.Source: Related Work · p. 8

Pre-training Models

  • Unlike related synthetic-data approaches, GAP uses generators to enhance joint utterance and structured-schema encoding.Prior work includes grammar-generated data and pre-training with existing data-to-text datasets.Source: Related Work · p. 8

Conclusion

The paper introduces generation-augmented pre-training to address three Text-to-SQL pain points and reports state-of-the-art performance on SPIDER and CRITERIA-TO-SQL.

Conclusion

  • GAP achieves state-of-the-art performance on both SPIDER and CRITERIA-TO-SQL.The framework uses four learning objectives to address three identified Text-to-SQL pain points.Source: Conclusion · p. 8
  • The paper identifies three pain points in Text-to-SQL semantic parsing and proposes a generation-augmented pre-training framework to alleviate them.The framework uses four different learning objectives.Source: Conclusion · p. 8
  • The reported effectiveness is evaluated on the SPIDER and CRITERIA-TO-SQL datasets.Source: Conclusion · p. 8

Appendices

The appendices detail synthetic pre-training data construction, training setup, and selected SPIDER examples illustrating column selection and complex-query behavior.

Pre-training Data

  • 2M utterance-table training instances are generated from 500K filtered Wikipedia tables, with four utterances produced per table.Tables with fewer than four columns or three rows are removed, along with tables failing other stated filters.Source: Pre-training Data · p. 11
  • 30K utterance-schema-SQL triples are built from parseable GitHub SQL, generated utterances, and schemas containing positive and sampled negative candidates.Each epoch combines these triples for GenSQL with 2M utterance-table pairs split among MLM, CPred, and CRec; training runs for six epochs on four Tesla V100 GPUs.Source: Pre-training Data · p. 11, Training Details · p. 11

Analysis for SPIDER Dataset

  • The examples compare RAT-SQL+BART with RAT-SQL+GAP on explicit and implicit column matching and complex-question handling.The passage reports improved column selection and an example where GAP generates a HAVING avg(LifeExpectancy) < 72 condition.Source: Analysis for SPIDER Dataset · p. 11, Table 9 · p. 12

Related papers

Loading 2012.10309v1…