PaperlayerView exact revision on arXiv

Source-linked AI summary

Structure-Grounded Pretraining for Text-to-SQL

Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, Matthew Richardson

arXiv:2010.12773v3cs.CLcs.AI

TL;DR

Text-to-SQL models must align natural-language references with database schemas, especially when users do not repeat exact column names. STRUG uses weakly supervised, structure-grounded pretraining on parallel text-table data, and improves over BERT-LARGE across evaluations while matching GRAPPA on Spider and outperforming baselines on more realistic sets.

  • Problem

    Text-to-SQL requires grounding varied column and value references in schemas, while common benchmarks explicitly mention column names and can understate this alignment challenge.

  • Method

    STRUG pretrains a text-table encoder with column grounding, value grounding, and column-value mapping objectives using weak supervision from the ToTTo parallel text-table corpus.

  • Results

    STRUG significantly outperforms BERT-LARGE in all evaluation settings, matches GRAPPA on Spider, and outperforms all baselines on more realistic evaluation sets.

  • Takeaways & Limitations

    Parallel text-table pretraining transfers alignment knowledge to text-to-SQL and can reduce reliance on large amounts of costly supervised training data.

  • Takeaways & Limitations

    One reported setting excludes examples where no column is compared against an entity.

Abstract

from arXiv · show

Learning to capture text-table alignment is essential for tasks like text-to-SQL. A model needs to correctly recognize natural language references to columns and values and to ground them in the given database schema. In this paper, we present a novel weakly supervised Structure-Grounded pretraining framework (StruG) for text-to-SQL that can effectively learn to capture text-table alignment based on a parallel text-table corpus. We identify a set of novel prediction tasks: column grounding, value grounding and column-value mapping, and leverage them to pretrain a text-table encoder. Additionally, to evaluate different methods under more realistic text-table alignment settings, we create a new evaluation set Spider-Realistic based on Spider dev set with explicit mentions of column names removed, and adopt eight existing text-to-SQL datasets for cross-database evaluation. STRUG brings significant improvement over BERT-LARGE in all settings. Compared with existing pretraining methods such as GRAPPA, STRUG achieves similar performance on Spider, and outperforms all baselines on more realistic sets. The Spider-Realistic dataset is available at https://doi.org/10.5281/zenodo.5205322.

1 Introduction

Text-to-SQL depends on aligning natural-language references to columns and values with database schemas, but existing benchmarks can make this challenge unrealistically easy. STRUG addresses the gap by pretraining on parallel text-table data and evaluating under reduced column-name cues.

  • Motivation: Text-table alignment requires identifying varied column and value references and grounding them in schemas, including when database cell access is unavailable.The challenge also includes generalizing to unseen schemas and referential language.
  • Motivation: Parallel text-table corpora provide alignment knowledge with less human effort than text-to-SQL datasets.Cell contents or annotations can identify links between value mentions and columns.
  • Approach: STRUG pretrains a text-table encoder with column grounding, value grounding, and column-value mapping objectives using weak labels from ToTTo.Labels come from revised descriptions and cell annotations or from string matching against table contents.
  • Evaluation: Spider-Realistic removes explicit column-name mentions from Spider development utterances to test text-table alignment under more realistic conditions.The paper also evaluates cross-database generalization using existing datasets.
  • Results: RAT-SQL with STRUG significantly outperforms BERT-LARGE and performs on par with GRAPPA on Spider.This comparison is reported for the widely used Spider benchmark.
  • Results: STRUG outperforms all baselines on Spider-Realistic and Suhr et al.'s datasets, while improving performance with limited supervised WikiSQL data.Pretraining uses 120k text-table pairs and typically converges in about five epochs in under four hours.

2 Related Work

Related work addresses cross-database generalization and schema linking, but STRUG learns text-table alignment directly from parallel text-table corpora rather than relying only on heuristics or supervised alignment labels.

  • Cross-Database Text-to-SQL: Cross-database text-to-SQL seeks generalization because annotating natural-language questions with SQL is expensive across possible databases.Existing models can exceed 80% exact matching accuracy on some single-domain benchmarks with sufficient in-domain data.
  • Cross-Database Text-to-SQL: STRUG targets text-table alignment directly through structure-grounded pretraining on parallel text-table corpora.This contrasts with approaches that ease alignment through other mechanisms.
  • Structure Grounding in Text-to-SQL: Schema-linking systems use heuristics or manually annotated fine-grained alignment labels, but heuristic matching may not generalize to varied column references.Supervised alignment annotations have nevertheless brought significant improvements.

3 Structure-Grounded Pretraining

STRUG pretrains a text-table encoder to address cross-database text-table alignment by learning column grounding, value grounding, and column-value mapping from parallel text-table data. It uses weak supervision from ToTTo and combines the three task losses for downstream integration.

  • Motivation: Text-table alignment requires linking natural-language references to columns and values in unseen database schemas.This challenge is central to cross-database text-to-SQL generalization.
  • Pretraining Framework: STRUG learns alignment knowledge through three structure-grounded tasks: column grounding, value grounding, and column-value mapping.These tasks are designed to capture text-table alignment directly during pretraining, unlike generic unsupervised objectives such as MLM.
  • Pretraining Objectives: The pretraining loss sums the column, value, and column-value mapping losses, and the reported experiments use equally weighted terms.Changing the relative weights did not produce significant improvement.
  • Weak Supervision: ToTTo supplies 120,761 text-table pairs with cell-level annotations and revised descriptions for deriving grounding labels.The labels identify columns containing highlighted cells and match cell values to phrases in descriptions.
  • Weak Supervision: STRUG supports human-assisted and automatic supervision settings, with automatic cell annotations obtained by exact string matching.In both settings, cell values prepare supervision but are not model inputs during pretraining.
  • Data Augmentation: Negative tables and column-name augmentation simulate databases containing multiple tables and many columns.The method randomly appends columns from Kneg sampled tables and replaces matched phrases to improve generalization.

4 Creating a More Realistic Evaluation Set

Spider-Realistic is constructed to evaluate text-table alignment when questions do not explicitly repeat schema column names. The set modifies selected Spider development questions while preserving their SQL queries and avoids introducing new query-structure challenges.

  • Motivation: Spider makes text-table alignment easier by explicitly mentioning column names in utterances.Real users may refer to columns with expressions that differ from the original schema names.
  • Construction: Spider-Realistic removes or paraphrases explicit column-name mentions in selected Spider dev questions while keeping SQL queries unchanged.Columns in SELECT clauses are excepted from the modification.
  • Construction: The modifications target columns compared against values or used in clauses such as ORDER BY.This focuses evaluation on alignment without adding challenges from new query structures.
  • Purpose: Spider-Realistic is designed to assess text-table alignment fairly without introducing extra query-structure challenges.The underlying SQL queries remain unchanged after the natural-language edits.

5 Experiments

Experiments evaluate STRUG with RAT-SQL and SQLova on Spider, realistic cross-database sets, and WikiSQL under limited-data settings. STRUG consistently improves over BERT-LARGE, with especially strong gains when explicit schema clues or abundant supervision are reduced.

  • Benchmarks and Base Models: Table 3 reports execution accuracy on Spider-Realistic and Suhr et al. evaluation sets, with exact match additionally reported for Spider-Realistic.The Suhr et al. results use filtered sets excluding examples with empty query results, and comparisons also distinguish training-data settings.
  • Experimental Setup: Experiments use BERT-LARGE, Adam pretraining for five epochs on four V100 GPUs, RAT-SQL for Spider, and SQLova for WikiSQL.Pretraining takes less than four hours; WikiSQL models train for up to 50 epochs and select the best checkpoint on the dev set.
  • Main Results: STRUG consistently improves RAT-SQL over BERT-LARGE on Spider, matches GRAPPA when database content is available, and corrects 76 of GRAPPA’s 270 wrong predictions.The Spider table separates schema-only and database-content settings and reports mean and standard deviation across three seeds.
  • Main Results: On realistic evaluation sets, STRUG improves over BERT-LARGE, exceeds GRAPPA by 2.9% on Spider-Realistic, and outperforms Suhr et al. across all datasets using only Spider training data.These settings remove or reduce explicit column-name clues, making alignment knowledge more relevant.
  • Main Results: With only 1% of WikiSQL training data, STRUG exceeds 0.70 accuracy and outperforms BERT-LARGE and TaBERT by a large margin.STRUG’s advantage over BERT-LARGE persists until half of the training data, while all models approach full-data performance; with 5% data, STRUG also accelerates training.

6 Conclusion

The conclusion presents STRUG as a weakly supervised structure-grounded pretraining method for text-to-SQL. It reports consistent gains over BERT-LARGE, competitive Spider performance with larger pretraining approaches, and future plans to expand the corpus and incorporate additional signals.

  • 6 Conclusion: STRUG uses novel prediction tasks over parallel text-table data to address text-table alignment in text-to-SQL.The method obtains pretraining labels through human-labeled cell associations or table-content heuristics, without complex SQL query annotation.
  • 6 Conclusion: STRUG significantly outperforms BERT-LARGE across evaluation sets while using only 120k text-table pairs for pretraining.On Spider, it performs on par with TaBERT and GRAPPA, which use substantially larger or additional pretraining resources.
  • 6 Conclusion: The authors identify STRUG as complementary to existing text-table pretraining methods and plan to enlarge the corpus and incorporate MLM and synthetic data.

Ethical Considerations

The paper describes text-to-SQL as a natural-language interface task intended to let non-experts query databases. It evaluates structure grounding in standard and more realistic datasets and reports stronger improvements in realistic settings.

  • Application: Text-to-SQL is presented as a fundamental capability for natural-language database interfaces that can let non-experts query databases.
  • Application: The study focuses on improving structure grounding, which links natural-language column and value mentions to database schema elements.
  • Evaluation and Computing Cost: Evaluation covers Spider and several more realistic datasets, with significant improvement over baselines especially in realistic settings.The reported computing setup uses four V100 GPUs for pretraining and one V100 GPU for text-to-SQL fine-tuning.

A.1 Filtering on the Suhr et al. (2020) Datasets

The Suhr et al. evaluation uses filtering rules that remove examples requiring unavailable value copying, empty results, or multiple final columns. These rules define a narrower evaluation scope.

  • Filtering Criteria: The filtering removes examples whose numeric or text query values cannot be copied from the utterance, except numbers 0 and 1.
  • Filtering Criteria: Examples with empty query results or count queries returning [1] are excluded from the filtered evaluation set.
  • Filtering Criteria: Queries requiring selection of more than one final column are removed.

B.1 Detailed Results on Spider and Spider-Realistic

STRUG improves Spider performance across difficulty levels and particularly strengthens structure-sensitive clauses in realistic, schema-only settings. Its gains are especially pronounced for GROUP BY on Spider-Realistic.

  • STRUG brings significant improvements across all Spider difficulty levels without favoring a particular subset.
  • STRUG mainly improves WHERE and ORDER BY accuracy, especially when database content is unavailable.
  • Table 9 reports F1 scores for component matching on the Spider-Realistic set.
  • STRUG yields greater improvement on Spider-Realistic, especially for GROUP BY clauses.

B.2 Detailed Results on WikiSQL

On WikiSQL, STRUG improves WHERE-related subtasks and retains strong performance when training data is severely limited. With only 1% of training data, it already achieves high WHERE-column and WHERE-value accuracy.

  • With only 1% of training data, STRUG exceeds 0.87 WHERE column accuracy and approaches 0.85 WHERE value accuracy.
  • STRUG mainly improves WHERE column and WHERE value accuracy on WikiSQL.
  • Figures 6 and 7 report WikiSQL model performance across training-data fractions and during training with 5% of the data.
Loading 2010.12773v3…