PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?

Peter Shaw, Ming-Wei Chang, Panupong Pasupat, Kristina Toutanova

arXiv:2010.12725v2cs.CL

TL;DR

Semantic parsing systems must handle both natural language variation and out-of-distribution compositional generalization, but prior evaluations and methods usually emphasize only one. The paper introduces TMCD splits and NQG-T5, which performs strongly across diverse evaluations, while the joint problem remains open.

  • Problem

    The paper addresses limited evidence about semantic parsers that handle both natural language variation and out-of-distribution compositional generalization.

  • Method

    The paper proposes TMCD splits for non-synthetic datasets and NQG-T5, a hybrid combining a grammar-based approach with pretrained T5.

  • Results

    Across five approaches and eight SCAN and GEOQUERY evaluations, NQG-T5 achieves average rank 1 versus 2.9 for T5, the best previous approach.

  • Takeaways & Limitations

    The study highlights the importance of diverse evaluations while identifying joint handling of compositional generalization and natural language variation as an open challenge.

  • Takeaways & Limitations

    Accuracy remains limited on out-of-distribution examples where NQG lacks coverage, and grammar induction is particularly unsuccessful for SPIDER.

Abstract

from arXiv · show

Sequence-to-sequence models excel at handling natural language variation, but have been shown to struggle with out-of-distribution compositional generalization. This has motivated new specialized architectures with stronger compositional biases, but most of these approaches have only been evaluated on synthetically-generated datasets, which are not representative of natural language variation. In this work we ask: can we develop a semantic parsing approach that handles both natural language variation and compositional generalization? To better assess this capability, we propose new train and test splits of non-synthetic datasets. We demonstrate that strong existing approaches do not perform well across a broad set of evaluations. We also propose NQG-T5, a hybrid model that combines a high-precision grammar-based approach with a pre-trained sequence-to-sequence model. It outperforms existing approaches across several compositional generalization challenges on non-synthetic data, while also being competitive with the state-of-the-art on standard evaluations. While still far from solving this problem, our study highlights the importance of diverse evaluations and the open challenge of handling both compositional generalization and natural language variation in semantic parsing.

1 Introduction

The paper asks whether semantic parsing can combine natural language variation with out-of-distribution compositional generalization, arguing that existing evaluations and approaches do not adequately cover both. It introduces broader non-synthetic evaluations and NQG-T5 as a hybrid response, while acknowledging that the challenge remains open.

  • Motivation: Seq2seq models handle natural language variation well but often generalize poorly to novel compositional utterances.Compositional generalization concerns novel combinations of elements observed during training.
  • Motivation: Most specialized compositional-generalization approaches are evaluated only on synthetic datasets, which are less representative of real-world language variation.Synthetic datasets enable precise evaluation of specific phenomena but do not capture the breadth of human-generated utterances.
  • Research question: The paper asks whether one semantic parsing approach can handle both natural language variation and compositional generalization.The motivation includes the need to evaluate non-synthetic and out-of-distribution examples because random splits can overestimate real-world performance.
  • Contributions: The authors propose TMCD splits for non-synthetic datasets and show that strong existing approaches do not perform well across diverse evaluations.TMCD splits complement existing evaluations by focusing on different aspects of the problem.
  • Contributions: NQG-T5 combines a grammar-based NQG approach with T5, improving compositional-generalization evaluations while remaining competitive on standard evaluations.Across five approaches and eight SCAN and GEOQUERY evaluations, NQG-T5 has average rank 1 versus 2.9 for the best previous approach.
  • Conclusion: The study concludes that diverse evaluations are important and that jointly handling compositional generalization and natural language variation remains an open challenge.The paper states that it is still far from affirmatively answering its research question.

2 Background and Related Work

Prior work includes synthetic compositional-generalization benchmarks, non-synthetic template splits, specialized architectures, pre-training, and data augmentation. These approaches improve particular evaluations but leave open the broader challenge of combining compositional generalization with natural language variation.

  • Synthetic evaluations: SCAN evaluates compositional generalization using commands mapped to action sequences, including length and primitive splits.The length split tests longer examples, while the primitive split tests recombinations such as “jump twice.”
  • Synthetic evaluations: Other synthetic benchmarks extend compositional-generalization evaluation beyond SCAN, including NACS, CFQ, and COGS.CFQ also motivates Maximum Compound Divergence splits.
  • Non-synthetic evaluations: Template splits for non-synthetic datasets prevent a parse template seen in training from appearing in test data.Templates are defined from target SQL queries with entities anonymized.
  • Approaches: Specialized architectures can reach 100% accuracy on several SCAN challenges, but they have been evaluated only on synthetic parsing tasks.These models, like NQG-T5, incorporate discrete structure.
  • Approaches: Herzig and Berant’s approach addresses both challenge types but requires manual task-specific engineering.It solves several SCAN challenges and performs strongly on standard and template GEOQUERY splits.
  • Approaches: Pre-training alone cannot solve several compositional-generalization challenges despite its effectiveness across other NLP tasks.The cited finding concerns large-scale pre-training and SCAN-style challenges.
  • Approaches: Data augmentation methods have also been proposed, and NQG-T5 outperforms previously reported results for these methods.The paper notes that more in-depth analysis is needed.

3 Target Maximum Compound Divergence (TMCD) Splits

TMCD extends compound-divergence-based splitting to non-synthetic datasets by defining compounds from target representations. Its construction preserves atom coverage while approximately maximizing divergence between training and test compound distributions.

  • Motivation: TMCD adds a non-synthetic compositional-generalization evaluation beyond existing template and length splits.The resulting generalization problem can be much more challenging than template splits and contributes to more comprehensive evaluation.
  • Definition: Unlike standard MCD, TMCD defines compounds using only the known syntactic structure of target representations.For example, FunQL atoms include “longest” and “river,” while longest(river) is a compound.
  • Definition: TMCD uses the same compound-divergence definition as Keysers et al., based on weighted compound-frequency distributions in training and test sets.The divergence uses a Chernoff coefficient with α = 0.1.
  • Split construction: TMCD constrains atom divergence by requiring every atom to appear at least once in the training set.This ensures that the model knows the possible target atoms to generate.
  • Split construction: A greedy procedure first randomly splits data, then swaps examples to satisfy atom coverage and approximately maximize compound divergence.The passage describes the initial stages of the split-generation algorithm.

4 Proposed Approach: NQG-T5

NQG-T5 combines a grammar-based semantic parser with T5, using NQG when it can derive an output and T5 otherwise. NQG induces a quasi-synchronous grammar without task-specific heuristics or precomputed alignments, then scores derivations with a neural parsing model.

  • NQG-T5: NQG-T5 combines NQG’s grammar-based parser with T5, returning T5’s output when NQG fails to produce one.The components are trained separately.
  • NQG Grammar Formalism: QCFG parsing supports efficient CKY-like inference without requiring grammar binarization.The formalism disallows unary source productions and allows two non-terminal symbols in the relevant source rules.
  • NQG Grammar Induction: NQG induces a quasi-synchronous context-free grammar over arbitrary source-target string pairs without task-specific heuristics or precomputed word alignments.Its formalism permits repeated target-side non-terminals with the same index.
  • NQG Grammar Induction: NQG grammar induction starts with one rule per training example and greedily adds rules that reduce codelength while preserving derivability of the training data.The objective encodes non-terminal and terminal symbols using average bitlength hyperparameters.
  • NQG Semantic Parsing Model: A discriminative latent-variable parser scores grammar derivations with contextualized BERT representations and uses dynamic programming to compute the exact MML objective.Inference selects the highest-scoring derivation with a CKY-like algorithm and outputs its target when it satisfies the task’s valid-target grammar.

5 Experiments

The experiments evaluate NQG-T5 and existing approaches across diverse synthetic and non-synthetic splits of SCAN, GEOQUERY, and SPIDER. NQG-T5 improves compositional generalization while retaining competitive performance on standard or natural-language-variation evaluations, although substantial room for improvement remains.

  • Experimental Setup: The study compares approaches across diverse evaluations targeting compositional generalization, natural language variation, or both.Main experiments use multiple splits of SCAN and GEOQUERY, with additional evaluation on SPIDER.
  • Experimental Setup: SCAN evaluation includes length, primitive, and MCD splits, while GEOQUERY includes standard, length, TMCD, and newly generated template splits.The GEOQUERY compositional splits contain 440 training and 440 test examples each for length and TMCD.
  • SCAN and GEOQUERY Results: Performance rankings on SCAN compositional splits are not very predictive of rankings on corresponding GEOQUERY splits.GGPS outperforms T5 on SCAN’s length split but performs significantly worse on GEOQUERY’s length split.
  • SCAN and GEOQUERY Results: NQG-T5 achieves average rank 1.0 across five approaches and eight SCAN and GEOQUERY evaluations, compared with 2.9 for T5.It improves over T5 on compositional generalization while maintaining T5’s performance on in-distribution natural language variation.
  • Discussion: The results leave substantial room for improvement in handling compositional generalization and natural language variation together.
  • SPIDER Results: On SPIDER-SSP, NQG-T5 remains competitive with T5 despite NQG modeling text-to-SQL poorly because SQL has complex syntax.On SPIDER-XSP, T5-3B and NQG-T5 are competitive with state-of-the-art approaches without database contents beyond table and column names.

6 Analysis

The analysis compares evaluation splits and examines why T5 and NQG succeed or fail across compositional generalization and natural-language parsing settings. It finds distinct strengths in grammar-based precision, seq2seq coverage, and the limits imposed by grammar induction and formalism expressiveness.

  • 6.1 Comparison of Data Splits: 53.9% versus 61.6% T5-Base accuracy separates SPIDER template examples with unseen atoms from the remainder.The unseen-atom subset comprises 30.3% of the test set.
  • 6.1 Comparison of Data Splits: TMCD is significantly more challenging than the template split for GEOQUERY, while the two splits are similarly challenging for SPIDER.Length splits are also challenging but produce a more predictable seq2seq error pattern.
  • 6.2 T5 Analysis: T5 outputs are not significantly longer than the maximum training length, producing poor performance on length splits.This creates a consistent error pattern in T5’s test-set predictions.
  • 6.2 T5 Analysis: 26% of incorrect GEOQUERY TMCD predictions first diverge where a trigram training language model assigns the gold symbol zero probability.This suggests decoder overfitting to training target-sequence distributions; 53% of errors involve an atom seen only once during training.
  • 6.3 NQG Analysis: NQG has high precision but limited coverage, while NQG-T5 uses T5 when grammar induction produces no output.For SPIDER, induction often yields rules that replace names or literals rather than representing nested structures, reflecting difficult syntax and weak span correspondence.
  • 6.3 NQG Analysis: NQG is limited by QCFG expressiveness and greedy grammar induction, which cannot directly represent some source or target relations that could support broader generalization.Examples include semantic similarity and logical equivalence between target strings.

7 Conclusions

The conclusion presents NQG-T5 as a simple combination of grammar-based precision and T5 coverage, while emphasizing that diverse evaluation and joint handling of variation and compositionality remain unresolved challenges.

  • 7 Conclusions: NQG generally has higher precision on out-of-distribution examples, whereas T5 has broader coverage but significantly lower precision there.NQG requires exact lexical overlap with induced rules for an inference-time derivation.
  • 7 Conclusions: NQG-T5 combines these strengths by returning T5’s output when NQG lacks coverage and provides a strong, simple baseline for future work.Its accuracy remains limited on out-of-distribution examples where NQG cannot produce coverage.
  • 7 Conclusions: Diverse benchmarks are important, and handling both out-of-distribution compositional generalization and natural language variation remains an open challenge for semantic parsing.The conclusion frames this as a broader unresolved research problem rather than a solved capability.

Ethical Considerations

The paper’s supported scope is limited to English semantic parsing datasets, and its NQG-T5 implementation relies on computationally expensive pre-trained T5 models. The surrounding methodological material defines the grammar framework used by NQG.

  • Ethical Considerations: The methods were evaluated only on English semantic parsing datasets, leaving generalization to languages underrepresented in small training sets untested.The authors present broader cross-language generalization as a hope rather than an evaluated result.
  • Ethical Considerations: NQG-T5 uses a pre-trained T5 model whose fine-tuning and inference are computationally expensive, especially at larger scales.NQG-T5-base can reduce resources relative to T5-3B on non-SQL datasets.
  • Ethical Considerations: QCFGs generalize synchronous context-free grammars by relaxing strict one-to-one alignment between source and target non-terminals.The grammar formalism captures a compositional mapping between source and target structures.

A.2 NQG Grammar Induction Details

NQG induces a compact transduction grammar through greedy, codelength-driven rule refinement while preserving derivability of all training examples. It then uses neural scores over anchored grammar rules for parsing.

  • Initialization: NQG initializes rules from training examples and adds identity rules for source-target substrings that exactly match.These initialization rules apply to GEOQUERY and SPIDER, but SCAN has no exact token overlap between its source and target languages.
  • Greedy grammar induction: The greedy search repeatedly adds a candidate split rule when it enables redundant rules to be removed while preserving every training derivation.The search objective is codelength reduction, and it stops when no candidate rules remain.
  • Greedy grammar induction: SPLIT replaces paired source-target substrings with indexed non-terminals, producing two rules that preserve aligned structure.For example, a rule pairing “largest state” with “largest ( state )” becomes rules for “largest NT[1]” and “state.”
  • Greedy grammar induction: NEW proposes splits only when one resulting rule is already derivable, while ELIM identifies rules made redundant by the candidate.The replacement rules maintain the invariant that every training pair remains derivable.
  • Search assumptions: Repeated target non-terminals are optionally allowed for SCAN and SPIDER, but not GEOQUERY because FunQL does not require such repetitions.The option avoids ambiguous choices when repeated substrings occur in target strings.
  • Search implementation: The induction procedure limits candidate generation to the shortest N training examples, then adds rules for longer examples that remain underivable.Candidate counts are polynomial in source and target length, motivating this efficiency strategy.
  • Neural parsing scores: Neural derivation scores decompose over anchored grammar rules and use BERT span representations combined with rule-index embeddings.The first and last wordpiece representations define the span input to feed-forward scoring networks.

B Experimental Details

The experiments use limited tuning and dataset-specific model capacities, with NQG and T5 trained under substantially different computational budgets. Training configurations vary across GEOQUERY, SPIDER, and SCAN.

  • Tuning: Hyperparameters for T5 and NQG were minimally tuned using random training-set splits, with the same settings reused across splits within each dataset.The tuning procedure used GEOQUERY and SPIDER; SCAN-specific tuning is not described here.
  • T5: T5 uses a learning rate of 1e−4, default fine-tuning settings otherwise, and 3,000 or 10,000 training steps for GEOQUERY and SPIDER.On a random GeoQuery split, T5-Base reached 94.2% accuracy after 3,000 steps.
  • NQG: NQG uses BERT Tiny for SCAN and SPIDER, BERT Base for GEOQUERY, d = 256, 256 fine-tuning steps, learning rate 1e−4, and batch size 256.BERT parameters are fine-tuned rather than frozen.
  • Computational budget: NQG training uses 8 V100 GPUs and takes under 5 minutes for SCAN and SPIDER, versus up to 90 minutes for GEOQUERY.T5 fine-tuning takes approximately 5–77 hours depending on dataset and model size.

B.2 Dataset Preprocessing

Preprocessing converts GEOQUERY and SPIDER outputs into structured representations while preserving dataset-specific input information. The appendix also documents schema truncation and reporting details affecting interpretation.

  • GEOQUERY: GEOQUERY uses variable-free FunQL, expands selected functions into conventional forms, and replaces entity mentions with shared placeholders.FunQL atoms are individual symbols, while compounds combine parent and child symbols in the logical-form tree.
  • SPIDER: SPIDER prepends the target database name and appends a serialized database schema containing table and column names to T5 inputs.The schema is parsed into compounds using an unambiguous CFG over the tokenized SQL string.
  • SCAN: SCAN requires no dataset-specific preprocessing.
  • Reporting: The induced grammar-size table reports that SPIDER has more induced rules than original examples because initialization adds identity rules.The appendix separately provides tables for induced grammar sizes and NQG standard deviations.
  • Reporting: NQG GEOQUERY results in Tables 2 and 5 average three runs, while NQG-T5 deviations reuse one fine-tuned T5 checkpoint.Thus, the NQG-T5 deviations omit variance from independently fine-tuned T5 checkpoints.

B.6 T5 GEOQUERY Errors

The appendix provides example T5-Base prediction errors from the GEOQUERY TMCD split as a concrete illustration of model failures.

  • Error analysis: Table 7 presents example T5-Base prediction errors on the GEOQUERY TMCD split.
Loading 2010.12725v2…