PaperlayerView exact revision on arXiv

Source-linked AI summary

Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures

Daniel Furrer, Marc van Zee, Nathan Scales, Nathanael Schärli

arXiv:2007.08970v3cs.CLcs.LG

TL;DR

Mainstream machine-learning methods often struggle to generalize compositionally, while SCAN-inspired gains may be task-specific. The paper evaluates specialized architectures and MLM pre-training on SCAN and CFQ, and finds that pre-training rivals specialized methods on one SCAN split and improves CFQ results; combined with an intermediate representation, it reaches a new CFQ state of the art.

  • Problem

    Machine-learning methods often fail to generalize compositionally, and MLM pre-training and intermediate representations had not been evaluated on semantic-parsing tasks designed to test compositional generalization.

    Source: §1 · p. 1

  • Method

    The paper evaluates specialized architectures and MLM pre-training on SCAN and CFQ, and combines pre-training with an intermediate representation for CFQ.

    Source: §1 · p. 2

  • Results

    Pre-training nearly matches SCAN-specialized gains on the “Add jump” holdout and significantly improves CFQ over comparable non-pre-trained models, while the evaluated specialized architectures show no significant gains; the combined approach reaches 42.1% on CFQ MCD-mean versus the previous best of 18.9%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Pre-training helps compositional generalization but does not solve it, and improvements by specialized architectures on one benchmark do not necessarily transfer to others.

    Source: §6 · p. 8

    Relative performance between Transformer and T5-small-NP is not fully consistent, and explaining these differences requires further investigation.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

While mainstream machine learning methods are known to have limited ability to compositionally generalize, new architectures and techniques continue to be proposed to address this limitation. We investigate state-of-the-art techniques and architectures in order to assess their effectiveness in improving compositional generalization in semantic parsing tasks based on the SCAN and CFQ datasets. We show that masked language model (MLM) pre-training rivals SCAN-inspired architectures on primitive holdout splits. On a more complex compositional task, we show that pre-training leads to significant improvements in performance vs. comparable non-pre-trained models, whereas architectures proposed to encourage compositional generalization on SCAN or in the area of algorithm learning fail to lead to significant improvements. We establish a new state of the art on the CFQ compositional generalization benchmark using MLM pre-training together with an intermediate representation.

1 Introduction

Machine-learning systems often fail to generalize compositionally, and SCAN-inspired gains may not transfer beyond their tailored task. This paper evaluates specialized architectures, MLM pre-training, and intermediate representations on SCAN and CFQ.

1 Introduction

  • The paper provides a broad review of SCAN and CFQ methods, including detailed SCAN-split results for Transformer and Universal Transformer models previously evaluated only on CFQ and SCAN MCD splits.The review is presented as the most comprehensive summary of results so far.Source: §1 · p. 2
  • CGPS and Neural Shuffle Exchange Network show no significant improvement over general-purpose sequence-to-sequence architectures when evaluated with DBCA on CFQ and SCAN.These architectures had been proposed to improve compositional generalization.Source: §1 · p. 2
  • Off-the-shelf MLM pre-training nearly matches SCAN-specialized gains on the “Add jump” primitive holdout and significantly improves CFQ performance over comparable non-pre-trained models.The paper evaluates pre-training on both SCAN and CFQ.Source: §1 · p. 2
  • Combining pre-training with an intermediate representation yields a new state-of-the-art CFQ score of 42.1% on the MCD-mean split, beating the previous best results of 18.9%.Source: §1 · p. 2

2 Background: Measuring Compositional Generalization

SCAN tests compositional generalization through purpose-built splits of commands and action sequences. DBCA offers a systematic way to construct challenging splits, and CFQ applies it to natural-language-to-SPARQL parsing.

2.1 SCAN

  • SCAN maps natural-language navigation commands to action sequences using a grammar that composes primitive commands into longer sequences.Its splits can hold out command lengths, primitive combinations, subcommands, or templates.Source: §2.1 · p. 2

2.2 DBCA and CFQ

  • DBCA constructs maximum compound divergence splits that maximize train-test compound divergence while keeping atom divergence small.This aims to make MCD splits compositionally challenging while keeping individual-rule distributions similar.Source: §2.2 · p. 2
  • CFQ is a large, realistic natural-language dataset for parsing questions into SPARQL queries, designed to measure compositional generalization with DBCA.Named entities are anonymized, so models need not learn all entities.Source: §2.2 · pp. 2–3

3 Architectures and Techniques

The paper surveys general sequence-to-sequence models, SCAN-inspired and algorithm-learning approaches, MLM pre-training, and an intermediate SPARQL representation. These methods are evaluated on SCAN and CFQ.

3 Architectures and Techniques

  • The paper compares general-purpose, specialized, and pre-trained approaches, with results presented in Section 4.Its general sequence-to-sequence overview includes LSTM, CNN, GRU, Transformer, Universal Transformer, and Evolved Transformer work.Source: §3 · p. 3, §3.1 · p. 3

3.2 SCAN-inspired Approaches

  • SCAN-inspired models decouple primitive input-output mappings from structural mappings; CGPS adds entropy regularization to its encodings.The paper evaluates CGPS to test whether SCAN gains carry over to DBCA splits of SCAN and CFQ, adapting its CFQ setup with input prefixes and hyperparameter search.Source: §3.2 · p. 3

3.2.1 Meta-learning for SCAN

  • SCAN meta-learning approaches use supplementary training episodes or support examples; Synth learns a symbolic grammar equivalent to SCAN’s grammar.Such approaches require task-specific supplementary data, including related grammars or crafted episodes.Source: §3.2.1 · p. 4

3.3 Algorithm Learning

  • The paper evaluates the Neural Shuffle-Exchange Network as an algorithm-learning approach for SCAN and CFQ, both expected to be learnable algorithmically because of their rule-based nature.Source: §3.3 · p. 4

3.4 Masked Language Model Pre-training

  • The paper fine-tunes pre-trained T5 variations on SCAN and CFQ and compares them with a T5-small model trained without pre-trained weights.SCAN fine-tuning uses about 20k steps because longer training led to overfitting on the small dataset; most runs are replicated five times.Source: §3.4 · p. 5

3.5 Intermediate SPARQL Representation

  • The intermediate SPARQL representation preprocesses outputs for training and post-processes them back into valid SPARQL at inference, improving CFQ performance.The representation groups subjects and objects to align the output structure more closely with the question.Source: §3.5 · p. 5

4 Results and Analysis

Across SCAN and CFQ, pre-training helps on many compositional splits but does not solve generalization, and specialized methods do not consistently transfer across benchmarks. The results also show that performance rankings among general sequence-to-sequence architectures vary by task and split.

4.1 Masked Language Model Pre-training

  • Pre-training benefits SCAN primitive, subcommand, and template holdouts and significantly helps on CFQ, but average accuracy falls 8.5% on the SCAN length split.The largest SCAN gains occur on “Add jump”; CFQ MCD gains were limited to lengths seen in training.Source: §4.1 · p. 5

4.2 Specialized Architectures

  • CGPS performs poorly on MCD splits and is outperformed by general-purpose architectures on CFQ and by LSTM+A on SCAN.Its underperformance versus general-purpose architectures is even more noticeable at moderate compound divergence; the authors suggest CGPS is not robust to compound-distribution shifts.Source: §4.2 · p. 6
  • NSEN performs poorly across evaluated splits; its maximum accuracy is 56.6% on CFQ’s random split, versus 98.3% on SCAN’s random split.The authors hypothesize that NSEN is insufficiently general for semantic parsing’s more complex structure; an ablation suggests autoregression alone is unlikely to explain its poor performance.Source: §4.2 · pp. 6–7

4.3 General Sequence-to-sequence Architectures

  • General sequence-to-sequence architecture improvements yield incremental, limited gains in compositional generalization settings.On the length split, Evolved Transformer performs much better than Transformer and Universal Transformer, mirroring T5-small-NP; the authors say further comparison is needed.Source: §4.3 · p. 7
  • On CFQ MCD, Evolved Transformer > Transformer > LSTM+A; on five of seven other SCAN splits, Transformer > LSTM+A and Evolved Transformer > Transformer.SCAN MCD is an exception: LSTM+A outperforms the other general architectures, a pattern absent from lower-divergence DBCA splits.Source: §4.3 · p. 7

4.4 Interpretation of Prior Work

  • Traditional SCAN splits make unambiguous architecture comparisons difficult because relative performance varies across splits; MCD provides a single, comprehensive measure.The paper notes that many models were not evaluated on all the traditional splits.Source: §4.4 · p. 7

5 Related Work

Related work surveys techniques that may support compositional generalization in semantic parsing, alongside approaches developed for related tasks. It highlights possible directions involving pre-training, output representations, data augmentation, and decoder design.

Other approaches

  • The related-work discussion also points to approaches beyond SCAN and CFQ as possible areas for future compositional-generalization research.These broader approaches are described as not directly applicable to the two benchmarks.Source: §5 · pp. 7–8

Pre-training on artificial languages

  • Pre-training on artificial languages is presented as a potential extension because CFQ maps natural language to SPARQL, whose tokens have at most a loose connection to English words.The paper discusses adapting prior language-modeling approaches to CFQ if a sufficiently large SPARQL corpus is available.Source: §5 · p. 7

Intermediate representations

  • Intermediate representations can simplify output mapping, and the paper sees potential for improved representations that preserve precision and expressive power while allowing programmatic manipulation and closer mapping to natural language.Prior examples include SemQL for text-to-SQL and lambda-DCS for text-to-SPARQL.Source: §5 · pp. 7–8

Data augmentation

  • Data augmentation may support compositional generalization, but introducing incorrect examples remains a challenge; it could also be combined with other approaches because it is architecture-independent.The cited work reports evidence from situated-agent tasks and preliminary results on some SCAN splits.Source: §5 · p. 8

Syntax-constrained decoders; Order independence in decoder

  • Syntax-constrained decoders reduce the effective output space or enforce a high-level output format, while order-independent decoding addresses the mismatch between sequential outputs and unordered SPARQL or SQL constraints.The paper considers grammar- and sketch-based decoders potentially applicable to CFQ; earlier sequence-to-set and non-deterministic-oracle approaches yielded modest gains on WikiSQL.Source: §5 · p. 8

DB schema encoders

  • Prior work on database schema encoders uses graph encoders to represent the target database schema for attention during question encoding and output decoding.The paper suggests explicit schema knowledge could potentially help disentangle generic language understanding from task-specific knowledge, including for CFQ.Source: §5 · p. 8

6 Conclusion and Outlook

The paper’s findings span pre-training, specialized architectures, general-purpose sequence-to-sequence models, and the difficulty of evaluating compositional generalization with traditional SCAN splits. It also points to possible future gains from output-language pre-training, improved intermediate representations, syntax-constrained decoding, and further CFQ adaptation.

6 Conclusion and Outlook

  • Pre-training helps compositional generalization but does not solve it.Source: §6 · p. 8
  • For the specialized architectures evaluated, improvements on one compositional generalization benchmark do not transfer to others.Source: §6 · p. 8
  • Improvements to general-purpose sequence-to-sequence architectures generally lead to corresponding incremental improvements in compositional settings.Source: §6 · p. 8
  • Traditional SCAN splits make unambiguous, comprehensive evaluation challenging, which may favor MCD splits as a more comprehensive measure.Source: §6 · p. 8
  • Pre-training on SPARQL, improved intermediate representations, and syntax-constrained decoding are proposed as possible routes to further gains.The pre-trained models used English input alone; the paper also notes potential gains from simplifying the output format.Source: §6 · p. 8
  • LANE and Synth are suggested for future evaluation, but would require more substantial adaptation to CFQ.Source: §6 · p. 9

A Contributions

The appendix identifies contributors’ roles and notes that hyperparameter searches were performed on a random CFQ split.

A Contributions

  • Daniel ran the Evolved Transformer, NSEN, and CGPS experiments; Marc built T5 pre-training infrastructure, ran experiments, and developed the intermediate SPARQL representation.Nathan researched related work, Nathanael provided project direction, and all authors contributed to scope, research direction, and writing.Source: A Contributions · p. 13

B Hyperparameters

C Ablation Study: Autoregression

An ablation removing autoregression from the LSTM baseline on CFQ showed only minimal performance degradation. This suggests autoregression alone is unlikely to explain NSEN’s poor performance on CFQ.

C Ablation Study: Autoregression

D Additional Experiments on the MCD Splits

Additional MCD-split experiments evaluate CGPS and NSEN against standard sequence-to-sequence baselines on SCAN and CFQ. The results report divergence-dependent performance patterns, including NSEN’s steep drops and Evolved Transformer’s SCAN benefits at medium divergences.

D Additional Experiments on the MCD Splits

E Intermediate SPARQL Representation

Intermediate SPARQL representations improve Transformer accuracy across CFQ compound-divergence splits, with grouping subjects and objects producing the largest gains. The representation successively groups clauses by subject, groups objects, and optionally sorts clauses.

E Intermediate SPARQL Representation

  • For the example question “Did M0 and M1 direct M2 and M3?”, grouping first collects clauses by subject, then groups the shared objects under each relation.The initial query has four clauses; after both grouping steps, each subject has one directed relation containing M2 and M3.Source: E Intermediate SPARQL Representation · pp. 13–14
  • The three transformations group clauses by subject, group subjects and objects, and then optionally sort the clauses alphabetically.The final representation is the same as the subject-and-object grouping, but sorted.Source: E Intermediate SPARQL Representation · pp. 13–14
  • Intermediate representations yield significant gains on every CFQ compound-divergence split, with grouping subjects and objects producing the largest wins.Sorting alphabetically does not seem to affect accuracy significantly.Source: E Intermediate SPARQL Representation · p. 14, Figure 3 · p. 15

F Effect of Pre-training Input and Output Length

Pre-training helps on many splits, but its effect depends on input and output length. It reduces average accuracy on the SCAN length split, and both models score zero beyond training-seen lengths on two other splits.

F Effect of Pre-training Input and Output Length

G Extended Related Work

The extended related work describes approaches that encourage compositional generalization or improve question answering but are not directly applicable to SCAN or CFQ. It discusses database-content encoding and neural module networks, alongside a length-based pre-training analysis.

G Extended Related Work

  • The discussed approaches are not directly applicable to the SCAN or CFQ tasks.Source: G Extended Related Work · p. 15
  • Database-content encoders attend to relevant database content, while content-blind parsing can suit settings where database access is restricted for privacy reasons.The passage presents these as separate semantic-parsing task settings.Source: G Extended Related Work · p. 15
  • Neural module networks combine reusable modules at evaluation, but their current form depends on a semantic parser and primarily biases answer retrieval rather than question understanding.The Neural State Machine also uses a variation of a semantic parser internally to map questions to reasoning instructions.Source: G Extended Related Work · p. 15
  • Figure 4 presents train/test sample-length distributions and pre-training performance effects by length across SCAN length, SCAN-MCD3, and CFQ-MCD1.Source: Figure 4 · p. 16

Related papers

Loading 2007.08970v3…