Source-linked AI summary
Good-Enough Compositional Data Augmentation
Jacob Andreas
TL;DR
The paper addresses failures of black-box sequence models to generalize compositional reuse from limited data while retaining their practical advantages. It introduces GECA, a simple model-agnostic augmentation protocol that recombines fragments from similar environments, and reports improvements across semantic parsing, SCAN, and low-resource language modeling. The authors also characterize GECA as limited and imprecise as a general linguistic principle.
Problem
Black-box models may fail to generalize simple compositional reuse from sparse training examples, while structured alternatives can be less effective on real-world data.
Method
GECA is a rule-based, model-agnostic protocol that synthesizes examples by replacing fragments with alternatives found in shared environments.
Results
GECA is effective across semantic parsing, SCAN compositionality diagnostics, and low-resource language modeling in six languages.
Takeaways & Limitations
The results indicate that progress toward compositional reuse can be made without fundamental changes to model architecture.
Takeaways & Limitations
GECA captures only a narrow and imprecise slice of compositionality and can make incorrect predictions about real language data.
Abstract
from arXiv · showhide
We propose a simple data augmentation protocol aimed at providing a compositional inductive bias in conditional and unconditional sequence models. Under this protocol, synthetic training examples are constructed by taking real training examples and replacing (possibly discontinuous) fragments with other fragments that appear in at least one similar environment. The protocol is model-agnostic and useful for a variety of tasks. Applied to neural sequence-to-sequence models, it reduces error rate by as much as 87% on diagnostic tasks from the SCAN dataset and 16% on a semantic parsing task. Applied to n-gram language models, it reduces perplexity by roughly 1% on small corpora in several languages.
1 Introduction
The paper introduces GECA, a rule-based, model-agnostic augmentation protocol that recombines observed fragments to encourage compositional generalization without changing predictor structure. It reports effectiveness across semantic parsing, SCAN, and low-resource language modeling.
- GECA motivation and approach: Language learners generalize that interchangeable fragments can appear in new environments, but unstructured generative models may fail on such sparse examples.The motivating example favors “The wug daxed” while rejecting “The sang daxed,” reflecting inferred syntactic categories.
- GECA motivation and approach: GECA softly encourages compositional reuse while preserving black-box predictors that perform well on real-world data.The approach addresses the tension between predictable symbolic extrapolation and the practical accuracy of neural function approximators.
- GECA motivation and approach: GECA recombines real training examples by replacing fragments with alternatives observed in similar environments.The protocol treats shared environments as evidence that fragments are substitutable, then uses additional occurrences to synthesize examples.
- Caveat: GECA is limited and imprecise as a linguistic principle, capturing only a narrow slice of compositionality and making incorrect predictions about real language data.The authors nevertheless present it as useful for generating synthetic training examples.
- Reported scope: GECA improves semantic parsing, solves two SCAN compositionality problems, and helps low-resource language modeling across six typologically diverse languages.In semantic parsing, improvements are comparable to or better than a task-specific augmentation method in several settings.
2 Background
The background contrasts predictable extrapolation from explicit symbolic models with the accuracy and scalability of black-box neural models. It motivates model-agnostic data augmentation as a way to encourage reuse without task- or architecture-specific changes.
- Motivation: Black-box neural models often outperform traditional structured approaches on real-world data, despite weaker predictable extrapolation from small datasets.Explicit symbolic processes make out-of-distribution behavior easier to characterize, while neural models can be more accurate with enough data.
- Motivation: The transition at which neural models learn predictable hierarchical generalization is not well characterized and may require more data than many real-world problems provide.The passage notes that such structure can emerge given enough data, but the relevant scale remains unclear.
- Existing approaches: Existing proposals improve function approximators or loss functions through task- and architecture-specific mechanisms.Examples include structured regularization, posterior regularization, explicit stacks, and composition operators.
- Data augmentation: Data augmentation offers a way to increase effective training-data scale artificially by transforming or combining existing examples.Such techniques are widely used in computer vision and have also improved NLP text classification, including with large-scale pretraining.
- GECA's scope: GECA targets a simpler, more general compositionality bias: reusing recurring training fragments in environments distinct from their original contexts.It assumes neither brackets nor types and can apply where approaches designed for specific settings cannot.
3 Approach
GECA augments sequence-modeling data by substituting fragments that share an identical lexical environment, motivated by distributional and syntactic evidence. The protocol permits broad fragment structures and accepts some errors when the resulting inductive bias outweighs data scarcity.
- Motivation: The procedure is motivated by syntactic categories and distributional evidence that context signals a fragment’s category.Its rationale combines the existence of well-formedness rules with contextual cues about fragment categories.
- Scope: GECA applies substitution beyond contiguous words, including gappy fragments and sequence pairs with a boundary symbol in translation examples.The same principle is illustrated for synthesizing “I dax. ▷Dajo.” from a shared bilingual environment.
- Core operation: GECA creates synthetic examples by replacing fragments with alternatives that occur in at least one identical lexical environment.A removed fragment produces a template, which is populated with another fragment.
- Formalization: A fragment is a set of non-overlapping spans, a template removes that fragment, and an environment keeps a k-word window around each removed span.These definitions support discontinuous fragments and localized context matching.
- Limitations: GECA is a narrow and imprecise account of compositionality that makes incorrect predictions about real language data.Examples include treating stress-derived nouns as interchangeable with verbs and overgeneralizing Mandarin words for “but.”
- Assumptions: GECA can still help when augmentation errors contribute less inaccuracy than the original shortage of training data.For conditional tasks, examples outside the real training support may be harmless while supplying useful inductive bias.
4 Diagnostic experiments
The diagnostic experiments test whether GECA enables compositional generalization in SCAN and NACS by synthesizing training examples from reusable fragments and templates. GECA lets sequence-to-sequence models succeed on held-out compositional uses, while also producing some unrelated examples.
- Experimental setting: The experiments use SCAN commands paired with discrete action sequences and test compositional uses of the primitive “jump” or modifier “around right,” including reverse-direction NACS mappings.The tests examine isolated commands or modifiers being incorporated into more complex commands.
- GECA procedure: GECA extracts fragments and templates from training sequences, records their shared environments, and synthesizes new examples by recombining them.Fragments may be discontinuous; generated examples are appended to the original dataset.
- Experimental setting: 395 additional examples are generated for the original jump split from 12,620 training examples, using 395 distinct templates and 6 distinct fragments.
- Results: GECA enables successful generalization where the baseline sequence-to-sequence model completely fails on the test set.Table 1 reports sequence-match accuracies averaged over 10 random seeds, with all improvements statistically significant.
- Analysis: Only three training examples provide the evidence needed to synthesize new jump usages; removing them leaves training accuracy unchanged but reduces test accuracy to zero.For the around right condition, GECA correctly replaces all LTURN occurrences with RTURN, demonstrating discontinuous and non-context-free substitution.
- Analysis: GECA synthesizes 5% of held-out examples for the add primitive condition and raises token co-occurrence overlap from 83% to 96%.For the add template condition, token co-occurrence overlap is 100% before augmentation; the protocol also generates spurious examples unrelated to the target generalization.
5 Semantic parsing
GECA improves semantic parsing across GEOQUERY representations and splits, while its gains depend on compositional reuse and dataset structure. The augmentation generally produces grammatical, semantically accurate examples, though some substitutions are problematic.
- GEOQUERY results: GECA approaches Jia and Liang’s augmentation on GEOQUERY logical forms’ standard Question split and outperforms it on the compositionality-focused Query split.On SQL expressions, GECA produces substantial Query-split improvements and state-of-the-art results.
- Synthesized examples: GECA aligns natural-language entity names with logical representations and generalizes across entity choices without specialized entity-linking machinery.It also generates plausible but unattested entities, such as a river named florida and a state named west wyoming.
- Synthesized examples: Some GEOQUERY substitutions are semantically invalid or ungrammatical, yet may still provide evidence about constituent boundaries, alignments, and hierarchical structure.One example asks for the population of a number, causing the corresponding SQL expression to fail.
- GEOQUERY analysis: 5%–9% increases in full example overlap accompany 3–4% increases in token co-occurrence overlap across GEOQUERY conditions.Full-example overlap rises 5% for both question splits, 6% for logical-form queries, and 9% for SQL queries.
- GEOQUERY analysis: 96% of 100 synthesized GEOQUERY query-split examples are grammatical, and 98% are semantically accurate.The manual analysis evaluates grammaticality and whether the natural language captures the meaning representation.
- Negative results: GECA provides no improvement on SCHOLAR, where synthesized examples add little or no overlap with held-out data and the dataset has less SQL nesting.These results suggest success is greatest when augmentation increases training–test co-occurrence similarity and the input has substantial recursion.
6 Low-resource language modeling
GECA is evaluated as an augmentation and smoothing method for low-resource language modeling across six languages and varied morphological settings. It yields modest, non-universal gains, but perplexity decreases across multiple languages and never increases.
- Experimental setting: GECA is tested on English, Kinyarwanda, Lao, Na, Pashto, and Tok Pisin, with training datasets ranging from 10K–2M tokens.The languages span complex noun-class and derivational morphology as well as comparatively simple morphological systems.
- Method: The best language-modeling performance comes from interpolating models trained separately on the original and augmented datasets.The interpolation weight is selected on validation data from 0.05, 0.1, and 0.5.
- Results: GECA decreases perplexity across multiple low-resource languages and never increases it, although improvements are more modest and not universal.The experiments use Kneser–Ney smoothed 5-gram models rather than high-capacity neural models.
- Method: In language modeling, GECA acts as a smoothing scheme that moves probability mass toward n-grams occurring in productive contexts.Unlike Kneser–Ney, its notion of context can look forward as well as backward.
- Synthesized examples: 79% of 100 synthesized sentences are grammatical, but only 51% are semantically acceptable in the manual analysis.The language-modeling evaluation measures generic semantic acceptability because explicit semantics are unavailable.
7 Discussion
GECA is a simple augmentation scheme that uses local phrase substitutions licensed by common contexts. The results show substantial improvements across diagnostic and natural datasets, while motivating richer representations and further study of compositionality.
- GECA identifies local phrase substitutions licensed by common contexts and uses them to generate additional training examples.
- Substantial improvements were demonstrated on diagnostic and natural datasets for semantic parsing and language modeling.
- The procedure relies on exact string matching to identify common contexts, leaving learned span representations and constituent-aware extraction as future directions.
- The results highlight failures to learn simple context-independent reuse while showing progress without fundamental changes to model architecture.