Source-linked AI summary
Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks
João Loula, Marco Baroni, Brenden M. Lake
TL;DR
The paper asks whether seq2seq models can systematically recombine familiar functional words, rather than merely learning new word embeddings. It repurposes SCAN to test novel combinations such as “jump around right” and finds impressive generalization only when training includes extensive evidence in the target pattern, with failures when models must apply compositional rules in genuinely new ways.
Problem
The paper addresses limited evidence about whether seq2seq models can systematically recombine familiar functional words, rather than only generalize after learning new-word embeddings.
Method
The authors repurpose SCAN and structured train-test splits to test novel contexts for familiar terms such as “around” and “right”.
Results
Seq2seq models generalize impressively to novel combinations of previously seen commands, but performance depends on extensive training in the specific pattern being generalized and fails for novel rule application.
Takeaways & Limitations
Generalization to familiar-word combinations is not systematic when models need to infer a compositional rule from its separate components.
Takeaways & Limitations
The paper leaves open which training evidence and model priors are needed for systematic generalization and how these abilities transfer to real-life seq2seq tasks.
Abstract
from arXiv · showhide
Systematic compositionality is the ability to recombine meaningful units with regular and predictable outcomes, and it's seen as key to humans' capacity for generalization in language. Recent work has studied systematic compositionality in modern seq2seq models using generalization to novel navigation instructions in a grounded environment as a probing tool, requiring models to quickly bootstrap the meaning of new words. We extend this framework here to settings where the model needs only to recombine well-trained functional words (such as "around" and "right") in novel contexts. Our findings confirm and strengthen the earlier ones: seq2seq models can be impressively good at generalizing to novel combinations of previously-seen input, but only when they receive extensive training on the specific pattern to be generalized (e.g., generalizing from many examples of "X around right" to "jump around right"), while failing when generalization requires novel application of compositional rules (e.g., inferring the meaning of "around right" from those of "right" and "around").
1 Introduction
Compositionality lets humans understand novel combinations from known components, but neural seq2seq models often require extensive training and struggle when generalization demands systematic composition rather than familiar patterns.
- 1 Introduction: Compositionality enables people to understand and produce novel combinations of known components, supporting rapid language learning from limited data.For example, knowing “to run” and “slowly” supports understanding “to run slowly” without prior exposure.
- 1 Introduction: SCAN evaluates seq2seq compositionality by testing commands that are novel combinations of components seen during training.The dataset frames simplified navigation-language translation as a grounded sequence-to-sequence task.
- 1 Introduction: Earlier work found strong zero-shot generalization on arbitrary train-test splits but failures when models had to extract algebraic composition rules.RNNs struggled with longer action sequences and with applying a familiar context to a newly introduced verb.
- 1 Introduction: The new-verb failure may reflect difficulty learning a meaningful embedding for an unfamiliar word rather than a purely compositionality-specific failure.RNNs failed to represent “to dax” similarly to known verbs, limiting similarity-based generalization.
- 1 Introduction: This paper instead tests whether models can recombine highly familiar functional words in novel contexts, such as interpreting “jump around right” from known component meanings.The results show considerable generalization, but performance drops sharply as the train-test difference becomes more systematic, despite simple composition rules being illustrated in training.
2 Generalizing functional terms with SCAN
The paper repurposes SCAN to test recombination of familiar functional terms, using its structured templates and symmetries to create novel command combinations for evaluation.
- 2 Generalizing functional terms with SCAN: SCAN translates commands generated by a phrase-structure grammar into action sequences through a semantic interpretation function.Its templates provide structured combinations of primitives, spatial terms, manner adverbs, and conjunctions.
- 2 Generalizing functional terms with SCAN: The dataset supports train-test splits ranging from random separation to longer sequences and sparse compositions of primitives.These splits were used to study increasingly demanding forms of generalization in earlier work.
- 2 Generalizing functional terms with SCAN: This paper repurposes SCAN to test familiar functional terms used in new contexts rather than rapid acquisition of new word embeddings.Manner adverbs such as “around” and “opposite” act as second-order modifiers over spatial modifiers and primitives.
- 2 Generalizing functional terms with SCAN: The experimental split can train on “walk left”, “walk right”, and “jump around left” while testing “jump around right”, requiring recombination of known words.The model must exploit left-right symmetry and learn abstract rules for higher-order modifiers.
3 Experiments
The experiments test whether recurrent networks recombine familiar functional words in novel SCAN contexts, rather than merely generalizing familiar templates across fillers. Networks generalize strongly when the target pattern is extensively represented, but struggle when they must derive a new composition from separately learned parts.
- Experiment 1: Generalizing to novel templates: The model generalized to “jump around right” when trained on other fillers of that template, confirming substantial but nonsystematic zero-shot generalization.This condition exposed the model to “Primitive around right” constructions with other primitives while withholding the target commands.
- Experiment 1: Generalizing to novel templates: Poor performance on “Primitive right” shows that systematic generalization fails even when the required construction is shorter, despite training examples illustrating left/right symmetry.The model saw related “Primitive left” commands and other templates but still struggled to bootstrap the simplest right construction.
- Experiment 1: Generalizing to novel templates: 98.43% versus 2.46% accuracy shows near-perfect generalization across primitives when “around right” was trained, but failure when “right” and “around” had not appeared together.The network correctly applies a familiar complex template to a new primitive, but cannot directly compose the two familiar modifiers in a novel combination.
- Experiment 1: Generalizing to novel templates: Accuracy reached 47.62% for “opposite right,” exceeding the harder “around right” condition and suggesting that memory demands interact with compositional generalization.“Around” requires a longer action sequence than “opposite,” so repeating the relevant steps may affect performance.
- Experiment 2: Impact of filler variety in learning a complex template: One distinct primitive filler was enough for near-perfect transfer to other fillers of “around right,” indicating analogical generalization without productive composition of separately learned modifiers.Experiment 2 varied the number of commands containing “Primitive around right” while holding the overall training regime fixed.
- Experiment 3: Impact of number of distinct training examples in learning a complex template: Performance rose to 70% with 64 added examples and peaked at 512, but the gradual ramp-up suggests accumulating case-specific evidence rather than extracting a general rule.Experiment 3 added progressively more “look around right” commands without adding other “Primitive around right” commands.
4 Conclusion
The findings show that seq2seq models can generalize impressively to novel combinations of familiar commands, but this ability is not systematic. Successful generalization still requires many examples matching the test context, despite prior exposure to the relevant words.
- Seq2seq models generalize to novel constructions such as “Primitive around right” after seeing the template with only one filler primitive, demonstrating impressive combination-level performance.
- This generalization does not reflect systematic composition of the rules associated with familiar functional terms such as “right” and “around”.
- Even after observing all test words during training, models require abundant additional examples in the same context as the test set to generalize successfully.
- Future work should identify which training evidence supports systematic generalization and test how it varies across primitives, manner adverbs, spatial expressions, and real-life seq2seq tasks.