A Benchmark for Systematic Generalization in Grounded Language Understanding

Laura Ruis; Jacob Andreas; Marco Baroni; Diane Bouchacourt; Brenden M. Lake

Source-linked AI summary

A Benchmark for Systematic Generalization in Grounded Language Understanding

Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, Brenden M. Lake

arXiv:2003.05161v2Mar 11, 2020cs.CLcs.AIcs.LG

TL;DR

Human language supports systematic recombination of familiar components, but neural networks struggle with such generalization. The paper introduces gSCAN, a grounded grid-world benchmark for testing it, and finds that strong baseline and compositional methods fail on most systematic splits.

Problem
The paper addresses whether neural models achieve genuine compositional generalization rather than exploiting artifacts of the context-insensitive SCAN benchmark.
Method
gSCAN grounds language in grid-world states and evaluates eight linguistic compositional generalization types using systematic train-test splits.
Results
Both the multi-modal baseline and GECA fail on the vast majority of gSCAN splits, succeeding only on object-class interaction, with GECA additionally improving on novel object-property composition.
Takeaways & Limitations
gSCAN provides a benchmark for developing models with more human-like compositional learning skills in grounded and context-sensitive language understanding.
Takeaways & Limitations
The benchmark uses symbolic rather than RGB world representations, and its transfer to more naturalistic settings remains uncertain.

Abstract

from arXiv · show

Humans easily interpret expressions that describe unfamiliar situations composed from familiar parts ("greet the pink brontosaurus by the ferris wheel"). Modern neural networks, by contrast, struggle to interpret novel compositions. In this paper, we introduce a new benchmark, gSCAN, for evaluating compositional generalization in situated language understanding. Going beyond a related benchmark that focused on syntactic aspects of generalization, gSCAN defines a language grounded in the states of a grid world, facilitating novel evaluations of acquiring linguistically motivated rules. For example, agents must understand how adjectives such as 'small' are interpreted relative to the current world state or how adverbs such as 'cautiously' combine with new verbs. We test a strong multi-modal baseline model and a state-of-the-art compositional method finding that, in most cases, they fail dramatically when generalization requires systematic compositional rules.

1 Introduction

The paper motivates gSCAN as a grounded benchmark for testing whether models can systematically generalize linguistic rules in context-sensitive worlds. It argues that existing approaches and benchmarks leave this broader form of generalization insufficiently tested.

Human language supports combining familiar words to interpret novel situations, whereas modern neural networks have not mastered comparable language-based generalization.
Context-sensitive interpretation can change both the referent of “the red small circle” and the action sequence required to reach it.
Existing proposals may account for SCAN’s challenges by overfitting to limitations of that context-insensitive benchmark rather than achieving genuine compositional generalization.
SCAN tests rule-based generalization but maps linguistic strings to fixed action sequences without a world state for interpreting commands.
Grounded SCAN (gSCAN) grounds command meanings in grid-world states and evaluates eight types of compositional generalization.
Across eight generalization splits, the baseline dramatically fails on all but one split, while GECA improves performance on only one additional split.

2 Related Work

Related benchmarks study compositional, grounded, or linguistic generalization, but gSCAN targets a broader combination of grounded semantics and rule-based linguistic generalization. The paper also examines why SCAN-oriented methods may not transfer.

Prior work evaluates compositional generalization through synthetic environments, including object relations, navigation tasks, and unseen verb-object combinations.
Unlike several related grounded environments, gSCAN focuses on linguistic, rule-based generalization motivated by systematic human language behavior.
SCAN evaluates abstract compositional rules by translating grammar-generated instructions into action sequences across systematic train-test splits.
Permutation-equivariant methods for SCAN address local substitutions among verb primitives, whereas gSCAN introduces grounded challenges beyond those local input-command permutations.
GECA augments training data by recombining sentence fragments, but its interchangeability assumption is unreliable in more realistic grounded language understanding.

3 The Grounded SCAN Benchmark

gSCAN pairs synthetic language commands with symbolic grid-world states and tests whether models can execute them under systematic train-test differences. Its benchmark spans contextual references, compositional meanings, interaction rules, and sequence lengths.

Instructions: gSCAN asks agents to execute instructions in a 2D grid world containing objects, using grounded semantics to evaluate linguistic generalizations.
Instructions: Adverbial modifiers such as “cautiously” and “while spinning” transform target action sequences in context-sensitive ways beyond SCAN-style substitutions and concatenations.
World model: World states are symbolic tensors encoding grid objects by color, shape, and size, together with the agent’s location and heading.
World model: Object size determines a latent light or heavy class, requiring one push for light objects and two pushes for heavy objects.
Data splits: The benchmark uses eight systematic splits, alongside random splits, while reusing training sets so multiple tests can evaluate distinct generalization types efficiently.
Data splits: Compositional tests include novel object-property combinations, directions, contextual references, and adverbs, with more than 300k shared training demonstrations.
Data splits: Length generalization holds out target sequences longer than 15 actions after training on sequences of length at most 15.

4 Baselines

The paper evaluates a multimodal seq2seq baseline that combines command and grid-world encoders to generate action sequences. It also applies GECA augmentation while testing generalization across compositional changes illustrated by several benchmark splits.

Baseline architecture: The baseline fuses a recurrent command encoder with a convolutional state encoder and generates actions through joint attention over instructions and grid cells.The input includes the command sequence and grid-world tensor; the decoder emits an action sequence autoregressively.
Training: The model is trained with supervised cross-entropy optimization using Adam, with ground-truth action sequences and a fixed horizontal-then-vertical training convention.Training runs for 200,000 steps with batch size 200, and the best model is selected using a 2,000-example development set.
Data augmentation: GECA augmentation modifies commands and corresponding world-state features while leaving the target action sequences unchanged.For example, a red-circle command can become a red-square command when the environment replaces the target object accordingly.
Generalization evaluations: The benchmark tests whether models generalize from familiar descriptions to novel compositions involving object properties, relative size terms, and verb-action combinations.The supplied figures illustrate composition of attributes and references, size-based contextual interpretation, and transfer from pulling to pushing a heavy square.

5 Experiments

The experiments evaluate gSCAN’s compositional generalization splits, testing whether models can recombine familiar linguistic and grounded concepts under systematic train–test differences. Both the baseline and GECA fail substantially on many splits, although performance remains strong for some familiar object-interaction inferences.

Experiments: The experiments test generalization across novel object-property compositions, target identification, directions, action modifiers, latent object classes, and action-sequence lengths.The benchmark includes held-out color–shape references, context-sensitive size interpretation, novel navigation directions, adverbial transformations, and longer sequences.
Novel composition of object properties: 23.51% ± 21.82 exact match is achieved by the baseline on the red-squares split, while GECA reaches 78.77% ± 6.63.GECA helps when the model must compose the familiar concepts red and square into an unseen target reference.
Novel composition of object properties: 54.96% ± 39.39 baseline accuracy exceeds GECA’s 34.92% ± 39.30 on the yellow-square split, where yellow squares were seen but never color-referred.The authors hypothesize that models overfit the more frequent square-only command pattern.
Novel direction: 0% exact match is obtained by both methods on the novel-direction split, despite partial navigation toward the target row or column.The baseline reaches the correct row or column 63.10% ± 3.90 of the time, and GECA 58.85% ± 3.45.
Target identification: 35.2% ± 2.35 baseline accuracy and 33.19% ± 3.69 GECA accuracy occur on the size-based target split, with target identification errors driving failures.The models often exploit color and shape to choose randomly between two same-colored objects rather than interpreting size relative to the world state.
Novel action sequence lengths: 94.98% ± 0.1 accuracy below the training length falls to 2.10% ± 0.05 overall for longer action sequences.For sequences of length 17, accuracy is 1.71% ± 0.38, and it falls below 1% at lengths ≥18.

6 Conclusion

The authors conclude that gSCAN exposes major weaknesses in current approaches to compositional generalization. They argue that future progress may require new architectural extensions, while noting that broader transfer remains uncertain.

6 Conclusion: Both the multi-modal baseline and GECA fail on most gSCAN splits, with GECA improving only on novel compositions of object properties.Both methods succeed only on inferring object class and using it for interaction.
6 Conclusion: gSCAN suggests that existing methods largely exploit artifacts in SCAN rather than fully addressing grounded compositional generalization.Grounding introduces more sophisticated semantics intended to remove those artifacts.
6 Conclusion: Future progress may build on meta-learning or equivariance methods, but applying them to gSCAN currently requires highly non-trivial extensions.The authors identify these approaches as possible directions if the benchmark’s generalizations can be captured appropriately.
6 Conclusion: The authors cannot yet know how progress on gSCAN will extend to other tasks until approaches capable of tackling it emerge.They discuss possible extensions to reinforcement learning and RGB images, but characterize some translations as difficult or potentially more demanding.

Broader Impact

The paper frames systematic generalization as a challenge for modern AI and presents gSCAN as a stimulus for further research. It anticipates potential benefits for learning efficiency, robustness, and human-computer interaction without expecting selective broader impacts.

Broader Impact: Systematic generalization characterizes human language and thought but remains challenging for modern AI systems.The benchmark is designed to stimulate further research on this topic.
Broader Impact: Advances in machine systematic generalization could improve learning efficiency, robustness, and human-computer interaction.The passage presents these as potential benefits rather than guaranteed outcomes.
Broader Impact: The authors do not anticipate broader impacts that selectively benefit some groups at the expense of others.

Supplementary Material

The supplementary grammar defines the command vocabulary and production structure used to generate input commands. It distinguishes intransitive and transitive verb categories through subscripts.

Supplementary Material: The command vocabulary includes intransitive walk, transitive push and pull, adverbs, nouns, and adjectives.Adverbs include spinning, zigzagging, hesitantly, and cautiously; nouns include circle, square, and cylinder; adjectives include colors and sizes.
Supplementary Material: The grammar composes verb phrases with adverbs, intransitive or transitive verbs, determiner phrases, noun phrases, adjectives, and nouns.
Supplementary Material: The subscripts i and t distinguish intransitive and transitive verb forms.

B World State Generation

World states are generated to make each instruction’s target uniquely identifiable while preserving distractors needed for size-based references. The procedure varies object sets according to which properties the instruction specifies.

B World State Generation: Each generated world state must contain a unique target object and, when size is mentioned, at least one distractor object.For a small square, the distractor is a larger square; for a small yellow square, it is a larger yellow square.
B World State Generation: Object placement randomly selects half of the possible objects while preserving differently sized pairs for each color-shape pair whenever size is mentioned.
B World State Generation: Shape-only targets use randomly colored and sized objects for other shapes, with a random half selected as distractors.
B World State Generation: Color-and-shape targets generate one randomly sized object for each non-target color-shape pair.
B World State Generation: Size, color, and shape targets generate two randomly sized objects per color-shape pair and select half the pairs at random.Objects matching the target’s shape, and color when specified, are sized relative to the target according to whether the modifier is big or small.
B World State Generation: World states enumerate possible target objects, relative directions, and distances between the agent and target.For example, a northeast target has a minimum distance determined by that relative position.
B World State Generation: Figure 7 presents four real data examples for grid size 6.

C Dataset Statistics

The appendix reports dataset statistics for compositional splits and details the forward-pass architecture used by the models. GECA expands training data but improves performance only on a narrow part of the test set.

Dataset statistics: Table 3 reports dataset example counts, including compositional training and test data with GECA augmentation.An example is unique when its command, target commands, or target-object location differs; the third column gives average unique world states per input command.
Dataset statistics: GECA correctly augments examples for missing red-square targets but fails to do the same for other relevant cases.This limitation explains why GECA does not improve the yellow-squares split and deteriorates on grounded SCAN.
Dataset statistics: GECA improves performance only on a narrow part of the test set.The supplied passage contrasts this limited gain with the absence of broader improvement.
Dataset statistics: Table 4 counts examples by target object and distinguishes objects placed as targets from objects referred to by color.References include descriptions such as “the red square” or “the small yellow square.”
Model forward pass: The baseline encodes commands with an LSTM and world states with convolutional features, then decodes target commands using attention.The state encoder uses convolutions with kernel sizes 1, 5, and 7; decoder inputs combine token embeddings with command and state context.

E (Hyper)parameters

The appendix gives the compute budget and lists model parameters not explicitly described in the main architecture section.

Compute: Training a model on the experimental data with a single GPU takes less than 24 hours.This is the reported compute requirement for one model training run.
Model parameters: Table 5 lists parameters for models not explicitly mentioned in Section 4.The table serves as a supplementary hyperparameter reference.

F Additional Results

Additional results include attention visualizations and analyses of how exact-match performance changes with target length. The length analyses show declining exact match even when tested targets are no longer than those seen in training.

Additional results: The appendix identifies these experiments as additional results referred to in Section 5.The results supplement the main-text experimental discussion.
Attention visualizations: Darker grid cells in Figure 8 indicate higher attention weights assigned by the models.The visualization exposes which world-state cells receive more attention.
Adverb split results: Exact match decreases as target length increases when generalizing “cautiously” after 50 demonstrations.The tested target lengths are not longer than those encountered during training.
Adverb split results: Exact match decreases as target length increases when generalizing “while spinning” to the verb “pull.”The tested target lengths are also no longer than those encountered during training in this experiment.