Source-linked AI summary
Compositional generalization through abstract representations in human and artificial neural networks
Takuya Ito, Tim Klinger, Douglas H. Schultz, John D. Murray, Michael W. Cole, Mattia Rigotti
TL;DR
The paper asks how compositional generalization supports rapid human transfer and whether its neural implementation can be reproduced in ANNs. Using a 64-context task, fMRI analyses, and task-informed pretraining, it finds abstract representations, strong zero-shot generalization, efficient learning, and convergent sensory-to-motor organization across humans and models.
Problem
The study addresses limited evidence about the behavioral and neural basis of compositional generalization in humans and its relevance to human-like ANN learning.
Source: §1 · p. 1Method
The authors combine a 64-context human task, whole-cortex fMRI, Parallelism Score analyses, and pretraining paradigms that encode compositional task primitives into feedforward ANNs.
Source: §1 · p. 2, §2.2 · p. 4, §2.3 · p. 4Results
Combined pretraining reached 92.15% zero-shot accuracy on the full C-PRO task, while abstract representations formed a sensory-to-motor hierarchy aligned across human cortex and ANN layers.
Source: §3.4 · p. 8, §3.7 · p. 9Takeaways & Limitations
Abstract representations are implicated in compositional generalization and can be embedded into ANNs through simple pretraining procedures.
Source: §4 · p. 10The study examines only sensory, logic, and motor gating rules and does not address other task components such as reward or stimuli.
Source: §4 · p. 10
Abstract
from arXiv · showhide
Humans have a remarkable ability to rapidly generalize to new tasks that is difficult to reproduce in artificial learning systems. Compositionality has been proposed as a key mechanism supporting generalization in humans, but evidence of its neural implementation and impact on behavior is still scarce. Here we study the computational properties associated with compositional generalization in both humans and artificial neural networks (ANNs) on a highly compositional task. First, we identified behavioral signatures of compositional generalization in humans, along with their neural correlates using whole-cortex functional magnetic resonance imaging (fMRI) data. Next, we designed pretraining paradigms aided by a procedure we term {\em primitives pretraining} to endow compositional task elements into ANNs. We found that ANNs with this prior knowledge had greater correspondence with human behavior and neural compositional signatures. Importantly, primitives pretraining induced abstract internal representations, excellent zero-shot generalization, and sample-efficient learning. Moreover, it gave rise to a hierarchy of abstract representations that matched human fMRI data, where sensory rule abstractions emerged in early sensory areas, and motor rule abstractions emerged in later motor areas. Our findings give empirical support to the role of compositional generalization in human behavior, implicate abstract representations as its neural implementation, and illustrate that these representations can be embedded into ANNs by designing simple and efficient pretraining procedures.
1 Introduction
The paper investigates compositional generalization as a mechanism for human transfer learning and as a candidate basis for more human-like ANN learning. It combines a 64-context human task, whole-cortex fMRI, and task-informed ANN pretraining to compare behavioral and neural abstractions across humans and models.
1 Introduction
1.1 Related work
- Prior machine-learning approaches often use specialized modules, including disentangled representations or neural-symbolic program synthesis, to support systematic generalization [17, 18, 35, 16, 20, 29, 37, 46].Source: §1 · p. 2
1 Introduction
- Parallel abstract representations encode changes in individual variables as linear shifts, making compositional structure measurable through Parallelism Score in neural and fMRI data.Source: §1 · p. 2
- Using a 64-context task, the study tests whether parallel abstract representations support human compositional behavior and whether task-informed pretraining induces analogous ANN representations.The study compares behavioral signatures, whole-cortex fMRI organization, ANN generalization, sample efficiency, and layerwise abstraction.Source: §1 · p. 2, §1.1 · p. 2
- The paper reports that pretrained ANNs show more abstract representations, strong generalization, sample-efficient learning, and a human-like sensory-to-motor hierarchy of rule abstractions.Source: §1 · p. 2
2 Methods
The methods combine the 64-context C-PRO task with geometric analyses of human and ANN representations. Parallelism Score quantifies context-invariant rule coding, while simple feedforward ANNs are trained sequentially with alternative pretraining regimes.
2.1 The Concrete Permuted Rule Operations (C-PRO) task paradigm
- The C-PRO paradigm recombines four logical, four sensory, and four motor rules into 64 task contexts, with four practiced contexts preceding novel-context testing.Each context contains one rule from each domain, and each context can present one of 256 stimulus combinations.Source: §2.1 · p. 3, Figure 1 · p. 3
2.2 The geometry of abstract neural representations
- High Parallelism Score represents the same rule change as a parallel displacement across contexts, allowing a learned rule effect to transfer across other task features.The method compares coding directions for the same rule dichotomy across contexts with matched secondary and tertiary rules.Source: §2.2 · pp. 3–4
2.3 Parallelism score
- Parallelism Score is the cosine similarity of coding directions for the same rules across contexts, with values near 1 indicating highly parallel context-invariant coding.The score is averaged across rule dichotomies within each domain and tested against shuffled-label null distributions with family-wise error correction.Source: §2.3 · p. 4
2.4 ANN construction and training
- The primary ANN is a two-hidden-layer, 128-unit-per-layer feedforward network mapping a 28-element input to four motor responses through ReLU transformations.Inputs encode task rules and stimuli; training uses cross-entropy with Adam, and models learn practiced contexts before novel contexts are added sequentially.Source: §2.4 · pp. 4–5, Figure 4 · p. 5
3 Results
Humans showed compositional transfer across novel C-PRO contexts, while abstract representations were distributed by rule content across cortex. Pretraining endowed ANNs with abstract representations that improved zero-shot generalization, sample efficiency, human-like behavior, and sensory-to-motor representational organization.
3.1 Behavioral signatures of rapid compositional generalization in humans
- Pretraining improved human-like compositional transfer: human accuracy increased with practiced rule overlap, while ANN performance ranked practiced, 2-rule-overlap, then 1-rule-overlap contexts.Humans achieved 84.17% on novel contexts, including 84.86% for 2-rule overlap and 83.48% for 1-rule overlap.Source: §3.1 · p. 6, Figure 7 · p. 8
3.2 Spatial and content-specific topography of abstract representations in human cortex
- Human abstract representations were content-specific: sensory PS localized mainly to higher-order visual and dorsal-attention areas, logic PS to frontoparietal areas, and motor PS to somatomotor cortex.Significant-region PS values were 0.15 for sensory, 0.22 for logic, and 0.29 for motor abstractions.Source: §3.2 · p. 6
3.3 Embedding prior knowledge into ANNs with simple pretraining tasks
- Primitive pretraining taught one-rule semantics, whereas Simple task pretraining taught two-rule sensorimotor or logical-sensory conjunctions simpler than the full three-rule C-PRO task.The pretraining tasks covered sensory features, motor mappings, Boolean relations, sensorimotor mappings, and logical-sensory gating.Source: §3.3 · p. 6, Figure 5 · p. 7
3.4 Pretraining induces abstractions, zero-shot performance, and sample efficiency
- Pretraining increased ANN Parallelism Score relative to Vanilla, with Simple task pretraining producing higher PS than Primitives pretraining.The comparisons were significant for Primitives versus Vanilla, Simple task versus Vanilla, Combined versus Vanilla, and Simple task versus Primitives.Source: §3.4 · p. 7
- Combined pretraining achieved 92.15% zero-shot accuracy on all 64 C-PRO contexts, exceeding Simple task pretraining at 70.57%.Vanilla performance remained near chance at 23.25%, while Primitives pretraining reached 31.51%.Source: §3.4 · p. 8
- Pretraining reduced the samples needed for generalization: a Vanilla model required up to 4.23x more total samples to match Combined-model performance.The Vanilla model reached 96.02% on held-out contexts after being trained on 60/64 contexts, whereas Combined pretraining provided the matched performance more efficiently.Source: §3.4 · p. 8
- PS and zero-shot performance increased together during Combined pretraining and were highly correlated, with r=0.88 and p<10e-9.This links the emergence of abstract hidden representations with improved generalization during training.Source: Figure 6 · p. 7, §3.4 · p. 8
3.5 ANN pretraining leads to human-like compositional generalization
- Simple task pretraining reached 86.79% generalization after four practiced contexts, matching human-like generalization, versus 57.97% after Primitives pretraining.The Vanilla model generalized poorly, with 28.79% accuracy on novel contexts despite 94.37% on practiced contexts.Source: §3.5 · p. 8, Figure 7 · p. 8
4 Discussion, Limitations, Conclusions
The study links abstract representations to compositional generalization in humans and ANNs, while identifying benefits, limitations, and future benchmarking needs.
4 Discussion, Limitations, Conclusions
- Pretraining embedded abstract representations that produced human-like generalization, sample-efficient learning, and convergent rule-specific abstraction hierarchies across human cortex and ANN layers.The hierarchy progressed from sensory abstractions in early areas or layers to motor abstractions in later ones.Source: §4 · p. 10
- The approach requires designing pretraining paradigms from prior knowledge of task structure, creating an initial design overhead despite improved sample efficiency.The authors contrast this requirement with approaches using low-norm initialization or built-in modular structures.Source: §4 · p. 10
- The study was limited to three rule types and one C-PRO task across 64 contexts, leaving other task components and broader multitask settings for future work.The authors specifically identify reward, stimuli, multitask settings, and comparisons with continual-learning or meta-learning paradigms as open directions.Source: §4 · p. 10
- The 64-context human-centric framework provides a benchmark for interpreting compositionality and comparing the strengths and limitations of humans and ANNs.The conclusion emphasizes that matching ANNs and humans on the same task creates a concrete interpretability target.Source: §4 · p. 10
Checklist
The checklist records affirmative reporting for reproducibility, asset documentation, ethics, data handling, and disclosure of limitations and scope.
Checklist
- The paper states that its claims reflect the contributions, discusses limitations and scope, and marks theoretical-results requirements as not applicable.Potential negative societal impacts were also marked not applicable, with ethics-guideline compliance affirmed.Source: Checklist · p. 14
- The paper reports that code, data, instructions, training details, error bars, and compute information were provided or addressed.The checklist marks these items as satisfied, including error bars for Figures 1, 5, 6, and 7.Source: Checklist · p. 14
- Existing assets were cited and their CC0 license was documented, while no new assets were included.The checklist identifies the previously published data as the existing asset.Source: Checklist · pp. 14–15
- Human-subject reporting included participant instructions, IRB approval, consent discussion, and de-identification of publicly available data.The checklist notes that compensation reporting was not applicable because the dataset was previously published.Source: Checklist · p. 15
A.1 Participants
The study used a publicly available, de-identified dataset and analyzed fMRI data from 96 consenting participants after prespecified exclusions.
A.1 Participants
- The dataset was publicly released under a CC0 license, and all participant data in the repository were de-identified.The passage identifies the dataset as originating from a previous study.Source: §A.1 · p. 16
- fMRI analyses included 96 participants after six MRI-technical exclusions and four incomplete-session exclusions.The analyzed sample included 54 females, with mean age 22.06 years and standard deviation 3.84.Source: §A.1 · p. 16
A.2 C-PRO task paradigm – additional details
The C-PRO paradigm combines sensory, logical, and motor rules into many audiovisual task contexts, with structured training and fMRI timing.
A.2 C-PRO task paradigm – additional details
- Stimuli varied color, orientation, pitch, and auditory continuity, yielding 256 audiovisual stimulus combinations per task context.Each context involved two sequentially presented audiovisual stimuli.Source: §A.2 · p. 16
- C-PRO combines four rules from each of three domains to form 64 unique task contexts and 16,384 context-stimulus trial combinations.Each context contains one logic, one sensory, and one motor rule; each task set was presented twice.Source: §A.2 · p. 16
- Participants practiced four counterbalanced contexts before scanning, and mean in-scanner performance was 84%, above the 25% chance level for all participants.The four practiced rule sets were selected so that all 12 rules were equally practiced.Source: §A.2 · p. 16
A.3 Analysis of human task performance data
Human task performance was analyzed across practiced contexts and novel contexts sharing one or two rules with prior tasks, as well as across repeated presentations of individual rules in new contexts.
A.3 Analysis of human task performance data
- Behavioral results from this analysis were reported in Figure 1.Source: §A.3 · p. 16
- 128 task accuracy scores per subject were grouped into practiced, 2-rule-overlap, and 1-rule-overlap contexts for behavioral comparison.There were no 0-rule-overlap contexts because participants were trained on every rule before testing.Source: §A.3 · p. 16
- Performance on each rule in novel contexts was evaluated across exactly 15 miniblocks after its practiced presentation.Each rule appeared 16 times in total, including the practiced miniblock.Source: §A.3 · p. 16
- Positive regression slopes tested whether performance improved as participants reused previously learned rules in novel contexts.A significant positive slope was interpreted as compositional learning through rule reuse across contexts.Source: §A.3 · pp. 16–17
A.4 fMRI acquisition
The study acquired whole-brain fMRI data during the task, applied standardized and custom denoising, and estimated rule-related activations from encoding-period beta series.
A.4 fMRI acquisition
- Whole-brain fMRI used a 3T Siemens Trio with 2.0 mm isotropic voxels, multiband acceleration factor 8, and eight task runs.A 14-minute resting-state scan preceded the task scans.Source: §A.4 · p. 17
A.5 fMRI preprocessing
- The Human Connectome Project minimal preprocessing pipeline was followed by CIFTI grayordinate surface analyses with motion and physiological nuisance regression.The custom regression used 64 nuisance parameters, including motion regressors and aCompCor components.Source: §A.5 · p. 17
A.6 fMRI activation estimation
- Vertex-wise task activations were estimated with beta-series regression using one regressor for each of 128 encoding miniblocks.Encoding periods lasted 3925 ms, or 5 TRs, to isolate rule representations from stimulus-response activity.Source: §A.6 · p. 17
A.7 ANN construction and batch training
The authors used a simple feedforward ANN and sequential C-PRO training, then pretrained networks on primitive rules and simpler two-rule tasks before evaluating full-task learning.
A.7 ANN construction and batch training
- The primary ANN had two 128-unit hidden layers and four motor-response outputs, mapping task inputs to response vectors with ReLU transformations.Weights and biases were initialized from U(−1/k, 1/k), where k is the number of inputs from the previous layer.Source: §A.7 · p. 17
- C-PRO training sequentially added novel contexts after practiced contexts, using batches of 256 stimulus combinations per context.Training continued until all 64 contexts had been trained.Source: §A.7 · pp. 17–18
- Training either stopped at 90% accuracy per context or used a fixed 200 gradient steps per context, depending on the experiment.Source: §A.7 · p. 18
A.8 ANN pretraining description
- Primitives pretraining taught individual rules separately within sensory, logic, and motor domains before later task combinations.Motor pretraining mapped an activated motor rule to its corresponding response, including the NOT transformation.Source: §A.8 · p. 18
- Logic primitives encoded relations such as BOTH/AND and EITHER/OR through Boolean responses, while sensory primitives used True/False judgments for rule-stimulus matches.Source: §A.8 · p. 18
- Simple task pretraining combined either sensory with motor rules or logic with sensory rules in two-rule contexts.The sensorimotor variant taught mappings to motor responses, whereas the logical-sensory variant taught Boolean inference over paired stimuli.Source: §A.8 · p. 18
- Primitives pretraining normally preceded Simple task pretraining, and reversing this order reduced generalization to chance despite matched stopping criteria.The standard order was also motivated by easier-to-more-difficult learning and interleaving to avoid catastrophic forgetting.Source: §A.8 · p. 19
A.9 ANN analysis
ANN analyses measured novel-context generalization, sample efficiency, and abstract representations, comparing pretrained networks with human fMRI organization across cortical and hidden-layer hierarchies.
A.9 ANN analysis
- Generalization performance was measured on novel rule recombinations, while sample efficiency counted trials needed to reach baseline accuracy on trained contexts.Generalization inefficiency was the normalized sample count divided by normalized novel-context performance.Source: §A.9 · p. 19
- ANN Parallelism Score was computed from rule-only hidden activations, treating hidden units as spatial features analogous to voxels within brain parcels.Stimulus inputs were zeroed to isolate rule representations.Source: §A.9 · p. 19
- Pairwise Parallelism Scores were computed for all rule dichotomies in both pretrained ANNs and human fMRI data.Source: §A.9 · p. 19
B Supplementary Figures
- With a fixed number of samples per learned context, pretrained models eventually generalized more efficiently than the initially more efficient Vanilla model.Source: Figure 3 · p. 21
- Reversing the pretraining order impaired unseen-task generalization and increased pretraining samples despite identical 99% accuracy stopping criteria.Source: Figure 4 · p. 21
- Human and ANN analyses showed analogous hierarchical organization, with sensory, logic, and motor abstractions strongest in sensory, association, and motor systems or corresponding layer depths.Supplementary analyses extended this comparison across pairwise dichotomies in logic, sensory, and motor rule domains.Source: Figure 5 · p. 22, Figure 7 · p. 23, Figure 8 · p. 23
- Sensory, association, and motor cortical systems were matched to three discretized hierarchical systems for comparing human fMRI with ANN layers.Source: Figure 6 · p. 22