Break It Down: Evidence for Structural Compositionality in Neural Networks

• Neural networks may either implicitly learn compositional solutions during training or rely on learned templates without explicit symbolic mechanisms.Source: §1 · p. 1
• The paper introduces structural compositionality as testing whether neural networks decompose compositional tasks into modular subroutines across language and vision.Source: §1 · p. 2
• Many models implement subroutines in modular subnetworks, although most do not exhibit perfect task decomposition.Source: §1 · p. 2
• Unsupervised pretraining leads to a more consistently compositional structure in language models.Source: §1 · p. 2
• The study applies model-pruning techniques to complex language and vision models to investigate the algorithms encoded in their weights.Source: §1 · p. 2

• Generalization failures alone cannot reveal whether a model uses modular representations or noncompositional templates.Source: §2 · p. 2
• In the Inside-Contact task, a model can either compare inputs with prototypes or decompose the task into Inside and Contact detectors.Source: §2 · p. 2
• Structural compositionality predicts subnetworks that compute one subroutine while leaving the other largely intact after ablation.Source: §2 · p. 2
• A noncompositional model should lack subnetworks selective for one subroutine, and ablating one should impair both subroutines similarly.Source: §2 · p. 3

3.1 Preliminaries

• A subnetwork is represented by a binary mask over base-model parameters, while the ablated model contains the mask’s complement.Source: §3.1 · pp. 3–4

3.2 Experimental Logic

• Base models are trained on compositional odd-one-out tasks to exceed 90% accuracy, then assessed for modular Inside and Contact implementations.Source: §3.2 · p. 4
• The learned subnetwork is evaluated on target-subroutine cases and cases requiring the other subroutine.Source: Figure 2 · p. 4, §3.2 · p. 5
• Structural compositionality predicts higher subnetwork performance on the target than the other subroutine, whereas entanglement predicts equal performance.Source: §3.2 · p. 5
• Ablating a compositional subnetwork should hurt target-subroutine performance more than other-subroutine performance, producing a negative target-minus-other difference.Source: §3.2 · p. 5

• The method learns a small binary mask over a frozen model’s weights to discover a subnetwork implementing a specified subroutine.Source: §4 · p. 5
• It uses deterministic continuous sparsification to discover subnetworks, following evidence that it can improve pruning sparsity and subnetwork performance.Source: §4 · p. 5

5 Vision Experiments

• Vision tasks compose the Inside, Contact, and Number subroutines into Inside-Contact, Number-Contact, and Inside-Number rules.Source: §5 · p. 5
• In the Inside-Contact task, images combine inside and contact properties, with (+ Inside, + Contact) rule-following and other combinations serving as odd-one-out cases.Source: §5 · p. 6
• Separate odd-one-out tasks discover +/- Inside and +/- Contact subnetworks, while target and other-subroutine datasets test whether each subnetwork is selective.Source: §5 · p. 6
• Models use a backbone and two-layer MLP to embed four images, predict the least-similar image, and train sparse binary masks for subnetwork discovery.Source: §5 · p. 6
• The ablation model is formed by subtracting each learned subnetwork from the compositional base model before evaluating target and other-subroutine performance.Source: §5 · p. 7

6 Language Experiments

• Language tasks use four sentences, three rule-following and one odd-one-out, to compose long-distance Subject-Verb Agreement and Reflexive Anaphora rules.Source: §6 · p. 7
• For singular Subject-Verb Agreement, models treat singular-subject singular-verb sentences as rule-following and mismatched number combinations as odd-one-out cases.Source: §6 · p. 7
• BERT-Small language experiments proceed analogously to vision experiments, using the [CLS] token representation as the sentence embedding and omitting the MLP.Source: §6 · p. 7

7 Results

• Subnetwork and ablation accuracy differences follow the predicted structural-compositionality pattern across the analyzed Resnet50 and BERT models.Subnetworks perform better on their target subroutine, while ablation reverses the target-versus-other performance difference.Source: §7 · p. 7, Figure 4 · p. 8
• ViT failed to exceed 90% performance on any compositional task and was excluded from subsequent analyses.Source: §7 · p. 7
• Some architecture-task combinations significantly favor structural compositionality, with high same-subroutine overlap and low overlap between different subroutines.Source: §7 · p. 7
• Evidence is mixed across tasks: Number-Contact Resnet50 models show strong structural compositionality for Contact but little evidence for Number.The results suggest Contact is implemented in a small modular subnetwork, whereas Number is represented more diffusely.Source: §7 · p. 8
• Random-model controls found no significant difference from the observed pattern in panels A, B, and F, while other panels differed significantly.Source: §7 · p. 8

8 Effect of Pretraining on Structural Compositionality

• For Resnet50, the study pretrains models on the experimental data using SimCLR before comparing them with models trained from scratch.Source: §8 · p. 8
• Across language tasks, pretrained BERT-Small models more reliably produce modular subnetworks than randomly initialized models.Source: §8 · p. 8
• Vision results do not suggest a benefit from pretraining.Source: §8 · p. 8
• Figure 5 reports that pretraining produces more modular subnetworks and appears to make subnetwork discovery more robust to random seeds.Source: Figure 5 · p. 9

9 Related Work

• Unlike approaches using auxiliary classifiers or assumed network locations, this method probes internal structure without those assumptions.The authors distinguish structural analysis from behavioral compositional-generalization studies and from component-attribution methods.Source: §9 · p. 9
• Mechanistic interpretability provides the broader framework for reverse-engineering algorithms implemented in neural-network weights.The paper applies this perspective to modular subroutines rather than only generic mechanisms for combining input features.Source: §9 · p. 9
• Earlier studies found that subnetworks solving particular SCAN and Mathematics Dataset partitions often failed to generalize across partitions.Csordás et al. (2021) therefore concluded that neural networks did not flexibly combine subroutines for full compositional generalization.Source: §9 · p. 9
• The paper extends modularity analyses by locating subnetworks for intermediate compositional subroutines in more complex vision and language tasks.This contrasts with earlier multitask work that did not search for subnetworks implementing specific operations such as SCAN’s “repeat.”Source: §9 · p. 10

10 Discussion

• Models often exhibit structural compositionality by decomposing tasks into subroutines implemented in modular subnetworks, without explicit encouragement.The results suggest that some pseudo-symbolic computations can be learned directly from data through standard gradient-based optimization, bearing on the role of explicit symbolic mechanisms.Source: §10 · p. 10

A Full Results

• The full-results appendix organizes evaluations of base and pretrained models, Wide Resnet50 subnetworks, vision pretraining, and absolute subnetwork and ablation accuracies.These results are presented in Tables 2–3 and Figures 6–12.Source: A Full Results · p. 15

A.1 Statistical Analysis of Main Results

• Statistical analysis finds significant nonzero performance for discovered subnetworks across architectures, tasks, and subroutines, and often for ablated models despite small samples.The analysis uses generalized linear models with robust clustered standard errors and tests whether subnetwork or ablated-model performance differs from zero.Source: §A.1 · p. 15

B Continuous Sparsification: Extended Discussion

• Continuous sparsification learns near-discrete masks by annealing sigmoid gates toward Heaviside gates while applying an L0-inspired sparsity penalty.The method is applied to frozen models to reveal internal structure, rather than jointly training model weights and masks.Source: B Continuous Sparsification: Extended… · pp. 18–19

C Mask Hyperparameter Search Details

• Mask selection requires extensive per-model and per-subroutine hyperparameter searches, retaining configurations that reach at least 90% task accuracy and maximize structural-compositionality scores.The complete search requires approximately 2448 GPU-hours and uses multiple learning rates, initializations, and mask configurations.Source: C Mask Hyperparameter Search Details · pp. 19–20

D ViT Hyperparameter Search Results

• Table 4 reports results of a ViT hyperparameter search, with all accuracies rounded to the nearest percentage point and computed on the relevant dataset’s validation set.Source: Table 4 · p. 21

E Vision Stimuli

• The vision stimuli include a +/- Number subroutine in which image classes differ according to whether they contain N or M shapes, with M ≠ N.Other vision subroutines are described elsewhere in the paper.Source: E Vision Stimuli · p. 21

F Language Data Details

• Language datasets use Marvin & Linzen (2019) templates while excluding constructions that would make subject-number or reflexive-anaphora subroutines ambiguous.The exclusions place relevant nouns in a consistent position and isolate subroutine organization rather than claims about the exact heuristic used to implement each subroutine.Source: F Language Data Details · pp. 21–22

G Vision Pretraining Details

• SimCLR pretrains ResNet50 and an MLP by generating two randomly augmented image views and maximizing their representational agreement with a contrastive loss using temperature 0.07.Augmentations include horizontal flips, affine transformations, color jitters, rotations, and grayscaling; training lasts 100 epochs with learning rate .0005, cosine annealing, and batch size 256.Source: G Vision Pretraining Details · p. 26

H Subnetwork Sparsity Data

• The sparsity appendix reports each subnetwork’s masking stage, active parameter count, and total number of parameters covered by its binary mask.Masking begins at stage 0, 3, or 4, and active parameters are the mask’s entries equal to 1.Source: H Subnetwork Sparsity Data · p. 26

I Control Experiment: Random Models

• The control repeats the mask-training procedure on randomly initialized models using comparable hyperparameters for each model–subroutine pair.The experiment tests whether performant subnetworks could arise independently of representations learned by the base model.Source: I Control Experiment: Random Models · pp. 26, 30
• Random-network masks produce patterns unlike trained-model results, and their ablations do not show the expected compositional behavior.ResNet50 ablations collapse performance to chance, whereas BERT-Small ablations can show the opposite target-versus-other trend; this supports a trained-model interpretation.Source: I Control Experiment: Random Models · pp. 30–31, Figure 16 · p. 34

I.1 Statistical Analysis of Main Results vs. Random Results

• A clustered-error GLM compares trained and random conditions across architectures, tasks, subroutines, and subnetwork-versus-ablation outcomes.The analysis also uses linear hypothesis tests to assess whether ablated-model performance differs between trained and random models.Source: §I.1 · pp. 31–32, Table 12 · p. 33

J Pruned Model Analysis

• Pruned ResNet50 models retain structural-compositionality results closely resembling those of the corresponding full models.The analysis trains sparse masks on Number-Contact models and reruns the main structural-compositionality procedure.Source: J Pruned Model Analysis · p. 32

K Subnetwork Overlap Analysis

• The analysis compares three subnetworks for each Inside and Number subroutine, then compares intersections of the two task-specific subnetwork sets.The model is a ResNet50 trained on Inside-Number, selected because masking began at the same layer for both subroutines.Source: K Subnetwork Overlap Analysis · p. 34, Table 13 · p. 36, Table 14 · p. 36
• Inside subnetworks show near-ceiling agreement, whereas Number subnetworks have substantially lower agreement and appear noisier.These within-task differences qualify the broader modularity pattern.Source: K Subnetwork Overlap Analysis · p. 34
• Random BERT-Small masks can preserve target-versus-other accuracy differences after ablation, unlike the expected pattern for a compositional model.This control result contrasts with the within-versus-between-task overlap evidence from trained models.Source: Figure 17 · p. 35
• Within-task subnetwork overlap exceeds between-task overlap, especially before the final MLP, reinforcing a modular organization within the base model.The comparison is quantified using per-layer intersection over union (IoU) across discovered subnetworks.Source: K Subnetwork Overlap Analysis · pp. 34–35, Figure 19 · p. 38