Neural networks are more modular than single neurons suggest

1 Introduction

• Selective neurons are a proxy for modularity, but syntax and semantics have not reliably separated at the individual-voxel level in the human language network.Whether those functions dissociate at another level of description remains open.Source: §1 · pp. 1–2
• Networks with mixed-selective units can still organize independent computations into separable subspaces, so unit-level specialization alone does not determine whether computation is modular.The paper frames subspace modules as rotated off the coordinate axes.Source: §1 · p. 2
• The proposed hierarchy distinguishes unit-level modularity, orthogonal modularity revealed by rotation, and linear modularity revealed by a bounded invertible transformation.The levels are formally nested, from disjoint neurons to potentially non-orthogonal subspaces.Source: §1 · p. 2
• The paper distinguishes functional modularity from distributed coding and superposition: its test is whether one computation can be selectively disrupted while another is spared, and vice versa.The criterion is a double dissociation, rather than how features are distributed across representational coordinates.Source: §1 · p. 3
• Across RNNs, vision models, and language models, the framework finds subspace-level modularity even where unit-level modularity is absent.Content and style, and syntax and semantics, separate despite overlapping neural populations; RNN decompositions recover double dissociations.Source: §1 · p. 3

2.1 A hierarchy of linear decompositions

• The hierarchy tests whether functional separation is possible by permutation, orthogonal rotation, or a bounded-condition-number invertible map, with the least constrained successful family indicating the representation’s geometry.The decomposition retains a residual subspace for information beyond the two target functions, avoiding conflation of decomposition with compression.Source: §2.1 · p. 3

2.2 Training objectives

• For RNNs with known task mappings, jointly trained task readouts require each subspace to retain enough information to perform its corresponding task.The objective combines task losses with level-specific regularization.Source: §2.2 · p. 4
• For pretrained vision and language models, contrastive learning uses minimal pairs that vary along one target axis while holding the other fixed, assigning separate losses to the two subspaces.Examples include stylizations of the same scene for content and style, and paraphrases or grammatical-template matches for semantics and syntax.Source: §2.2 · p. 4

2.3 Evaluation

• RNN decompositions are evaluated by ablating each target subspace during inference and checking for selective impairment of its task while sparing the other.The ablation is applied at every timestep, then the representation is mapped back through the inverse transformation.Source: §2.3 · p. 4
• Vision and language decompositions are evaluated by nearest-neighbor retrieval, measuring whether each subspace preferentially retrieves examples with its intended property.The reported method uses penultimate-layer representations, averaged over spatial positions for vision or tokens for language unless otherwise noted.Source: §2.3 · pp. 4–5

3.1 Multitask cognitive RNNs (Yang’s task)

• Tanh networks rise from DDI = 0.00 at unit level to 0.37 orthogonal and 0.47 linear, while softplus networks show double dissociations at all levels.Softplus networks show |FTV| = 0.74, with 74% of units specialized; tanh networks show |FTV| = 0.39, with 30% specialized.Source: §3.1 · pp. 5–6

3.2 Content-Style Modularity in Vision Models

• Rs = 16.1 and Rc = 5.3 under invertible transforms, versus near-chance selectivity for permutation and moderate selectivity for orthogonal decompositions.Invertible transforms consistently achieve the strongest content–style separation across several vision architectures, with ResNets an exception where style retrieval is weak across subspace types.Source: Table 1 · p. 7, §3.2 · pp. 7–8

3.3 Syntax–Semantics Modularity in Language Models

• Across syntax–semantics tests, invertible transformations produce the strongest modularity, followed by orthogonal and then permutation-based decompositions.For Pythia 410M, selectivity persists across layers, and syntax retrieval remains faithful despite little token overlap; semantic paraphrases often share tokens.Source: Table 2 · p. 8, §3.3 · p. 8

3.4 Neural Alignment of Learned Subspaces

• Style better predicts V1/V2, content higher visual areas; semantic features outperform syntactic ones in language cortex, while original sentence representations predict best overall.Across modalities, caption semantics better predict visual cortex, and image content predicts language cortex close to original sentence representations.Source: Figure 8 · p. 9, §3.4 · pp. 9–10

4 Discussion

• Subspace-level modularity can make specific computations selectively accessible and intervenable through a known linear transformation, even without dedicated neurons.The implications of modularity for robustness, generalization, and learning dynamics remain open.Source: §4 · p. 10
• Decomposing representations into functional subspaces allows analysis of which components align with brain data, and may clarify whether cross-model convergence reflects shared semantic structure.The evolution of subspace structure across training and layers, and extensions to multi-way decompositions, remain unexplored.Source: §4 · p. 10
• Residual information can contain or support task-relevant information, but appears shared or redundant rather than selectively associated with either task.Its functional organization and role remain open questions.Source: §4 · p. 10

E.1 Differentiable Constraints via GeoTorch and Sinkhorn Relaxation

• Geometric constraints are imposed through a parametrization strategy rather than by directly optimizing constrained parameters.For orthogonal and invertible transformations, the implementation uses the GeoTorch library.Source: §E.1 · p. 14
• Unconstrained parameters are maintained internally and mapped through a differentiable, manifold-specific function whenever the weight is accessed.Source: §E.1 · p. 14
• For orthogonal and invertible transformations, the parametrization strategy uses GeoTorch.Source: §E.1 · p. 14

E.1.1 Orthogonal Constraint (geotorch.orthogonal)

• The target manifold for the orthogonal constraint is the Special Orthogonal group, SO(n).Source: §E.1.1 · p. 14
• GeoTorch forms a skew-symmetric matrix from the unconstrained raw parameter and applies a trivialization map to produce a valid orthogonal matrix.Source: §E.1.1 · p. 14
• The default trivialization is the matrix exponential; the Cayley map is cheaper for large n but cannot represent orthogonal matrices with eigenvalue −1.The exponential map is surjective, so every orthogonal matrix is reachable.Source: §E.1.1 · p. 14

E.1.2 Linear Invertible Constraint (geotorch.invertible)

• The target manifold for the invertible constraint is the General Linear group with positive determinant.Source: §E.1.2 · p. 14
• The constraint is imposed through an SVD-based parametrization with orthogonal factors and strictly positive singular values.Source: §E.1.2 · p. 14
• Two independent skew-symmetric raw parameters generate the orthogonal factors through the matrix exponential or Cayley map, while softplus maps a raw vector to positive singular values.The constrained weight is then reconstructed from these components.Source: §E.1.2 · p. 14

• Direct optimization over d × d permutation matrices is combinatorial, so training uses a differentiable relaxation over the Birkhoff polytope.Source: §E.1.3 · p. 14
• Sinkhorn normalization alternates row and column normalization from exp(A), producing a doubly-stochastic matrix S that is differentiable with respect to A.Source: §E.1.3 · p. 15
• After training, the relaxed S is projected to the nearest permutation P with Hungarian linear assignment, yielding a hard, discrete axis-aligned decomposition for inference.Source: §E.1.3 · p. 15

• Semantic paraphrases show a broader spread of same-position token overlap, whereas syntactic counterparts cluster near zero overlap.The comparison covers sentence pairs within each semantic or syntactic set in the Pereira dataset.Source: Figure E3 · p. 17
• 800 MS-COCO images were paired with 800 style images to create 640,000 stylized training pairs, with 10,000 pairs each held out for validation and testing.Content pairs preserve object identity and spatial layout while changing surface appearance; style pairs preserve texture, palette, or rendering while changing the depicted object or scene.Source: Figure E1 · p. 15, §E.2 · p. 17
• 3,900 MSR paraphrase pairs supplied semantic counterparts, and one syntactic counterpart per anchor was generated with Gemini 2.5 Flash and manually inspected.Semantic counterparts preserve meaning across syntactic forms; syntactic counterparts preserve grammatical structure while changing lexical content and meaning.Source: Figure E2 · p. 15, §E.2 · p. 17
• The language inference set uses 627 Pereira sentences from five participants who completed Experiments 2 and 3, with 70 semantic and 70 syntactic counterparts generated per sentence.Source: §E.2 · p. 18
• The RNN analyses include 256-unit continuous-time networks for Mante’s context-dependent motion-versus-color task and Yang’s multitask cognitive paradigm.In the Mante task, networks receive noisy motion and color evidence and use a binary context signal to determine which direction to report; subspaces are tested by selective ablation across the two decisions.Source: §E.2 · p. 18
• The Yang analysis compares Anti versus DM1 as disjoint, DM1 versus DlyDM1 as inclusive, and DM1 versus CtxDM1 as mixed in shared computation.The networks cover 11 cognitive tasks; DlyDM1 adds a delay and working memory to DM1, while CtxDM1 adds context-dependent input gating to DM1’s comparison machinery.Source: §E.2 · p. 18

• The ResNet family includes ResNet-50, ResNet-34, ResNet-18, and ResNet-101, using skip connections in deep convolutional networks [He et al., 2016].Source: §E.3.1 · p. 19
• ConvNeXt Tiny, Base, and Large are modernized CNNs incorporating ideas such as large kernels and inverted bottlenecks [Liu et al., 2022].Source: §E.3.1 · p. 19
• Swin Small, Tiny, Base, and Large are hierarchical vision Transformers that process images with shifted windows [Liu et al., 2021].Source: §E.3.1 · p. 19
• The ViT family includes Small, Tiny, Base, and Large models that treat images as sequences of non-overlapping patches [Dosovitskiy et al., 2020].Source: §E.3.1 · p. 19
• DINO Small and Base are self-supervised representations learned through knowledge distillation and support both ViT and ResNet backbones [Caron et al., 2021].Source: §E.3.1 · p. 19
• The VGG roster comprises VGG 11, 13, 16, and 19; VGG-16 has 13 convolutional and 3 fully connected layers [Simonyan and Zisserman, 2014].Source: §E.3.1 · p. 19

• GPT-2 XL is a decoder-only Transformer trained with next-token prediction and is described as a baseline autoregressive language model [Radford et al., 2019].Source: §E.3.2 · p. 19
• Pythia 410M was trained on approximately 300B tokens, and the experiments use its final available checkpoint at step 143,000 [Biderman et al., 2023].Source: §E.3.2 · p. 19
• Pythia 160M was trained on approximately 300B tokens, and the experiments use its final available checkpoint at step 143,000 [Biderman et al., 2023].Source: §E.3.2 · p. 19

E.4.3 Yang multitask cognitive paradigm

• For the disjoint Anti–DM1 pair, softplus networks had |FTV| = 0.74 and 74% specialized units, versus tanh networks’ |FTV| = 0.39 and 30% specialized units.Both activation types nevertheless achieved >95% accuracy on all 11 tasks.Source: §E.4.3 · p. 21

E.4.4 Decomposition training; E.4.5 Causal evaluation; E.4.2 Mante context-dependent integration task

• For each task pair and decomposition level, modules were trained with a task-sufficiency objective, then evaluated on held-out RNN trials by ablating a subspace, reconstructing hidden states, and measuring task accuracy.Trained networks achieved 85–90% accuracy on non-zero-coherence trials, where attended coherence was nonzero and a correct answer existed.Source: §E.4.2 · p. 20, §E.4.4 · p. 21, §E.4.5 · pp. 21–22

E.4.6 Targeted dimensionality reduction (TDR) baseline

• TDR axes predicted motion and color well (R2 = 0.77 ± 0.01 and R2 = 0.76 ± 0.01) and were near-orthogonal, but ablating any single axis changed accuracy by less than 0.5%.The context axis had R2 > 0.999; the results distinguish information predicted by a direction from a direction causally necessary for computation.Source: §E.4.6 · p. 22

E.6 Context-dependent integration (Mante task)

• At d = 15, linear decomposition reached DDI = 0.277±0.026 and produced symmetric motion–color double dissociations, while unit-level and orthogonal DDI stayed near zero.Ablating the motion subspace reduced motion accuracy by ∼39% with collateral color impairment below 4%; the linear DDI rose with dimension and plateaued around d ≈15–20.Source: §E.6 · pp. 22–23, Figure E4 · p. 23

E.7 Evaluation of style and content subspaces on Silhoutte-Edge Stimuli; E.8 Evaluation of style and content subspaces on Conflicting Cues

• On silhouette–edge retrieval, content outperformed style for instance- and category-level criteria; on conflicting-cue images, content retrieved shape categories and style retrieved texture categories.Style retrieval accuracy was high across both conflicting-cue subspaces, partly because the task required only coarse texture-category consistency.Source: §E.7 · p. 23, §E.8 · p. 24

E.9 Sensitivity to the Flexibility of the Learned Linear Map

• Flexible linear maps did not guarantee modularity: random maps and shuffled-label fits failed to match learned DDI, while successful dissociations persisted across broad dimensional allocations and condition numbers.Yang inclusive and mixed task pairs remained nonmodular even under their most favorable tested settings; visual content/style selectivity remained high when neither subspace exceeded approximately 40% of representation dimensions.Source: §E.9 · pp. 24–25
• Content–category dissociation fails while content–style dissociation is replicated, and content remains similarly retrievable from both content–category subspaces.The authors conclude that the learned transformation does not automatically produce modularity; the tested overlapping pair remains non-dissociable.Source: §E.9 · pp. 26–27

E.10 What does the residual subspace contain?

• The residual subspace achieves the highest style retrieval, while its ablation reduces motion and color performance similarly, indicating shared or redundant task-relevant information.The residual is retained by reconstruction loss rather than task-specific supervision and may contain nuisance, shared, or redundant information.Source: §E.10 · p. 27

E.11 Licenses and terms of use for existing assets

• Experiments use publicly available datasets and pretrained checkpoints from official release sites or repositories, with licenses or access terms summarized in Table E13.For assets governed by access agreements or dataset-specific terms, the authors report the official terms of use.Source: §E.11 · p. 38

E.12 Impact Statement

• The impact statement identifies potential implications of the analytical framework for interpretability, model comparison, and computational neuroscience.The work examines whether modular structure can emerge through orthogonal and invertible transformations even when absent from native representations.Source: §E.12 · p. 38

NeurIPS Paper Checklist

• The current submission does not yet include a public code release or complete reproduction package, and the authors plan to release code and detailed reproduction instructions in the camera-ready version.They report adding error bars for all plots in sections 3.4 and 3.1; for sections 1 and 2, experiments used a single split, so error bars cannot be provided at the time.Source: NeurIPS Paper Checklist · pp. 41–42
• The authors mark safeguards as not applicable because they analyze existing pretrained models and public datasets and release no new high-risk model, scraped dataset, or generative system.Source: NeurIPS Paper Checklist · p. 43
• The authors answer yes that existing assets are credited and their licenses or official access terms are summarized in Appendix Section 10, Table E13.Source: NeurIPS Paper Checklist · p. 44
• The authors report that no new dataset, model, or documented software asset is released in the submission.They describe the contribution as a framework and empirical analysis built on existing assets.Source: NeurIPS Paper Checklist · p. 44
• The authors report no new crowdsourcing or human-subject data collection, relying instead on previously collected public datasets with human annotations or neural recordings.They state that the paper does not involve new human-subject experimentation.Source: NeurIPS Paper Checklist · pp. 44–45
• The authors mark LLM usage as not applicable to the core method, stating that any LLM use was limited to writing, editing, and formatting support.Source: NeurIPS Paper Checklist · p. 45