Transformers converge to invariant algorithmic cores

Joshua S. Schiffman

Source-linked AI summary

Transformers converge to invariant algorithmic cores

Joshua S. Schiffman

arXiv:2602.22600v2Feb 26, 2026cs.LGcs.AI

TL;DR

Mechanistic interpretability faces underdetermination because models can realize the same behavior through different internal structures. The paper introduces ACE to extract and validate task-relevant algorithmic cores, finding compact invariant mechanisms across synthetic tasks and six language models, including a shared one-dimensional agreement core.

Problem
Mechanistic interpretability lacks evidence that internal mechanisms generalize across different realizations implementing the same behavior.
Method
ACE extracts active, task-relevant activation subspaces, causally validates them with ablations, and fits operators to identify their computations.
Results
ACE recovers compact invariant mechanisms across synthetic tasks and six pretrained language models, including a shared one-dimensional core that causally steers subject–verb agreement.
Takeaways & Limitations
The findings shift mechanistic analysis from realization-specific circuitry toward invariant computational cores that can support interpretation and control.
Takeaways & Limitations
Whether algorithmic cores remain low-dimensional for multi-step reasoning tasks is untested, and systematic task decomposition in multifunctional models remains open.

Abstract

from arXiv · show

Training selects for behavior, not circuitry: many weight configurations can implement the same function. Studying any single trained neural network thus risks describing accidents of one training run rather than the computation itself. This work shifts focus from what transformers happen to do to what they must do by extracting algorithmic cores, compact subspaces that are necessary and sufficient for a task and that recur across independently trained models. Here, Algorithmic Core Extraction (ACE) is introduced to isolate these subspaces, causally validate them, and recover the algorithms they implement across settings ranging from synthetic tasks to large-scale pretrained models. Markov-chain transformers embed three-dimensional cores in nearly orthogonal subspaces yet recover identical transition spectra. Modular-addition transformers form compact cyclic cores at grokking that later inflate under continued regularization, redundantly distributing the same computation across many functionally equivalent modes. This functional redundancy is found to accelerate the transition from memorization to generalization, yielding an inverse scaling law for grokking time. In six language models spanning more than two orders of magnitude in scale (GPT-2 Small/Medium/Large, LLaMA-3.1, Gemma-2, and Qwen2.5), subject-verb agreement is governed by a single, steerable axis that aligns across architectures. Flipping this axis inverts grammatical number throughout open-ended generation. Together these results suggest that beneath the apparent complexity of trained transformers lies a simpler, shared computational structure, and that targeting invariants rather than parameterizations may offer a more tractable path to mechanistic understanding and control. Code: https://github.com/joshseth/cores

1. Introduction

The paper frames mechanistic interpretability as an underdetermined problem: training constrains behavior but generally not its internal realization. It introduces ACE to identify compact, causally necessary and sufficient algorithmic cores that recur across independent models, including a shared one-dimensional core for subject–verb agreement.

Motivation: Training constrains input–output behavior but generally does not constrain how that behavior is realized internally, creating underdetermination for mechanistic interpretability.This raises whether explanations found in one realization generalize to others.
Method: Algorithmic Core Extraction (ACE) isolates low-dimensional subspaces that are necessary and sufficient for a task and shared across independent realizations.The method shifts analysis from realization-specific circuitry toward invariant mechanisms.
Findings: Across settings of escalating complexity, ACE shows that functionally equivalent models can converge on compact, invariant mechanisms and recovers ground-truth Markov chain dynamics in single-layer transformers.These applications provide evidence that extracted cores can isolate interpretable mechanisms.
Language-model application: ACE identifies a shared, one-dimensional core that causally steers subject–verb agreement during open-ended generation across six distinct language models.The core is presented as a steerable mechanism shared across architectures.

2. Methods

Algorithmic Core Extraction (ACE) adapts minimal-realization ideas to transformers by isolating directions that are jointly active and task-relevant. It validates these cores causally and recovers their computational structure from core coordinates or fitted operators.

Extract: ACE adapts the Kalman decomposition’s minimal-realization principle to identify compact transformer subspaces implementing task functions.The approach is motivated by many-to-one structure–function relationships and empirical recovery through balanced truncation.
Extract: ACE computes activation-task Jacobian interactions and uses their singular vectors to find directions that are jointly active and relevant.For a layer with hidden dimension D, mean-centered activations H are paired with task-output Jacobians J before singular-value decomposition.
Extract: Singular values rank directions by joint activity and relevance, while the leading r directions define the algorithmic core.QR decomposition converts the selected directions into an orthonormal basis Q and core projector P := QQ⊤.
Validate: A core is sufficient when projection Ph preserves task performance and necessary when the complementary component h −Ph reduces performance to near chance.These projection and complement tests provide causal validation of the extracted subspace.
Identify: ACE recovers computational structure by examining core coordinates z = Q⊤h or fitting operators such as z_t+1 ≈ Az_t and inspecting their spectra.Operator fitting can use least squares to characterize the dynamics represented within the core.

3. Algorithmic Core Necessity and Sufficiency

Across independently trained transformers solving the same four-state Markov task, ACE recovers compact three-dimensional cores that are causally necessary and sufficient for performance. These cores occupy nearly orthogonal parameter subspaces yet statistically encode equivalent Markov dynamics up to a change of coordinates.

Recovering algorithmic cores: Three independently seeded transformers with dmodel = 64 and dff = 256 learned near-zero-cosine-similarity weights while achieving near Bayes-optimal test accuracy.ACE was applied to their 64-dimensional hidden states to search for shared internal representations.
Geometric dissimilarity, statistical equivalence: Projector overlap was 0.02–0.04 and principal angles were 75◦–90◦, showing that equivalent cores occupied nearly orthogonal subspaces.These cores nonetheless met equivalent causal criteria for task performance.
Geometric dissimilarity, statistical equivalence: Mean cross-core CCA correlations reached near unity, indicating statistical alignment despite geometric divergence.Figure 1 reports projector overlaps near zero, nearly orthogonal principal angles, and mean cross-core CCAs reaching near unity.
Algorithmic cores encode Markov dynamics: R2 oracle > 0.98, and fitted core operators matched the Markov transition matrix’s non-trivial eigenvalues to within a few percent.The recovered 3D cores therefore encode transition dynamics up to a change of coordinates.
Recovering algorithmic cores: ACE extracted 3D cores that were necessary and sufficient for performance under ablation, despite substantial differences between learned parameterizations.The cores were recovered from 64-dimensional hidden states in the three one-layer transformers.

4. Algorithmic Core Emergence and Evolution

ACE reveals that modular-addition transformers crystallize compact rotational cores at grokking, then redistribute the same computation across increasingly redundant dimensions under continued weight decay. This redundancy both explains core inflation and predicts faster grokking as functional redundancy or weight decay increases.

Core emergence: At grokking, three modular-addition transformers crystallized low-dimensional cores that were causally necessary and sufficient for task performance.The models used dmodel = 128, dff = 512, |V| = 53 and generalized near epoch 800 after remaining near chance.
Core emergence: Least-squares operator fits recovered cyclic rotational dynamics without prespecifying an algorithm, although rotational modes differed across training runs.At grokking, fitted eigenvalues snapped onto the unit circle, while conjugate eigenvalue pairs varied between models.
Core evolution: Continued weight decay inflated cores from approximately 15 to 60 dimensions by increasing redundant encoding, while performance-sufficient dimensionality remained stable.Disabling weight decay after grokking kept cores more compact, indicating that regularization drives redundant saturation rather than simplification.
Redundancy and grokking: Minimum-norm optimization spreads weight across all functionally equivalent Fourier modes, making weight decay a redistribution force rather than a simplification force.For modular addition mod p, Fourier symmetry provides ⌊p/2⌋ equivalent 2D rotational modes, and the minimum-norm solution activates every mode.
Redundancy and grokking: Grokking time shrinks inversely with both weight decay and functional redundancy, with empirical sweeps validating the predicted dependence.The theoretical high-redundancy regime gives an inverse scaling law, while an architectural constant pcrit determines whether grokking can occur.

5. Scaling ACE to LLMs: A Universal 1D Core

Across six pretrained language models from four families, subject–verb agreement is governed by a one-dimensional causal core localized to late layers. The aligned, steerable axis tracks grammatical number and can invert agreement in open-ended generation.

Localizing a shared 1D agreement core: Across six models from four families, ACE identifies a highly potent, one-dimensional agreement core in late layers, while early layers show minimal causal influence.The models include GPT-2 Small, Medium, Large, LLaMA-3.1, Gemma-2, and Qwen2.5.
Causal validation and control: The core axis acts as a graded number coordinate: its projection predicts the singular–plural logit margin across models.This behavior aligns with the linear representation hypothesis.
Causal validation and control: Core inversion changes “The key next to the cabinets” from P(is) = 0.51 to 0.01 and P(are) = 0.06 to 0.71.These prompt-level changes provide causal control over grammatical number.
Alignment across LLMs: Agreement coordinates align tightly within GPT-2, with Spearman’s ρ = 0.88–0.92 and Pearson’s r = 0.92–0.97.Because the cores are one-dimensional, cross-model alignment reduces to fixing a sign convention and comparing projected coordinates.
Core steering inverts grammar in open-ended text: Adaptive core-axis steering induces systematic agreement violations throughout open-ended generation across all six models.The intervention is applied at each decoding step and modulated according to each token’s sensitivity to number agreement, leaving irrelevant tokens untouched.
Core steering inverts grammar in open-ended text: The shared core codes a global grammatical-number variable rather than a narrow, verb-specific heuristic.The summary characterizes this late-layer subspace as necessary, sufficient, controllable, and aligned across six models from four families.

6. Discussion · Appendix

The discussion frames transformer computation as low-dimensional invariant structure that recurs across training runs, preserves task-relevant dynamics, and may better define mechanistic essence than implementation coordinates. It also highlights redundancy-driven grokking, representational drift, and open questions about whether these cores remain low-dimensional for complex reasoning.

6. Discussion: Transformer computations may be governed by low-dimensional mechanisms that recur across independent training runs despite substantial parameter variation.These findings motivate a different conceptualization of mechanistic interpretability.
6. Discussion: Algorithmic cores identify causal subspaces and preserved dynamics, whereas circuits and sparse features primarily describe implementation coordinates.When features or circuits recur, the approaches may converge; when they diverge, invariance distinguishes structural essence from artifact.
6. Discussion: Algorithmic cores may function as internal world models by recovering ground-truth task structure rather than merely mapping inputs to outputs.Examples include Markov transition spectra and cyclic operators for modular arithmetic.
6. Discussion: Redundancy accelerates grokking because functionally equivalent realizations populate a zero-loss manifold, allowing weight-decay pressure to accumulate across valid modes.Weight decay biases exploration toward minimum-norm, maximum-margin solutions after perfect training accuracy.
6. Discussion: Models from different initializations can implement identical cores in nearly orthogonal subspaces, revealing substantial representational drift despite functional agreement.This parallels system drift, where phenotype is preserved while underlying wiring diverges; recombining divergent solutions can produce hybrid incompatibility.
6. Discussion: The agreement core remains one-dimensional across six models spanning four architectures and 117M to 32B parameters, while multi-step reasoning remains untested.This result suggests core dimensionality may not depend on model scale and is compatible with effective low-dimensional LoRA updates.
6. Discussion: The paper concludes that transformer computation may be organized around low-dimensional invariants preserved across training runs, necessary and sufficient for task performance, and structured like the tasks.Whether this view scales to contemporary language-model complexity remains uncertain, but the proposed principle is to focus on what is preserved rather than what is particular.

A. Algorithmic Core Extraction

Algorithmic Core Extraction (ACE) adapts the minimal-realization principle from linear systems to transformers by identifying hidden directions that are both input-driven and output-relevant. It extracts a compact core subspace and causally tests whether that subspace is sufficient and necessary for task behavior.

Motivation: Functional equivalence allows different internal structures to implement identical input–output behavior, motivating a search for minimal realizations rather than individual parameterizations.In linear systems, only states that are both reachable and observable contribute to behavior; other states can vary without affecting function.
Algorithmic core extraction: ACE identifies transformer directions that are jointly active and relevant, extending the Kalman-decomposition principle without requiring a closed-form decomposition.Activity is estimated from hidden activations, while relevance is estimated from Jacobians mapping activations to task-relevant outputs.
Algorithmic core extraction: ACE computes singular modes of the interaction between activation activity and output relevance, then projects the leading modes back into activation space to define the core subspace.The singular values provide a criterion for selecting the core rank.
Causal validation: A subspace is sufficient when core-only interventions preserve task performance and necessary when removing the core reduces performance to approximately chance.The rank can be refined to the smallest value satisfying both causal criteria while maintaining baseline accuracy.
Activity and relevance: ACE can reduce to standard PCA when activity and relevance coincide, but it distinguishes variance concentration from task-relevant computation when they diverge.This distinction is especially important in complex models where high-variance directions may not align with the target task.

B. Markov Chain Experiment

The experiment trained three independently seeded causal transformers to predict sequences from a four-state Markov chain. It then extracted algorithmic cores and evaluated their representations, fitted dynamics, spectra, and prediction performance against chance and Bayes-optimal baselines.

Experimental setup: Three single-layer causal transformers were trained with independent seeds on next-token prediction for sequences generated by a four-state Markov chain.Each model used d_model = 64, d_ff = 256, and |V| = 4.
Markov-chain target: α = 0.75 and β = 0.25 produced transition eigenvalues λ ∈ {1, 0.75 + 0.25i, 0.75 − 0.25i, 0.5} and uniform stationary distribution π = [0.25, 0.25, 0.25, 0.25].The λ = 1 Perron–Frobenius eigenvalue was later excluded when comparing fitted core operators with ground truth because it reflects normalization.
Training protocol: Training used AdamW with learning rate 10^-3, no weight decay, 40 epochs, 3,000 sequences of length 32, and batch size 64.The sequences were generated by the transition matrix T.
Evaluation: Model performance was compared with chance accuracy and Bayes-optimal accuracy, representing most-common-token prediction and the best stochastic one-step prediction, respectively.Core operator fits were also compared against an oracle ceiling for next-token prediction.
Core extraction and dynamics: Cores were extracted at a 99.9% rank-energy threshold without ablation refinement, with H computed on all test activations and J defined by f(h) := logits(h).Mean-centered hidden-state sequences were projected into core coordinates z_t = Q^T h_t, and a linear operator was fit by least squares for next-step dynamics.

C. Modular Addition Experiment

The modular-addition experiment trained three transformers on addition modulo 53 and found grokking by epoch 800. ACE then extracted causally important cores and analyzed their cyclic transition operators and rotational spectra, including changes under continued weight decay.

Experimental setup: Three two-layer transformers learned a + b ≡ c (mod 53) task from all 2809 input pairs, split evenly into training and test sets.Each model used d_model = 128, d_ff = 512, and vocabulary size |V| = 53.
Training and grokking: Epoch 800 marked grokking, defined as the first analysis point when all three models achieved perfect test accuracy.Training used AdamW with learning rate 10^-3, batch size 512, weight decay ω = 1, and continued for 2 × 10^4 epochs.
Training and grokking: At epoch 900, each transformer was duplicated into maintained-decay and disabled-decay regimes to test continued weight decay after grokking.The two regimes used ω = 1 and ω → 0, respectively.
Core extraction: ACE selected core rank using a 99% energy threshold and refined it with ablations, using test activations and 64 Jacobian samples to assess causal importance.The Jacobian target function was f(h) := logits(h).
Operator analysis: A ridge-regularized linear operator was fit to centered answer-class core centroids after SVD reduction, with generalization tested on held-out cycle transitions.Calibration and evaluation used disjoint contiguous blocks of answer classes, and only transitions whose endpoints belonged to the relevant block were included.
Operator analysis: Rotational modes were eigenvalues of A with magnitude near 1, grouped into at most 27 frequency bins by angle, identifying cyclic spectral structure.Conjugate pairs were merged by mapping bins k and 53−k together, producing one k = 0 bin and 26 nonzero oscillatory bins.

D. Subject–Verb Agreement Experiment · E. Grokking Dynamics

Across six language models, subject–verb agreement is captured by a compact one-dimensional core whose ablation tests establish sufficiency, necessity, and grammatical-number inversion. The experiment extracts this core by layer sweeps and applies adaptive interventions during open-ended generation.

D. Subject–Verb Agreement Experiment: Six models spanning GPT-2 Small, Medium, Large, LLaMA-3.1, Gemma-2, and Qwen2.5 were analyzed for subject–verb number agreement.The models range from 117M to 32B parameters and from 12 to 64 layers.
D. Subject–Verb Agreement Experiment: Candidate cores were extracted at every layer, and the layer with maximal flip effect was selected as each model’s core location.Layer selection was based on ablation performance.
D. Subject–Verb Agreement Experiment: Open-ended generation used a per-token adaptive intervention that reflects the final-token hidden state across the hyperplane orthogonal to the unit-norm core axis.The steering strength is determined adaptively at each decoding step using the core axis and mean activation.
D. Subject–Verb Agreement Experiment: At each decoding step, a clean gating pass checks probability mass on agreement-relevant verbs before deciding whether intervention is applied.The relevant tokens are is, are, was, and were; low agreement-related probability mass yields no intervention.
D. Subject–Verb Agreement Experiment: A one-dimensional subject–verb agreement core emerged in every model despite substantial differences in training, parameterization, and architecture.Large spectral gaps support the conclusion that the extracted subspaces are effectively one-dimensional.
D. Subject–Verb Agreement Experiment: Core-only ablations preserve agreement, core removal collapses performance below chance, and core flipping induces near-perfect grammatical-number disagreement.The ablation results establish sufficiency, necessity, and causal control of agreement behavior.
D. Subject–Verb Agreement Experiment: When agreement is detected, steering strength is calibrated from the clean margin and a probing perturbation to produce a minimal margin flip.The generation margin uses logsumexp over singular-verb logits, reflecting probability-space competition between singular and plural verb groups.

E.1. Mathematical Model

The model describes post-memorization training as weight-decay drift opposed by a corrective force along ψ, with optimizer noise treated as zero-mean stochasticity. Under sufficient capacity, margin relaxation toward a generalization threshold predicts grokking time through separable architectural and optimizer constants.

Margin dynamics: Assuming p < dmodel and an initially unstructured memorized state, the model takes m(0) ≈ 0 and reduces margin evolution to a linear relaxation equation.The margin is defined as m(t) := ⟨α(t), ψ⟩.
Post-memorization dynamics: Post-memorization updates combine weight-decay drift −ωα(t), corrective motion γ(t)ψ maintaining zero loss, and zero-mean optimizer noise ξ(t).The corrective force is assumed to scale as γ(t) ≈ cω.
Direction of ψ: The minimum-norm update that increases the margin by one unit is parallel to ψ, so the corrective gradient direction is strictly aligned with ψ.This follows from the constrained minimum-norm problem and the Cauchy–Schwarz inequality.
Predicting grokking time: Grokking time is defined as the first hitting time τ at which the margin reaches the generalization threshold δ.The continuous gradient-flow time is obtained by solving m(τ) = δ.
Predicting grokking time: The discrete-time mapping separates pcrit := δ/κ as an optimizer-independent capacity floor from Ω∝(ηω)−1 as the optimizer-dependent relaxation timescale.Ω absorbs discrete step-size dynamics, learning rate, momentum, and AdamW adaptive preconditioning.

E.2. Grokking Sweeps and Scaling Fits

The study measures modular-addition grokking delays across weight-decay and modulus sweeps, then fits power laws and a deterministic ODE to estimate scaling exponents and macroscopic constants.

Measurement setup: Grokking delay is defined as τgrok := τgen − τmem, where memorization and generalization times are the first steps reaching 0.99 train and test accuracy.One-layer transformers use dmodel = 128, dff = 512, AdamW with lr=1e-3, and a 50/50 train/test split.
Experimental sweeps: The experiments average over 12 random seeds while sweeping weight decay at fixed p = 53 and modulus across seven prime values.The weight-decay sweep uses minibatch size B = 512, whereas the modulus sweep uses full-batch gradient descent because dataset size grows quadratically with p.
Scaling fits: Scaling exponents are estimated by ordinary-least-squares power-law fits in log–log space, while Ω and pcrit come from nonlinear fits of the deterministic ODE solution.The fitted ODE is τ(p) = −Ωlog(1 −pcrit/p), and goodness-of-fit is reported using R2.