PaperlayerView exact revision on arXiv

Source-linked AI summary

The Residual Stream's Effective Depth

Barak Gahtan, Ido Galil, Alex M. Bronstein

arXiv:2609.31098v1cs.LG

TL;DR

The paper asks how many statistically distinguishable residual states a transformer traverses and introduces effective depth, a scalar based on similarity across layer distance. Across decoder-only models, measurements are mostly below an orthogonal-update structural reference, with matched references tying the additional gap to correlated updates rather than evidence that depth is unused.

  • Problem

    Nominal depth is an imperfect surrogate for the number of statistically distinguishable residual states traversed by a transformer.

    Source: §1 · pp. 1–2

  • Method

    Effective depth summarizes CKA-based similarity autocorrelation across accumulated hidden states into a Bartlett-weighted full-lag scalar.

    Source: §4 · pp. 5–6

  • Results

    Across sixteen decoder-only language models, nearly every default measurement is below the orthogonal-update reference, and matched references attribute the gap largely to correlated updates.

    Source: §1 · p. 3, §6 · p. 12

  • Takeaways & Limitations

    Deff is best read as a calibrated global diagnostic of accumulated-state redundancy, not as evidence that only a few layers matter or as a pruning method.

    Source: §1 · p. 2, §7 · pp. 12–13

    Empirical claims are restricted to decoder-only autoregressive language models, and Deff is not a strict sampling-efficiency or beta-mixing estimate.

    Source: §1 · p. 3, §3 · p. 5

Abstract

from arXiv · show

We introduce \emph{effective depth} ($\Deff$), a scalar diagnostic that treats the layer-wise residual stream of a transformer as a discrete-time process, measures how representation similarity decays with layer distance, and aggregates that profile into one number. Across sixteen decoder-only language models, $\Deff$ separates a structural consequence of residual accumulation from an empirical one: even maximally diverse orthogonal updates have the closed-form reference $F_L = 2L/(L+1)<2$, yet fifteen of sixteen default measurements lie below $F_L$ (Qwen3.5: 32--44\%, OLMo-2: 40--41\%, Pythia: 23--28\%). Matched references show that the gap is not caused by the persistent initial state or update-size imbalance, but is largely a calibrated signature of correlated residual updates rather than evidence that depth is unused. Symmetric position-0, token-normalisation, and top-PC controls show the regime is not reducible to BOS or top-PC artefacts: the lone above-reference default outlier joins the same regime, and all sixteen models are sub-reference after token-normalisation or top-1-PC removal. Intermediate checkpoints show that the regime is established early in OLMo-2 and stable through 5T tokens, while Pythia-1.4B follows a distinct decreasing trajectory. A controlled residual-carry intervention supports the mechanism, and $\Deff$ is best read as a \emph{global} accumulated-state diagnostic, not as a capability score or pruning method.

1. Introduction

Although nominal depth is often treated as a proxy for expressive power, evidence from pruning and representation similarity motivates asking how many distinguishable residual states transformers traverse. The paper introduces effective depth as a scalar summary of accumulated-state similarity and interprets its measured gap below a structural reference as a calibrated signature of correlated updates, not evidence that depth is unused.

1. Introduction

  • Even mutually orthogonal updates yield Deff = 2L/(L + 1) < 2, so low normalized depth can arise structurally without implying that a model uses only a few layers.This is an idealized maximally diverse update reference, not a null distribution or target.Source: §1 · p. 2
  • Effective depth aggregates a CKA-based similarity autocorrelation across layer distances into a scalar diagnostic of redundancy in the accumulated residual stream.The paper measures accumulated hidden states because these are the information states downstream computation receives.Source: §1 · pp. 2–3
  • Across sixteen decoder-only models, nearly every default measurement falls below the orthogonal-update reference, and matched references attribute the gap largely to measured update correlations rather than initial-state carry or update-size imbalance.The regime persists under position-0, token-normalisation, and top-PC controls; the above-reference outlier joins it under each control.Source: §1 · pp. 2–3
  • OLMo-2 establishes a 40–41% gap by its first post-initialisation checkpoints and remains near it through 5T tokens, while Pythia-1.4B decreases from 44% at 2B tokens to 26% at 300B tokens.The trajectories indicate a family-conditioned regime rather than one determined solely by training progress.Source: §1 · p. 3
  • A residual-carry intervention found that reductions mild enough to preserve optimisation left Deff(h)/L unchanged, while larger reductions that raised it substantially broke optimisation.The authors therefore frame Deff as a global accumulated-state diagnostic, not a capability score or pruning method.Source: §1 · p. 3

2. Related Work

Prior work identifies layer redundancy through pruning, routing, and measures of residual structure, while representation studies characterize similarity across depth. This paper positions Deff as a full-lag scalar complement to local or ablation-based views.

Layer redundancy and pruning

  • Men et al. (2024) use Block Influence, the cosine similarity between a block’s input and output, while Gromov et al. (2024) remove up to 50% of layers with small perplexity loss.Other cited approaches leverage redundancy through early exit, structured removal, conditional routing, or stochastic dropout.Source: §2 · p. 3
  • Deff formalises redundancy by aggregating the full lag spectrum, with Block Influence corresponding to its lag-1 term.The cited pruning and routing works identify redundancy through ablation or layer-level decisions.Source: §2 · p. 3

Depth utilization and residual structure

  • Prior residual-structure work reports near-identity deep blocks, highly similar adjacent representations, and smaller update norms in later layers, while other studies describe distinct inference stages across depth.The paper presents its orthogonal-update baseline as complementary to these accounts and Dupdate as a global extension of the per-layer view.Source: §2 · p. 4

Mechanistic perspective and massive activations

  • Mechanistic studies describe residual streams as communication channels and feed-forward layers as key-value memories, while position-0 activation work studies within-layer and within-sequence effects.Deff instead measures accumulated-stream redundancy across depth and samples; the cited position-0 controls preserve the sub-reference regime.Source: §2 · p. 4

Representation similarity and intrinsic dimension

  • CKA is used for its orthogonal invariance, and the Bartlett aggregation is presented as a scalar summary of layer autocorrelation rather than an estimator of a specific mixing coefficient.The discussion connects these choices to representation-similarity and intrinsic-dimension literature.Source: §2 · p. 4

3. Background

The background introduces CKA and finite-lag weighting as ingredients for summarizing layer similarity. It explicitly limits the analogy to scalar aggregation rather than treating trained models as stationary chains.

3. Background

  • CKA provides a representation-similarity measure, and the paper motivates weighted full-lag aggregation through classical long-run-variance and sampling-efficiency estimators.The cited sources are Kornblith et al. (2019), Newey and West (1987), and Geyer (1992).Source: §3 · pp. 4–5
  • The paper borrows finite-lag weighting only as a scalar summary of layer-similarity profiles; trained models are not stationary chains, so Deff is not a beta-mixing estimator or strict sampling-efficiency estimate.Connections to mixing theory are deferred pending additional norm-process assumptions.Source: §3 · p. 5

4. Method

The method treats accumulated hidden states as a layer-indexed process, measures similarity across layers, and aggregates the resulting autocorrelation profile into effective depth. It uses a structural reference for accumulated states and a Bartlett-weighted sum over all lags.

4. Method

  • Deff is computed from accumulated hidden states rather than layer updates because the target is the distinct information states the network passes through and pruning or skipping acts on the accumulated stream.The embedding layer is excluded from the layer-wise process.Source: §4 · p. 5
  • With zero update cross-covariance, maximally diverse updates yield Deff = 2L/(L + 1) < 2, the structural reference used to measure additional accumulated-state redundancy.Matched scalar references retain measured update-size profiles, h0 carry, and update-similarity profiles.Source: §4 · pp. 5–6
  • The layer similarity autocorrelation is the analogue of scalar autocorrelation along depth, and the method does not claim it estimates a specific mixing coefficient.The estimator is based on CKA; the cited validation reports nearly identical alternatives using unbiased HSIC-CKA and cosine.Source: §4 · p. 5
  • A Bartlett taper down-weights high-lag terms and sums all L−1 lags without bandwidth selection, producing a full-lag scalar summary rather than a strict sampling-efficiency estimate.Alternative participation-ratio and effective-rank summaries yield qualitatively identical conclusions.Source: §4 · p. 6
  • Under i.i.d. inputs, finite fourth moments, and nonzero population covariance matrices, the CKA-based autocorrelation estimator is consistent as the number of inputs grows.The proposition states consistency for fixed layers and feature dimension.Source: §4 · p. 6

5. Experiments

Across sixteen decoder-only language models, most default measurements fall below the orthogonal-update reference, and matched references attribute much of the excess gap to correlated updates. Position and norm controls, checkpoint trajectories, and residual-carry probes further qualify how to interpret this global diagnostic.

Models and setup

  • The evaluation covers sixteen publicly available decoder-only language models using 10,000 FineWeb-Edu passages per model; encoder-only and vision transformers are out of scope.The reported reference is FL = 2L/(L+1), the closed-form Deff for accumulated orthogonal updates.Source: §5 · p. 6

A sub-reference operating regime

  • 15 of 16 models fall below the orthogonal-update reference, with gaps of 32–44% for Qwen3.5, 40–41% for OLMo-2, and 23–28% for Pythia.Gemma-3-12b is the lone above-reference default outlier at −17.6%.Source: §5 · p. 7, Figure 1 · p. 7

Matched references

  • Matching update sizes and the persistent initial state leaves every model sub-reference, while matching update correlations reverses the gap sign for every model.The gap is therefore not mainly an h0 or update-norm artifact, but is largely explained by correlated updates.Source: §5 · p. 8

Position and norm controls

  • All models in the six-model control panel are sub-reference under the tested controls, and all sixteen are sub-reference after token-normalisation or top-1-PC removal.Position-0 norm spikes can strongly shift measured gaps, but do not explain the regime away.Source: §5 · p. 8, Figure 2 · p. 9

Training dynamics

  • OLMo-2 establishes a 40–41% gap early and stays near it through 5T tokens, while Pythia-1.4B decreases from about 44% at 2B tokens to 26% at 300B.The trajectories indicate a family-conditioned pattern rather than one determined solely by training progress.Source: §5 · pp. 9–10

Capability and local decisions

  • Within Qwen3.5, lower Deff/L co-moves with higher MMLU and ARC-Challenge scores, but Deff is not treated as a capability score.Deff summarizes global accumulated-state similarity; local layer removal should instead use local influence or ablation scores.Source: §5 · pp. 10–11

Residual-carry mechanism

  • In a controlled 12-layer nanoGPT γ-sweep, only residual-carry reductions that substantially raise Deff(h)/L also break optimisation.This supports residual carry as a causal contributor in that controlled setting.Source: §5 · p. 10

Random-weight baseline

  • Trained and random-weight Deff/L remain within a maximum absolute difference of 0.018 across completed model pairs, indicating a strong residual-geometric component.The random-weight values are seed-stable in the evaluated checks, while trained-versus-random direction varies by architecture.Source: §5 · p. 9, Table 1 · p. 11

6. Discussion

The discussion frames Deff as a global measure of accumulated-state redundancy, not a count of unused layers. Its interpretation depends on calibration against residual accumulation and remains bounded by the diagnostic’s scope.

What Deff measures

  • The empirical signal is the additional gap below FL, since orthogonal updates already produce Deff = 2L/(L + 1) < 2.Matched references attribute the gap to measured update similarity rather than a hidden count of unused layers.Source: §6 · pp. 11–12

What the regime tells us

  • Position-0, token-normalisation, and top-PC controls, together with training trajectories, indicate a family-conditioned regime rather than a pooling artifact or solely a training-progress effect.The diagnostic helps distinguish family-level geometry from position-0 or norm-driven distortions.Source: §6 · p. 12

Mechanism

  • Residual carry is a supported contributor: the update-level diagnostic remains in the same low band, and only large carry reductions raise Deff while breaking optimisation in the controlled probe.Pre-norm, norm-growth, and Procrustes effects may also compound with residual accumulation.Source: §6 · p. 12

Limitations and scope

  • Deff is a corpus-based diagnostic restricted empirically to decoder-only LMs on FineWeb-Edu; CKA misses pure residual-stream rotations, and trained models are not stationary chains.It is not a literal effective-sample-size or beta-mixing estimator, nor a local pruning score.Source: §6 · p. 12

7. Conclusion

The conclusion presents Deff as a calibrated global diagnostic of accumulated residual-stream geometry. It separates structural effects of residual accumulation from empirical gaps and distinguishes global summaries from local layer decisions.

Conclusion

  • Across sixteen decoder-only LMs, the calibrated gap is mostly sub-reference and matched references tie it to correlated updates, while the tested controls preserve the regime.The orthogonal-update baseline is FL = 2L/(L+1) < 2.Source: §7 · p. 12

Interpretation

  • Deff is a model-level diagnostic of residual-stream geometry, not evidence that only a few layers matter.Its use is to map a family relative to the accumulation reference and identify distortions revealed by controls.Source: §7 · pp. 12–13

Practical use

  • Deff provides a global coordinate, while BI-like scores are the local companion for layer-level decisions.The paper positions the distinction as a way to separate state redundancy from local pruning decisions.Source: §7 · p. 13

Appendix A. Full Effective-Depth Measurements

Table 2 reports the numeric values underlying Figure 1.

Appendix A. Full Effective-Depth Measurements

Appendix Roadmap

The supplement maps evidence supporting the paper’s main claims, from Deff’s mathematical and synthetic foundations through calibration, robustness, and boundary checks.

Appendices S1–S2

  • Appendices S1–S2 establish Deff’s basic properties, the orthogonal-update reference, and controlled constructions separating accumulated states from layer updates.Figure S1 contrasts the constructions with real models.Source: Appendix Roadmap · p. 16

Appendices S3–S5

  • Appendix S3 calibrates the gap to the universal reference against update size, the persistent initial state, and measured update correlations.Appendices S4–S5 expand the controls and prove CKA convergence.Source: Appendix Roadmap · p. 16

Appendices S6–S14

  • Appendices S6–S14 test aggregation robustness, document extraction and CKA details, and collect auxiliary diagnostics and representation-similarity checks.These include lag truncation, alternative aggregations, and statistical robustness checks.Source: Appendix Roadmap · p. 16

Appendices S15–S20

  • Appendices S15–S20 support Deff as a global residual-stream diagnostic and report pruning boundaries, bootstrap intervals, and an external capability-scaling check.The evidence includes a controlled residual-carry intervention.Source: Appendix Roadmap · p. 16

Appendix S1. Properties of Deff

Deff is built from CKA-based layer similarity and has interpretable limits: it reaches L for independent layers and 1 for perfectly correlated layers.

Definition and empirical scope

Limiting cases

Algebraic structure

  • For ˆϱ(k) in [0, 1], Deff = L/S, with S having the algebraic form of a Bartlett spectral density estimator at frequency zero.The paper does not claim that ˆϱ has the positive-semidefinite structure of a true autocorrelation sequence.Source: Appendix S1. Properties of Deff · p. 17

Synthetic validation

  • The synthetic-construction comparison uses independent and orthogonal states to test whether Deff rises when states are decorrelated.The table also includes orthogonal-update sweeps, where accumulated-state Deff can remain low despite diverse updates.Source: Table S1 · p. 18

Appendix S2. Controlled Synthetic Constructions and Empirical Validation

Controlled constructions show that Deff distinguishes state similarity from update diversity: it rises for decorrelated states, while accumulated states can remain similar even with orthogonal updates.

Independent and orthogonal states

  • For i.i.d. Gaussian states, Deff/L = 0.258 and ˆϱ(1) = 0.093; explicitly orthogonal states give Deff/L = 0.996 and ˆϱ(1) = 0.0001.The i.i.d. result is below the ideal independence limit because finite-sample linear CKA remains positive when d/N is non-negligible.Source: Appendix S2. Controlled Synthetic… · p. 18

Orthogonal-update construction

  • Mutually orthogonal per-layer updates are accumulated as hℓ = sum of f_j for j ≤ ℓ, so diverse updates can still yield similar states.For common-covariance updates with zero cross-covariance, the population construction gives Deff = 2L/(L + 1) < 2.Source: Appendix S2. Controlled Synthetic… · p. 19

Interpretation

  • The real-model results combine similarity from residual accumulation with correlated updates; low Deff(h) alone does not imply identical layer updates.The construction confirms that even perfectly orthogonal updates can produce low accumulated-state depth.Source: Appendix S2. Controlled Synthetic… · p. 19

Orthogonal-update depth sweep

Appendix S3. Matched Reference Baselines

Matched references show that update size and the persistent initial state do not explain the sub-reference gap, while measured update correlations largely account for it.

Update-size and h0 controls

  • All sixteen models remain below Fnorm and Fh0+norm when references retain measured update sizes, with or without the persistent initial state.The universal FL reference assumes equal-norm, mutually orthogonal updates and no persistent initial component.Source: Appendix S3. Matched Reference Baselines · p. 20

Update-correlation control

  • For every model, Fh0+corr predicts lower Deff than observed after retaining measured update sizes, h0, and update-similarity lags.The scalar reference preserves the measured update-similarity profile.Source: Appendix S3. Matched Reference Baselines · p. 20

Interpretation

  • The gap below FL is largely explained by correlated updates, making FL a first structural yardstick rather than a definitive null.Observed accumulated states retain more diversity than the scalar reference preserving update sizes and average similarities.Source: Appendix S3. Matched Reference Baselines · p. 21

Appendix S4. Full Position-0 and Norm Controls

The appendices test the robustness of the measured regime across position, token-normalisation, top-PC, metric, aggregation, and sampling checks, and examine norm interventions and residual-carry causality. They also delimit Deff’s pruning and capability scope and cross-check CKA with alignment-free GW while retaining CKA for its closed-form reference.

Appendix S4. Full Position-0 and Norm Controls

  • All sixteen models are sub-reference after token-normalisation and top-1-PC removal; first-non-BOS pooling is sub-reference for fifteen, with Pythia-70M the exception.Gemma-3-12B, the canonical above-reference default, becomes strongly sub-reference under every control; Gemma-3-27B is also above-reference by default, while Gemma-3-4B is near the reference, and both additional sizes become sub-reference under the controls.Source: Appendix S4. Full Position-0 and Norm… · pp. 21, 23, Table S4 · p. 23, Table S5 · p. 23
  • In trained Gemma-3-4B, 12B, and 27B checkpoints, removing post-branch normalisation drives all sizes into a highly sub-reference regime, while removing QK normalisation amplifies the default-pooling anomaly.At initialisation and after short 270M-scale FineWeb-Edu pretraining, the above-reference default anomaly is absent; these limited probes do not reproduce Gemma’s full training history, so the anomaly is not established as a pure architectural or initialisation-level effect.Source: Appendix S4. Full Position-0 and Norm… · pp. 22–23, Table S6 · p. 24, Table S7 · p. 24

Appendix S6. Maximum Lag Sensitivity; Appendix S7. Alternative Aggregation Functions

Appendix S10. Additional Experimental Details

  • Random-weight Deff/L has standard deviation at most 2 × 10^-4 across five seeds, and trained values lie outside the random-seed range and mean ±2σ band in all seven tested cases.Trained values move below random-weight references for Qwen3.5 and OLMo-2-1B, but above them for larger OLMo-2 and Llama-3.1-8B.Source: Table S10 · p. 29, Appendix S10. Additional Experimental… · p. 29

Appendix S11. Geodesic Efficiency vs. Deff/L

  • Deff/L and geodesic efficiency correlate positively but moderately (r = 0.65), capturing related but distinct aspects of layer utilisation.A model can have a relatively straight residual-stream path while still producing highly similar intermediate representations.Source: Appendix S11. Geodesic Efficiency vs.… · p. 29, Figure S6 · p. 31

Appendix S9. Alternative Tapers for Deff; Appendix S13. Multi-Metric Robustness; Appendix S14. Instruction-Tuned vs. Base Models

Appendix S15. ShortGPT Analysis: Shallow Models

  • For Pythia-70M (L = 6), Deff’s Bartlett aggregation captures decorrelation across all five lags, whereas a lag-1-only diagnostic misses the profile’s decay from ϱ(1) = 0.803 to ϱ(5) < 0.4.The taper is nearly flat for shallow models but suppresses high-lag terms in deep models, where lag-1 similarity dominates.Source: Appendix S15. ShortGPT Analysis:… · p. 32

Appendix S16. Per-Layer Update Diagnostic

  • The update-based Dupdate/L is only modestly higher than accumulated Deff/L and remains below 0.16 for all completed models, consistent with both residual accumulation and correlated updates contributing to low effective depth.The mean update-to-accumulated norm ratio falls from 0.54 at L = 12 to 0.29–0.37 at L = 32–40 and 0.18 at L = 64.Source: Appendix S16. Per-Layer Update… · pp. 32–33

Appendix S17. Controlled Residual-Carry Intervention

  • At γ=0.5 and 0.25, Deff(h)/L rises to 0.124 and 0.268 versus 0.094 at γ=1.0 in every seed, while validation loss plateaus near 3.35; γ=0.75 leaves it unchanged.The intervention supports residual carry as a causal contributor to accumulated-state effective depth in this char-level nanoGPT setup, but the larger reductions in γ that raise Deff(h)/L break optimisation.Source: Appendix S17. Controlled… · pp. 33–34, Figure S7 · p. 35

Appendix S18. Bootstrap Stability of Deff/L

Appendix S19. Pruning Boundary Checks

  • BI is substantially better than local leave-one-out ΔDeff at ranking safe layer removals on OLMo-2-1B and Llama-3.1-8B, consistent with Deff being global rather than per-layer.The exploratory eight-model tolerance check is suggestive, not conclusive: the primary depth-normalized fraction passes the threshold at n=8 but fails the leave-one-out check, while raw tolerance is robust; absolute Deff and gap-to-F_L are not useful predictors.Source: Appendix S19. Pruning Boundary Checks · pp. 35–36, Table S16 · p. 36, Table S18 · p. 37

Appendix S20. Capability-Relevant Scaling Check

  • Within Qwen3.5, lower Deff/L co-moves with higher MMLU and ARC-Challenge scores (r = −0.896 and r = −0.913), but does not establish predictive information beyond scale.Cross-family modern-only correlations are mixed, HellaSwag is partly driven by the Pythia-versus-modern era gap, and GSM8K-CoT scores are unreliable in this run; capability prediction is not used as evidence for Deff.Source: Appendix S20. Capability-Relevant… · pp. 37–38, Table S19 · p. 38, Table S20 · p. 39

Appendix S21. Alignment-free similarity check: a Gromov-Wasserstein pilot

  • The alignment-free GW view is qualitatively concordant with CKA: high cross-layer CKA similarity corresponds to small GW metric distortion, and GW distance grows with layer distance.The paper retains linear CKA because its orthogonal-update reference F_L = 2L/(L + 1) is closed-form; no analogous calibrated GW reference is available.Source: Appendix S21. Alignment-free… · pp. 38–40, Table S22 · p. 40
Loading 2609.31098v1…