Source-linked AI summary
Function Lives Where Variance Doesn't: Task-Weighted Charts of a Language Model's Computation
Alexandre Quemy
TL;DR
Activation geometry does not determine how many dimensions a model's downstream computation uses, so the paper measures dimension relative to a chosen function. Task-weighted charts use that function's metric; across six models, next-token prediction needs most of the residual-stream width, although variance gives little indication of where the function lies.
Problem
Representation geometry describes where activations lie, but does not establish how many coordinates downstream computation consumes.
Source: §1 · p. 1Method
Results
Across six models, next-token prediction needs 70–90% of residual-stream width to stay within 5% of intact perplexity, while variance predicts none of the requirement.
Source: §1 · p. 1Takeaways & Limitations
At tight budgets, charts trained under the functional metric preserve predictions better than variance-based or optimal linear compression.
Source: §8 · p. 8The requirement-dimension readings come from the best chart families constructed, which cannot certify the minimum and may overestimate it.
Source: §3 · p. 3
Abstract
from arXiv · showhide
How many dimensions does a language model's computation actually use? The question is ill-posed until one names a functional. Task-weighted charts make it well-posed: low-dimensional coordinate systems fit against a chosen functional of the representation, under the functional's own metric, turning distillation into plain least squares. Across six models from three families, spanning 70m to 7B parameters, next-token prediction needs 70--90% of the residual stream's width to stay within 5% of intact perplexity, a width consumed by the rare tail of language, and the variance profile predicts none of it: two directions carry 90% of GPT-2's activation variance and almost none of its function. Dimension is per-functional: the model's own uncertainty reads from six coordinates where the full predictive distribution needs hundreds; and it grows with depth. The dissociation is exploitable: when only a few dimensions can be kept, charts trained under the functional's metric preserve the model's predictions better than variance-based or optimal linear compression.
1 Introduction
The paper distinguishes activation geometry from the dimensions the downstream computation actually uses. It measures this gap across models and introduces task-weighted charts to count dimensions for a chosen function.
1 Introduction
- Activation geometry describes where representations lie, but does not establish how many coordinates downstream computation consumes.The dimensions relevant to interpretability and compression may therefore differ from those of the activation cloud.Source: §1 · p. 1
- Across six models, next-token prediction needs 70–90% of residual-stream width to stay within 5% of intact perplexity, with no privileged low-dimensional core.The required dimension declines smoothly, while the variance profile predicts none of it.Source: §1 · p. 1
- Task-weighted charts compress hidden states through a k-dimensional bottleneck trained against a chosen functional under its Gauss–Newton (Fisher) metric.The metric turns distillation into least squares; the charts use factorized dictionaries fit by two-stage variable projection.Source: §1 · p. 1
2. A cross-family anatomy of the residual
Required dimension depends on the functional, token, and depth, whereas variance collapse varies by architecture. The paper also reports that functional metrics improve compression and that several alternative approaches fail.
2. A cross-family anatomy of the residual
- Required dimension is about 5 to 400 on the same activations, varies by token, and grows toward the head; across six models, next-token prediction needs 70–90% of width.Frequent tokens survive tight budgets better than rare tokens; variance collapse is an architectural accident, not a predictor of function.Source: §2 · pp. 1–2
- At tight budgets, Fisher charts beat optimal linear charts in all six models; pulled-back layer metrics extend the advantage across depth, and fitted charts transfer across corpora without retraining.Source: §2 · p. 2
- Four alternatives fail for measured reasons: variance subspaces miss the function, trajectories resist useful extrapolation, re-ranking cannot improve logits, and narrow-bottleneck block replacement fails.Source: §2 · p. 2
2 Related work
Prior work characterizes representation geometry, intrinsic dimension, and compression, but these measure different properties or address complementary goals. This paper focuses on the dimensions a fixed downstream consumer needs.
2 Related work
- Contextual representations have been described as narrow cones, dominated by rogue dimensions, or locally isotropic within clusters; this paper finds dominant directions carry little next-token function.The function instead lies in the variance tail.Source: §2 · p. 2
- Transformer intrinsic-dimension estimates report low local dimension with a depth-dependent hump, while this paper's required dimensions answer a different question.Both estimates can be correct; low fine-tuning-objective dimension also coexists with high required dimension for the frozen model's predictive state.Source: §2 · p. 2
- Prior metric-weighted compression targets weights; this work applies the idea to states by whitening reconstruction targets with the head's Fisher metric.The paper reports that metric choice, rather than chart architecture, determines how much function survives compression.Source: §2 · p. 2
- Sparse decompositions seek many interpretable features, whereas this paper counts dense coordinates for a fixed consumer; downstream-KL SAE training is a related metric-based approach.Unlike learned lens translators, patch-and-continue runs the model's actual remaining computation; the tail metric uses the mean tail Jacobian as a metric rather than a readout.Source: §2 · p. 2
3 The instrument
A chart represents hidden states in a lower-dimensional coordinate system, but the training metric determines which reconstruction errors matter. The paper uses consumer-specific metrics and defines required dimension by near-preservation of task loss.
3 The instrument
- A chart encodes a hidden state into k coordinates and decodes it into an approximation; its loss metric determines which state-reconstruction errors it avoids.The consumer is the unembedding head at the final layer and the remaining blocks at an intermediate layer.Source: §3 · p. 2
- The identity metric makes capacity follow data variance, while the head's logit metric weights logit directions equally and frees its kernel directions.Source: §3 · p. 2
- The Fisher pullback approximates the KL between the model's predictive distributions under a state perturbation, averaged over empirical states.Its softmax curvature suppresses logit shifts and nearly suppresses directions that only move improbable logits.Source: §3 · p. 3
- Two-stage dictionary charts learn coordinates and solve dictionary weights by variable projection; nested prefixes trace loss against dimension, while greedy knockout tests surviving coordinates.Whitening the target by G^1/2 makes mean-squared error equal second-order KL, preserving the closed-form solve.Source: §3 · p. 3
- Requirement dimension is the smallest k at which a chart keeps task loss within tolerance of the intact model, but the chart family can only provide an upper reading.Stronger chart families improve tight-budget perplexity, yet the near-intact requirement point is unchanged across the three tested families; evaluation uses held-out WikiText-103.Source: §3 · p. 3
4 Validating the instrument
On a frozen random MLP with known latent structure, the charts recover the latent dimensions closely; when the task uses only some latents, the measured requirement can exceed the true task dimension by one.
4 Validating the instrument
- At d=2 and D=64, charts recover the latents with R2 = 0.999, and both readouts return exactly 2.The latents are retained as ground truth.Source: §4 · p. 3
- At d=4 and D=512, recovery reaches R2 = 0.996 and the knockout returns exactly 4.D=512 is identified as language-model width.Source: §4 · p. 3
- When a task uses only 2 of 4 latents, the readout returns 3, an upper bound one dimension above the task requirement.The TwoNN intrinsic-dimension estimator agrees throughout the validation experiments.Source: §4 · p. 3
5 Function lives where variance doesn’t
Across six models, next-token prediction requires most of the residual width even though variance profiles differ sharply. Required dimensions also vary by token rarity, and steering comparisons show that Fisher directions move outputs more than variance directions.
5 Function lives where variance doesn’t
- The comparison uses residual activations from 100k WikiText-103 validation tokens, fits charts and covariances on 20k sampled positions, and evaluates perplexity on held-out chunks disjoint from fitting positions.For GPT-2 small, the measured blocks are 0, 6, 10, and 11, with block 11 post-norm.Source: §5 · p. 3
- At GPT-2’s final layer, participation ratio is 1.4 at small and 1.17 at medium with rank90 = 1, yet at k=32 perplexity rises from 29.3 to 269 under activation PCA and to 362±29 with a pure-reconstruction VP chart.The variance-faithful chart does no better than PCA, while the function occupies hundreds of tail directions.Source: §5 · p. 4
- Pythia and Qwen have wider final-layer variance (PR 113, 23, 31), while Qwen’s middle layer collapses (PR = 1.09, rank90 = 1); Pythia shows no such collapse.Despite the architectural differences, the required dimensions remain high; Qwen’s middle-layer pattern also appears in Qwen3 and Qwen2.5-7B.Source: §5 · p. 4
- 34% of tokens survive at k=8, while 39% need more than 128; the median frequent-quartile token needs k=8 and the rare-quartile token more than 128.Requirement correlates with rarity (Spearman 0.49), and the split persists when the metric upweights rare tokens.Source: §5 · p. 4
- Across all six models, 70–90% of width is needed to come within 5% of intact perplexity, despite differing final-layer variance profiles.The table compares final-layer variance concentration with the fraction of width required for near-intact perplexity.Source: Table 1 · p. 4, §5 · p. 4
- At equal perturbation norm, top-8 Fisher directions move the output by 6.0 nats on average, versus 0.56 for random directions and 0.12 for top-8 variance directions.The same Fisher direction shifts logits consistently across contexts (mean pairwise Δ-logit cosine 0.84).Source: §5 · p. 4
6 The anatomy of GPT-2 small
The dimensions needed to preserve GPT-2 small vary by functional and depth, and the directions worth keeping depend on each layer’s consumer. At tight budgets, training under the functional’s metric improves preservation, while replacing a block through a narrow state bottleneck can fail despite fitting the map.
6.1 Dimension is a property of the functional
- Entropy needs only a handful of coordinates, while the full predictive distribution needs hundreds and approaches intact only near half-width; TwoNN stays in the low teens.Figure 1 shows that removing the two directions carrying 90% of activation variance costs nothing, while Fisher charts at k=32 preserve more content than activation PCA.Source: §6.1 · pp. 4–5, Figure 1 · p. 5
6.2 Required dimensions grow with depth, and each layer has its own metric
- Within 15% of intact performance, the required budget grows from ∼256 dimensions at layer 0 to ∼384 from layer 3 onward; no layer is cheap.The measurement projects each layer’s state to k dimensions, reinserts it, and runs the remaining blocks unchanged.Source: §6.2 · p. 5
- Figure 2 shows frequent tokens surviving at k=8 and rare tokens needing more than 128; at matched norm, top-variance directions move the output less than random directions.The figure’s left panel is the per-token requirement distribution; its right panel compares causal steering directions.Source: Figure 2 · p. 6
- At k=32, charts trained under each layer’s averaged consumer metric beat every linear baseline at all eleven depths (178 vs. 646 at layer 0).The averaged metric fails as a direction ranker because position-specific tail Jacobians blur distinctions; as a training target, it shapes what charts retain.Source: §6.2 · p. 6
6.3 The metric decides what a tight budget captures
- A plain autoencoder on the same Fisher-whitened target reads 78, 2.4× better than the dictionary; Matryoshka nesting costs 15–20% but still beats it.The autoencoder lowers the curve up to k ≈384, while near-intact points align with the linear family’s anchors.Source: §6.3 · pp. 6–7
- 362 for pure reconstruction, 512 under uniformly whitened logits, and 184 under the Fisher metric: with dictionary and solver fixed, the metric alone changes chart quality.Blending 10% of the logit metric into Fisher yields 221 vs. 184; merely selecting a linear subspace under Fisher reads 226 vs. 236 under the logit metric.Source: §6.3 · p. 6, Table 3 · p. 7
- On wikitext-2, fitted charts reproduce in-corpus quality (175 vs. 184±17 at k=32), while linear subspaces degrade from 236 to 316; the ordering persists on TinyStories.Refitting and evaluating on TinyStories reproduces both the near-intact point (k ≈384–512) and variance collapse (rank90 = 2).Source: §6.3 · p. 7
6.4 Replacement fails on state width, not map fidelity
- A k=32 block replacement reads worse than deleting the block, while widening the same map to k=384 recovers a factor of three; deletion preserves the full 768-wide state.The pinch effect explains the contrast: a narrow replacement bottlenecks the state, whereas deletion omits only one block’s update; the paper’s compression lesson is to compress computation between states.Source: §6.4 · p. 7
7 Across families, generations, and scale
Across model families and scales, next-token prediction requires a large share of the residual width, while variance structure does not predict functional rank. Tight-budget Fisher charts outperform optimal linear charts, and the required dimensions also vary with depth and token function.
7 Across families, generations, and scale
- At k=32, Fisher charts beat optimal linear charts in all six models, with the largest advantage reported for Qwen2.5 at 42%.Reported values include GPT-2 small 184±17 vs. 236, Pythia 210±2 vs. 285, and Qwen2.5 110±3 vs. 189.Source: §7 · pp. 7–8
- Across the replicated block-replacement rows, widening the same chart machinery recovers most of the gap, while whether a fitted map beats deletion depends on block replaceability.The table compares full-rank affine maps, deletion, k=32 Fisher chart-maps, and wide chart-maps.Source: §7 · p. 8, Table 4 · p. 8
- At k=32, patch-and-continue fails at every measured depth in every model; Qwen’s collapsed middle layer needs k ≈256 for near-intact reading (14.3 vs. 13.2).The depth trend rises in GPT-2 and Qwen, but is flat-to-falling in Pythia.Source: §7 · p. 8
- 70–90% of residual width is required to stay within 5% of intact perplexity across six models, with no elbow in the requirement curves.Figure 3 shows the shared curve shape; Pythia’s logit-metric sweep declines smoothly from 285 dimensions at k=32 to 51 at k=448, against intact perplexity 50.4.Source: §7 · pp. 7–8, Figure 3 · p. 8
- Variance collapse is architecture-dependent, while TwoNN stays 10.8–15.2 as required dimensions grow from 448 to 2560.Pythia never collapses, whereas GPT-2 collapses at final layers and Qwen at its middle layer.Source: §7 · p. 8
8 Conclusion
The paper argues that computation’s dimension is meaningful only relative to a named functional, and that activation variance alone cannot establish what a representation does. Its findings caution against treating variance-selected subspaces as meaningful and identify the consumer’s metric as a practical lever for compression.
8 Conclusion
- A subspace selected for variance cannot be assumed meaningful; encoding claims should be checked with the consumer in the loop.The paper reports that top variance directions are causally inert where variance concentrates.Source: §8 · p. 8
- The manifold intuition captures recoverable structure, but manifold dimension is an inert decision signal because decoding-time uses tested lose to logits.The logits already price the model’s uncertainty for the tested uses.Source: §8 · p. 8
- At tight budgets, what survives compression depends on the metric used to train charts, rather than their architecture.The paper proposes preconditioning steering vectors by G and, at middle layers, by the tail metric; top eigendirections of G could serve as control handles.Source: §8 · p. 8
Limitations
The study’s trained methods at 7B are limited to final-layer Fisher charts, and its requirement dimension depends on the accepted tolerance. The tolerance analysis also finds that random same-size projections do not preserve model behavior and that average quadratic prediction misses rare-token costs.
Limitations
- At 7B, trained arms cover only final-layer Fisher charts; deeper trained methods stop at 0.6B, leaving the frontier regime unmeasured.Evaluation is also limited to one corpus family, with transfer tests only between WikiText-103, WikiText-2, and TinyStories.Source: Limitations · p. 10
A Tolerance sensitivity of the requirement dimension
- At τ = 5%, models require 70–90% of width; at τ = 15%, the range is 50–75%, so the quoted dimension depends on tolerance.The curves decline smoothly without an elbow; Table 5 gives grid readings at three tolerances.Source: A Tolerance sensitivity of the… · p. 10, Table 5 · p. 10
- Random same-size projections destroy model performance: perplexity reaches 2.9 × 10^4 at k=32 and 6.8 × 10^5 at k=512.The reported values are four to seven orders of magnitude above intact, showing that the learned subspaces—not dimension alone—matter.Source: A Tolerance sensitivity of the… · p. 10
- The matched quadratic predictor places GPT-2 within 15% of intact by k ≈128, while measurement needs 384–512 because rare-token costs are missed by the average.At 7B, the comparison is within 8% at k=512 against a measured 2048.Source: A Tolerance sensitivity of the… · p. 10
B Two controls: metric reweighting and massive activations
Two controls show that rare-token requirements persist under inverse-frequency metric weighting and that variance deletions preserve the mean state. The chart-fitting procedure standardizes inputs, optionally whitens targets, and fits nested charts through variable projection using WikiText-103 data.
B Two controls: metric reweighting and massive activations
- Upweighting rare tokens leaves the rare-quartile median requirement at 256 and changes the rarity correlation only from 0.49 to 0.43.At k=8, 31% of tokens survive under inverse-frequency weighting, compared with 34% under Fisher weighting.Source: B Two controls: metric reweighting and… · p. 11
- Variance-direction deletions are harmless because projections preserve the mean state and remove only variation, whereas zeroing nearly constant coordinates removes the fixed bias and collapses the model.The top variance directions correlate with neither position nor token frequency (both |ρ| < 0.08).Source: B Two controls: metric reweighting and… · p. 11
C Chart fitting in detail
- Chart fitting standardizes inputs, optionally whitens targets by the task metric, and alternates a closed-form ridge solve with encoder optimization through the solution.Each k-coordinate prefix is trained as a chart, with an orthogonality penalty and early stopping on a 20% validation split.Source: C Chart fitting in detail · p. 11
- Hyperparameters are held constant across arms, models, and layers except for k and target whitening; the Fisher metric uses 4,096 fitting states.A GPT-2 small k=32 Fisher chart takes about 9 minutes per seed, while one 7B Fisher-chart seed takes about an hour.Source: C Chart fitting in detail · p. 11
- Fitting uses 20k sampled positions from 100k WikiText-103 validation tokens, with perplexity evaluated on disjoint contiguous chunks and block replacement on the test split.The Fisher subset comes from the fitting split, so training-time quantities do not use evaluation data.Source: C Chart fitting in detail · p. 11
D Negative results, with mechanisms
Variance-based intuitions fail as decision rules: next-token function lives in variance-tail directions, and dimensional requirements vary with depth. At tight budgets, task-weighted charts outperform linear charts across measured layers.
D Negative results, with mechanisms
- Chart projection adds 2.4 nats of excess loss; trajectory extrapolation loses to reusing the current state, and geodesic re-ranking lowers top-1 accuracy as β grows.Projection discards tail dimensions the head needs; state sequences are driven by incoming tokens, while re-ranking double-counts geometry already priced by the logits.Source: D Negative results, with mechanisms · pp. 11–12
F Per-depth results
- At nearly every depth, the final-head logit metric is the worst basis, while tail-metric subspaces never decisively outperform activation-PCA.At fixed k, results broadly improve with depth, but closing the last 15% to intact still requires more dimensions; early states are consumed at low dimension, unlike late states.Source: F Per-depth results · p. 12, Figure 7 · p. 13
E The dissociation at 7B, on text
- Removing Qwen2.5-7B’s top variance direction, carrying 17% of activation variance, costs 0.06 nats; keeping k=128 of 3584 dimensions destroys content words.The figure shows the dissociation directly on text at 7B; 14× scale softens neither effect.Source: E The dissociation at 7B, on text · p. 12, Figure 6 · p. 13
F Per-depth results
- At k=32, trained charts win at all eleven depths; at k=64, they win at eight of eleven, with early-layer exceptions where activation-PCA is strong.The exceptions are layers 1, 2, and 5.Source: F Per-depth results · p. 12, Table 6 · p. 14