PaperlayerView exact revision on arXiv

Source-linked AI summary

Language Model Activations Inhabit Privileged Error-Correcting Basins

Matthew Finlayson, Francisco Pernice, Eric Todd, Amir Zur, Daniel Wurgaft, Fenil R. Doshi, Vasudev Shyam, Matt Feiszli, Satchel Grant, Lucius Bushnaq, Tal Haklay, Usha Bhalla, Matthew Kowal, Thomas Fel, Jack Merullo, Atticus Geiger, Xiang Ren, Owen Lewis, Ekdeep Singh Lubana

arXiv:2610.04183v1cs.AIcs.CLcs.LG

TL;DR

The paper asks whether language models’ robustness to activation perturbations reflects forward-pass dynamics that funnel activations toward coherent regions. It probes activation-space geometry, identifies attracting basins, and applies that geometry to adaptive steering, reporting improved English-to-Chinese steering over fixed-strength steering.

  • Problem

    The paper asks whether constraining dynamics in language-model forward passes funnel perturbed activations toward coherent regions.

    Source: §1 · p. 2

  • Method

    The authors probe activation-space geometry to identify attracting basins, then adapt steering strength to move activations into target feature basins.

    Source: §1 · p. 2, §4.2 · p. 10

  • Results

    For English-to-Chinese steering, the parameter-free method yields a 19% reduction in non-Chinese and 4% increase in Chinese tokens under greedy decoding.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present activation-space geometry as a way to interpret language-model computation and control model behavior.

    Source: §1 · p. 3

    The work is exploratory and focuses on feature directions and activations obtained from data, leaving automatic discovery of basins or other geometric features for future work.

    Source: §5 · p. 10

Abstract

from arXiv · show

Language models exhibit remarkable robustness, continuing to produce coherent text even when their activations are perturbed by interventions like linear steering. We hypothesize that this robustness is a result of passive dynamics, i.e., constraining mechanisms in the forward pass that funnel activations toward "good" regions that produce coherent outputs. To investigate these hypothesized error-correcting mechanisms, we probe the geometry of language model activation space by observing the action of model layers on low-dimensional curves. In doing so, we discover that model activations occur within a cluster of distinct attracting basins, which differentiate natural activations geometrically from distributionally similar synthetic activations. Applying this lens to language model steering, we observe feature-specific basins along semantic steering directions, and find that steering moves activations between these basins. To demonstrate the active role of this geometry in neural computation, we show that adaptively modulating steering strength to transport activations across basins improves inter-language steering, significantly increasing the probability of sampling tokens from the target language compared to fixed-strength steering. Our findings establish analysis of activation space geometry as a promising approach to interpreting and controlling language models.

1 Introduction

The paper proposes that constraining dynamics funnel language-model activations into coherent regions, then investigates those regions as attracting basins and uses their geometry to guide steering. It reports that adaptive steering into target basins improves English-to-Chinese steering over fixed-strength steering.

1 Introduction

  • Inspired by biological systems that offload explicit control to physical constraints, the paper characterizes activation-space geometry and frames steering as basin transport.Its stated contributions include geometrically characterizing basins and a parameter-free method for selecting steering strength from activation-space geometry.Source: §1 · pp. 1–3
  • Natural activations lie within a cluster of distinct attracting basins, while distributionally similar synthetic activations show different norm growth and gradually revert toward moderate growth across layers.The authors interpret this pattern as evidence of a distributed error-correcting mechanism.Source: §1 · p. 2
  • Refusal, compliance, and language directions have feature-specific basins, and the English basin occupies a larger angular region than the Chinese and Greek basins.Steering into the refusal basin at one token position is sufficient to trigger refusal of a benign request.Source: §1 · p. 2
  • For English-to-Chinese steering, the parameter-free method yields a 19% reduction in non-Chinese and 4% increase in Chinese tokens under greedy decoding.The method adapts steering strength so the perturbed activation lands in the target feature basin, improving over fixed-strength steering.Source: §1 · pp. 2–3

2 Probing activation space geometry

The section probes activation-space geometry by forwarding low-dimensional spherical paths through model blocks and analyzing the resulting curves and their derivatives. The reported norm-growth profiles show a basin around natural activations across models.

2 Probing activation space geometry

  • A spherical probe passes through an activation of interest, is forwarded through a model block, and yields an output curve whose derivatives indicate local expansion or compression.The probe can be generalized to two dimensions, where change in tangent-plane area measures expansion.Source: §2 · p. 3, Figure 2 · p. 3
  • Select experiments retain only a beginning-of-sequence token because the authors expect feedforward layers to mediate the error-correction phenomenon they study.The retained token also avoids the unusual behavior that can follow removing the entire context in some models.Source: §2 · p. 3
  • The experiments sample natural activations from FineWeb and The Pile and compare them with norm-matched, shuffled, and Gaussian synthetic activations across four models.The models are Llama 3.1 8B, Llama 3 8B Instruct, OLMo 3 7B, and Qwen 3 8B.Source: §2 · p. 3
  • The authors use spherical rather than linear probes to preserve norm across the path, after preliminary linear probes produced large-norm activations at path extremes in some models.Those models normalize the residual update, and the extreme points asymptotically approach f(x) = x + c.Source: §2 · p. 3
  • Across models, norm-growth profiles show a distinctive basin around natural activations, which often sit at a minor local norm-growth maximum.Figure 3 plots perturbation angle against relative norm growth for context-ablated tokens sampled from The Pile; non-ablated context results are reported as similar.Source: Figure 3 · p. 4

3 Transformer block dynamics privilege natural activations

Transformer blocks privilege natural activations through distinctive norm-growth geometry and dynamics that move synthetic inputs toward natural activation regions. Natural activations occupy local norm maxima and clustered basins separated by velocity barriers, unlike synthetic counterparts.

3.1 Natural activations inhabit local norm growth maxima.

  • Natural activations lie in regions of modest norm growth surrounded by high-growth regions, with local maxima at the activation in several layers.For OLMo and some Qwen layers, the antipode also lies in a smaller radial basin.Source: §3.1 · p. 4
  • 0.965 (Qwen), 0.983 (Llama), and 0.997 (OLMo) maximum AUROC for Gaussian activations using a simple growth threshold to distinguish natural from synthetic activations.The norm-growth profile of natural activations is distinct from norm-matched, Gaussian, and shuffled activations.Source: §3.1 · p. 4, Figure 4 · p. 5
  • Synthetic norm growth approaches natural rates through successive layers, as norm-matched probes enter growth basins and shuffled probes eventually reach local maxima.Figure 5 shows synthetic activation growth approaching natural rates; Figure 6 shows the norm-matched probe entering a basin almost immediately and the shuffled probe reaching a local maximum.Source: §3.1 · pp. 4–5, Figure 5 · p. 5, Figure 6 · p. 5

3.2 Natural activations inhabit attracting basins

  • Natural activations occupy velocity basins with peaks near ±90°, and natural pairs show barriers absent from synthetic pairs (barrier-classifier AUROC 0.938–0.996).The barrier classifier uses barrier height, assigning zero to activations in a shared basin.Source: §3.2 · p. 6, Figure 7 · p. 6
  • Attracting basins for sets of up to three natural activations pack together within one hemisphere.The expansion maps visualize regions of stretch across the probe surface.Source: §3.2 · p. 6, Figure 8 · p. 6

3.3 Qualitative uniqueness of natural activation space geometry

  • When a two-dimensional sphere passes through activation pairs, natural pairs produce a double-lobed output while synthetic pairs produce a single lobe.The lobes correspond to the two natural activations in the pair.Source: §3.3 · p. 6

4 Steering transports activations across basin boundaries

The section treats steering as movement between feature-specific activation basins, illustrated with refusal and language steering. It then uses this geometry to motivate adaptive steering strength.

4 Steering transports activations across basin boundaries

  • The section hypothesizes that steering moves activations between basins, while successive layers error-correct perturbations toward natural basins.The authors test this account using refusal and language steering tasks.Source: §4 · pp. 6–7

4.1 Identifying and steering between the refusal and compliance basins

  • Steering only the first response token of a benign request into Llama’s refusal basin triggers refusal and shifts subsequent activations into upper basins.The refusal steering vector is derived from the difference between end-of-user-message activations for harmful and harmless requests.Source: §4.1 · p. 7, Figure 10 · p. 8
  • A relatively stronger intervention at one token position has the same behavioral effect as the typical low-strength steering applied at every request and response position.The authors use this setup to demonstrate steering between basins and suggest it may be less intrusive.Source: §4.1 · p. 7

4.2 Steering between language feature basins for greater behavioral control

  • On 10,000 held-out tokens, en–zh steering yields 17% non-Chinese and 59% Chinese characters; tuning fixed strength yields 11% non-Chinese and 68% Chinese characters.The remaining generations were non-language-specific.Source: §4.2 · p. 8
  • English, Chinese, and Greek activations occupy distinct language-specific basins; English appears largest, with barriers between language basins forming a ⊤ shape.The 2D probes use orthogonalized en–zh and en–el directions; Figure 11 shows low-expansion basins for each language.Source: §4.2 · pp. 8–9, Figure 11 · p. 8
  • Adaptive steering outperforms fixed-strength methods across the tested settings, often bringing results within a few percentage points of the oracle upper bound.The comparison covers mean probability mass and greedy decoding rates for Chinese and non-Chinese tokens on in- and out-of-distribution test sets.Source: Table 1 · p. 9, §4.2 · p. 10
  • Language basins align with steering behavior: activations in one basin decode to English tokens and those in another to Chinese tokens, motivating strength selection to avoid overshooting or undershooting the target basin.Evaluation uses 10,000 held-out English Wikipedia tokens and 10,000 FineWeb tokens excluding Wikipedia.Source: §4.2 · pp. 9–10
  • For each instance, adaptive steering targets the deepest velocity minimum in the steered direction, reverting to tuned fixed strength when no minimum is found in either direction.This fallback occurs less than 0.5% of the time across evaluation sets.Source: §4.2 · p. 10

5 Conclusion

The conclusion presents activation-space geometry as a lens on language-model computation and steering, while emphasizing that the investigation remains exploratory.

5 Conclusion

  • The authors characterize natural activations as occupying error-correcting attracting basins and use feature-specific basins to motivate parameter-free, basin-aware steering.They report improved steering over fixed-strength steering in the English–Chinese setting.Source: §5 · p. 10
  • The authors recommend studying activation-space geometry through layer dynamics as a useful lens for interpretability.Their conclusion connects this analysis to understanding model behavior and computation.Source: §5 · p. 10
  • The work is exploratory, leaving open questions about similar activations, circuits, and finding consequential geometric features without relying on data.The authors anticipate that activations for the same token in slightly different contexts may occupy a low-dimensional manifold.Source: §5 · p. 10

A Additional experimental results

The additional experiments report that natural and synthetic activations differ in multi-layer measures, while a single-layer velocity profile does not distinguish them significantly.

A Additional experimental results

  • Most tested layers of most models score high on norm-growth ROC-AUC for separating 128 natural and synthetic test activations.The reported table describes near-perfect separability in those cases.Source: Table 2 · p. 15
  • Velocity profiles show no significant difference between natural and synthetic activations when measured through a single transformer block.Figure 16 presents per-layer speed around natural activations and compares natural with synthetic profiles across models.Source: A Additional experimental results · p. 16, Figure 15 · p. 16, Figure 16 · p. 16
  • Velocity-barrier classification shows high separation across all models and synthetic activation types.Table 3 reports the velocity barrier-based classification comparison.Source: Table 3 · p. 16

B Details for 2D sphere images

The appendix specifies how the authors construct and project 2D sphere probes around single activations or activation pairs.

B Details for 2D sphere images

  • For a single activation, two random tangents define the sphere, whose transformed surface is projected onto the plane spanned by f(x) and the initial path derivative.The path begins at f(x) and follows a 1D sphere parameterization along one tangent.Source: B Details for 2D sphere images · p. 17
  • For an activation pair, the second activation is norm-matched to the first and supplies one tangent direction, with a second tangent chosen randomly.The pair probe’s path passes through the transformed activations f(x) and f(x′).Source: B Details for 2D sphere images · p. 17
  • The pair probe’s transformed surface is projected onto the plane spanned by the sum of the two transformed activations and the sum of their path derivatives.Source: B Details for 2D sphere images · p. 17

Related papers

Loading 2610.04183v1…