Characterizing stable regions in the residual stream of LLMs

1 Introduction

• Interpolating between residual-stream activations provides the paper’s probe of how activation changes affect model outputs.Source: §1 · p. 1
• Stable regions show minimal output changes within them but large changes near boundaries, and appear to align with semantic distinctions between prompts.Activations from different regions produce different next-token predictions.Source: §1 · p. 1
• Figure 1 depicts solid-color regions emerging and boundaries sharpening across OLMo-7B training stages.Colors encode output similarity to three model-generated activations in a 2D residual-stream slice.Source: §1 · p. 1, Figure 1 · p. 2

2 Related work

• Linear-region counts and VC-dimension bounds offer theoretical insight, but their account of practical network expressivity remains unclear.Source: §2 · p. 2
• Stable regions are studied as residual-stream regions that map to classes of very similar outputs, and are reported to be much larger than previously studied polytopes.Source: §2 · p. 2
• The linear representation hypothesis treats neural-network features as directions in activation space; normalization places residual-stream activations close to a hypersphere.High dimensionality means most linear interpolations remain close to that hypersphere.Source: §2 · p. 2
• Prior work interpolated between residual-stream activations and observed nonlinear output changes, with follow-up studies connecting the topic to SAE research.The cited works include Heimersheim & Mendel, Giglemiani et al. [4], Lee & Heimersheim, Gurnee, and Lindsey.Source: §2 · p. 2

3 Methods

• The method defines a prompt-specific patched forward pass that takes a residual activation and returns last-layer residual-stream activations at the final sequence position.It replaces the activation after the first layer before applying the remaining layers; the reported output is before unembedding and softmax.Source: §3 · p. 2
• Stable regions are expected to have low output sensitivity away from their boundaries and high sensitivity near or across them.Source: §3 · p. 2
• The method interpolates between prompt activations with α from 0 for pA to 1 for pB, then measures L2 output distance from a clean pA run.The patched activation is A + α(B − A), with A and B generated by pA and pB.Source: §3 · p. 2
• Figure 2 presents relative output distance across α for similar and dissimilar Qwen2-0.5B prompt pairs, plus normalized logit difference between their top predictions.Source: Figure 2 · p. 3
• Experiments use Qwen2 and OLMo models from 0.5B to 7B parameters and evaluate distance at 50 uniformly spaced α values.Source: §3 · p. 3

Illustrative examples

• Dissimilar prompt pairs show flat output-distance regions separated by a sharp jump, while similar pairs change nearly linearly across interpolation.The authors interpret this as dissimilar prompts occupying distinct stable regions and similar prompts likely sharing one.Source: §4 · p. 3

Impact of model size

• Median maximum slope increases with model size in both OLMo and Qwen2, confirming sharper relative-output-distance curves.Figure 3 plots median relative output distance across model sizes and maximum slope against parameter count.Source: Figure 3 · p. 4, §4 · p. 4

Illustrative examples

• Stable regions appear larger than previously studied polytopes: semantically different prompts cross one boundary while hundreds or thousands of gates change sign.The comparison is based on interpolation between semantically different prompts.Source: §4 · p. 4
• For dissimilar prompt pairs, the normalized logit difference between top predictions changes similarly to output distance, suggesting that distance tracks interpretable prediction changes.Source: §4 · p. 4

Impact of training progress

• Randomly initialized models have nearly linear curves; training sharpens them, with the effect plateauing earlier for OLMo-1B than OLMo-7B.Figure 4 shows selected checkpoints and maximum slope against training-token count.Source: §4 · p. 4, Figure 4 · p. 5
• In a two-dimensional activation slice, solid-color regions emerge and their boundaries sharpen during training.Colors encode similarity to three model-generated activations; the authors suggest sharpening may reflect more stable regions, sharper boundaries, or both.Source: §4 · pp. 4–5

Scope

• The study examines only a handful of relatively small Transformer language models; similar results were seen for GPT, Pythia, Phi, and Llama, with Gemma a notable exception.Source: §5 · p. 5

Evidence for region size

• The claim that stable regions are relatively large compared with polytopes is supported only indirectly; direct estimates of their size and number remain future work.Source: §5 · p. 5

A Comparison of OLMo and Qwen2 model families

• Table 1 compares the OLMo and Qwen2 model configurations.Source: Table 1 · p. 7

B Similar and dissimilar prompts

• Examples contrast dissimilar prompt pairs with similar pairs that differ by a closely related final word.Source: B Similar and dissimilar prompts · p. 7

C Details of the 2D slice visualization

• Synthetic activations X are created in a 2D slice, with P the orthogonal projection of C onto the line containing A and B, and α and β ranging from -0.25 to 1.25.This allows examination of a broad area around and between the three reference activations.Source: C Details of the 2D slice visualization · p. 7

D More 2D slice plots

• The cited passages identify six items as Figures 5 through 10, without specifying their placement, subject matter, dimensionality, or whether they are plots.Source: Figure 5 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

• Patching after the seventh layer yields smoother curves, yet a significant relative-distance jump remains near the interpolation midpoint.The qualitative comparison uses Qwen2-1.5B; Figures 23 and 24 show patching after the first and seventh layers, respectively.Source: E Shapes when patching after 1st vs… · p. 12, Figure 23 · p. 13, Figure 24 · p. 13

• Hundreds or thousands of gates switch signs between prompts, although interpolation experiments suggest crossing only one stable-region boundary.The gate-sign count is a weak lower bound on polytope boundaries crossed; Figure 25 shows the gate-switch distribution for Qwen2-0.5B.Source: F Direct comparison with polytopes · pp. 13–14, Figure 25 · p. 14

• Maximum derivatives below 1.2 correspond to pairs sharing a last token, while values above 10 correspond to pairs with different last tokens.The analysis uses the maximum derivative of relative distance; Figure 26 shows its distribution, whose long right tail corresponds to pairs crossing stable-region boundaries.Source: G Relationship between output… · p. 14, Figure 26 · p. 14

G.1 Low sensitivity pairs (same last token)

• The examples also include snippets about scattered people and a command to show oneself to children.These fragments appear in a single paired example.Source: §G.1 · p. 15
• Examples pair fragments mentioning Jeddak of Helium and a question about hitting someone with a rock.Both excerpts are presented as a pair in this section’s same-last-token group.Source: §G.1 · p. 15
• Other pairs juxtapose references to New York and to glory, multitude, and their associated context.The excerpts begin with “hide the city of New York under it” and “and their glory, and their multitude.”Source: §G.1 · p. 15

G.2 High sensitivity pairs (different last tokens)

• One paired example juxtaposes a reference to grandfather’s opinions with a fragment about local guides and a fatal outcome.The excerpts are presented together in this section’s different-last-token group.Source: §G.2 · p. 16
• The examples also pair a reference to the emigrant Evrémonde with a fragment about conformity.Both snippets are included in the section’s high-sensitivity group.Source: §G.2 · p. 16
• Another pair places a question about who someone is beside a reference to a birthright and a mess of pottage.The fragments come from different passages in the paired example.Source: §G.2 · p. 16