PaperlayerView exact revision on arXiv

Source-linked AI summary

Component and Dimension Sparsity in Transformer Refusal Mechanisms

Vincent Siu, Glenn Grant-Richards, Vlad Pavlovich, Yizhou Sun, Dawn Song, Chenguang Wang

arXiv:2610.06903v1cs.CLcs.AIcs.LG

TL;DR

Activation steering changes language-model behavior through internal activations, but which components assemble refusal directions is not well understood. This paper decomposes refusal steering across four open-weight models. Steering 28–48% of upstream components retains 88–101% of full effectiveness, with further sparsity across residual dimensions.

  • Problem

    Current steering methods typically intervene at one residual-stream point without resolving which attention and MLP components assemble the steering vector.

    Source: §1 · p. 1

  • Method

    The paper decomposes refusal steering into interventions on individual attention and MLP components and residual-stream dimensions across four open-weight models.

    Source: §1 · p. 1, §2.3 · p. 3

  • Results

    Steering 28–48% of upstream components retains 88–101% of full steering effectiveness, and roughly 50% of residual dimensions retain 85.7–98.6% of full-mechanism effectiveness.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    The findings identify refusal steering as a structured, identifiable mechanism rather than a diffuse effect across all transformer components.

    Source: §7 · p. 10

    The analysis establishes how one refusal direction at one intervention layer is assembled, not the full refusal subspace.

    Source: §2.3 · p. 4

Abstract

from arXiv · show

Activation steering manipulates large language model behavior by intervening on internal activations, but the mechanistic basis of these interventions remains poorly understood. We decompose refusal steering into component-level interventions across four open-weight models, identifying the sparse subsets of attention and MLP components whose steering suffices to reproduce the full behavioral effect. We find that refusal directions concentrate in sparse component mechanisms comprising 28--48\% of upstream components, retaining 88--101\% of steering effectiveness. Within these mechanisms, effective steering further concentrates in approximately 50\% of residual stream dimensions, retaining 85--98\% of the component-mechanism baseline, consistent with a privileged basis structure. Sparsity thus operates at two levels: which components are steered, and which dimensions within those components carry the signal. Together these findings show that refusal is not diffusely encoded across a transformer but assembled by a structured, identifiable mechanism, providing a foundation for mechanistic understanding of how refusal behaviors are represented and steered. To facilitate reproducibility, we release all code and raw experimental results in https://github.com/wang-research-lab/Refusal_Mechanisms.

1 Introduction

Activation steering directly intervenes on model activations, but typical residual-stream interventions leave unclear which attention and MLP components assemble the steering direction. Across four open-weight models, refusal steering is concentrated in sparse component sets and residual dimensions.

1 Introduction

  • Typical steering intervenes at one residual-stream point without resolving which attention and MLP components assemble the steering vector.The paper decomposes refusal steering across four open-weight models at the component and residual-dimension levels.Source: §1 · p. 1
  • Steering 28–48% of upstream components retains 88–101% of full steering effectiveness, while roughly 50% of residual dimensions carry the signal.Randomly rotating the refusal direction reduces steering performance to near-zero under the same coordinate masking; attention supplies the dominant discriminative signal, while MLP contributions are more distributed and can suppress it.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • High-magnitude component writes do not share the refusal direction’s coordinate subspace, suggesting axis-aligned sparsity aggregates component-specific signals.The reported pattern is not a uniformly shared subspace written by all components.Source: §1 · p. 2

2 Background

Attention and MLP outputs add to a shared residual stream, making individual components independently intervenable. The paper defines refusal directions by harmful-versus-harmless activation differences and studies component-level mechanisms as sufficient decompositions of one steering effect.

2.1 Residual Stream

  • Attention and MLP components contribute additively to the shared residual stream, permitting independent component interventions.The forward pass composes these updates across layers.Source: §2.1 · p. 3

2.2 Refusal Directions via Difference-in-Means

  • The refusal direction is extracted from the difference between harmful and harmless prompts’ mean residual activations at each layer.The paper follows the difference-in-means technique and uses the direction at the selected intervention layer for steering.Source: §2.2 · p. 3

2.3 From Residual Stream to Mechanisms

  • A mechanism is a set of attention and MLP components jointly sufficient to reproduce a steering intervention’s behavioral effect, not necessarily a minimal set.The analysis concerns one direction at one intervention layer, not the full refusal subspace; Table 1 reports component fractions and steering ASR relative to a dense residual-stream baseline.Source: §2.3 · pp. 3–4, Table 1 · p. 4

3 Methodology

The methodology selects a refusal direction and intervention layer, then approximates a behaviorally sufficient component mechanism with two-stage greedy elimination. Experiments use four open-weight models and evaluate reported results on held-out prompts.

3.1 Direction Extraction and Layer Selection

  • The intervention layer is selected as the layer whose difference-in-means refusal direction achieves the highest ASR on the full test set.Candidate directions are extracted at the last post-instruction token position, before response generation.Source: §3.1 · p. 4

3.2 Component Mechanism Identification

  • Component outputs are individually orthogonalized before addition to the residual stream, seeking a set that reproduces steering all upstream components simultaneously.Exact subset identification is computationally prohibitive, and component redundancy can make their apparent roles change when another component is removed.Source: §3.2 · p. 4
  • Stage 1 prunes components using mean projection improvement, then Stage 2 validates behavioral necessity by removing components with the smallest ASR drop.Stage 1 stops at the 95% projection threshold; Stage 2 stops when further removal would reduce ASR by more than 0.05 in absolute terms.Source: §3.2 · p. 5
  • The two-stage greedy procedure approximates a minimal behaviorally necessary component set, and its reported sparsity is stable across threshold and tie-breaking checks.Held-out ASR varies by at most five points across the threshold grid; random tie-breaking preserves an invariant core of 8–12 components.Source: §3.2 · pp. 4–5

3.3 Models and Data

  • Experiments cover four open-weight models, with 256 training, 463 validation, and 512 test prompts; reported results generalize to the held-out test set.Harmful prompts come from six datasets, while harmless prompts for direction extraction come from ALPACA.Source: §3.3 · p. 5

4 Results

Across four models, sparse component mechanisms reproduce most of the full refusal-steering effect, and steering remains effective when restricted to a subset of residual dimensions. The sparse intervention also retains capability better than dense steering in most reported model-benchmark pairs.

4.1 Component Sparsity

  • 28.2–48.1% of upstream components reproduce 88.3–101% of the full-component steering effect across four models.The identified complements largely fail to elicit refusal, although Mistral-3.2-Small and Qwen3-4B show some complement steering.Source: §4.1 · pp. 5–6, Figure 3 · p. 7

4.2 Dimensional Sparsity

  • At the 95% norm threshold, roughly half the residual dimensions retain 85.7–98.6% of full steering effectiveness within the component mechanisms.Several models match or slightly exceed full-steering ASR at that threshold; performance drops sharply only below approximately 50% of the squared norm.Source: §4.2 · pp. 6–7, Figure 2 · p. 6
  • After random rotation, masking the same number of dimensions yields 0.6–11.2% ASR, versus 96.8–101.5% in the native basis.This indicates that the observed dimensional sparsity depends on alignment with the residual-stream coordinate system.Source: §4.2 · p. 7

4.3 Capability Retention

  • The sparse mechanism is equal or better than dense steering in ten of twelve capability cells, with the largest reported advantage on Phi4-Mini.There, dense steering costs 22.4 MMLU points and 28.1 GSM8k points and raises perplexity by 2.59, while the mechanism keeps both benchmarks within 2 points of baseline and perplexity within +0.11.Source: §4.3 · pp. 7–8, Table 2 · p. 8

5 Component-Specific Analyses

The component-level analyses examine how individual components contribute to the refusal direction and whether their writes share the same dimension preferences. The findings distinguish concentrated attention contributions from component-specific write profiles whose aggregate is sparse.

5.1 How Components Write the Refusal Direction

  • The identified mechanisms reconstruct 50–63% of the refusal direction, with the remainder attributable to individually sub-threshold components pruned during elimination.Attributions are computed by projecting each component’s harmful-versus-harmless difference-in-means output onto the refusal direction.Source: §5.1 · pp. 8–9, Figure 4 · p. 8
  • Attention components make the largest individual contributions across models, while MLP contributions are smaller and more distributed across layers.Qwen3-4B has negative MLP contributions at layers 12 and 14, though pruning them degrades ASR by around 0.1.Source: §5.1 · p. 9
  • In Mistral-3.2-Small and Qwen3-4B, an early attention component contributes more than many components 7–10 layers closer to the intervention site.The authors suggest its write may propagate nearly intact because subsequent components largely preserve the encoded signal.Source: §5.1 · p. 9

5.2 Component Write Profiles and Dimension Specificity

  • Across all four models, high-magnitude component writes are distributed across selected dimensions rather than tracking the refusal-direction envelope, indicating component-specific profiles whose aggregate is sparse.The estimator has a caveat: near-zero means can reflect either small contributions or cancellation across a non-unimodal activation distribution.Source: §5.2 · pp. 9–10, §5.1 · p. 9

6 Related Work

Related work frames refusal as an activation-space behavior, activation steering as a way to modify target behaviors, and mechanistic interpretability as a route to identifying sparse circuits and component interactions.

Safety and Refusal

  • Prior work represents refusal as a linear, affine, or multidimensional activation-space subspace, while fine-tuning and adversarial prompts can bypass these mechanisms.These bypasses motivate further mechanistic understanding.Source: §6 · p. 10

Activation Steering

  • Activation steering identifies target-behavior directions and modulates them through vector addition or orthogonalization, commonly intervening at one residual-stream location.Directions are commonly extracted from contrastive data pairs and used for behavior elicitation or concept removal.Source: §6 · p. 10

Mechanistic Interpretability

  • Mechanistic interpretability has identified sparse behavior-related subgraphs, while activation patching and path tracing provide causal methods for component-level intervention studies.The privileged-basis hypothesis predicts that high-salience features are axis-aligned rather than distributed in the residual stream.Source: §6 · p. 10

7 Conclusion

The conclusion presents refusal steering as structured at both the component and residual-dimension levels across the studied architectures.

7 Conclusion

  • Across architectures, 28–48% of upstream components suffice for 88–101% of the steering effect, while approximately 50% of residual dimensions carry the signal.The paper notes potential implications for understanding and auditing safety-relevant behaviors in deployed models.Source: §7 · p. 10

Ethics Statement

Across the supplied sections, the paper describes how it selects refusal directions and why sparse coordinates can retain steering signal. It also discusses alternative mechanism-search strategies, compute costs, and the study’s risk scope.

Ethics Statement

  • The authors report that their jailbreaking effects are generally lower than prior work and state that the study does not meaningfully introduce new threats beyond it.They identify Mistral-Small-3.2 steering as an exception, with a one-percent relative improvement over prior methodologies.Source: Ethics Statement · p. 17

A Approximate Compute and Time Budgets

  • The full pipeline takes approximately 8–9 hours on smaller models and 23 hours on Mistral-Small-3.2-24B; validation pruning runs comprise 354–446 evaluations per model.Stage 2 ASR-based pruning is the dominant cost; individual runs take 1–4 minutes depending on model size.Source: A Approximate Compute and Time Budgets · p. 17

B Alternative Mechanism Determination Strategies

  • Alternative searches face distinct constraints: additive greedy search is order-sensitive, exhaustive search is intractable, proxy losses may not maximize ASR, and Shapley attribution assumes invalid order independence.In additive search, early choices can undervalue components useful only in combination, while some attention additions can yield gibberish outputs.Source: B Alternative Mechanism Determination… · p. 18

C Layer Sweep Results

  • The layer sweep tests difference-in-means directions at each layer input and uses the highest-ASR direction as the mechanism-search target and Table 1 reference.Directions are extracted from the last post-instruction token.Source: C Layer Sweep Results · p. 18, Figure 6 · p. 18

D Refusal Direction Distributions

  • Across four models, refusal directions are roughly Gaussian around zero with heavy tails; outlier counts range from one dimension in Phi4-Mini to eight in Qwen3-4B.Outlier signs vary across models, which the authors interpret as model-specific representational geometry rather than a universal encoding convention.Source: D Refusal Direction Distributions · p. 19

E Choosing Magnitude and Norm for Sparse Steering

  • Large-magnitude coordinates mark dimensions with stronger harmful–harmless mean separation, and their concentration in the squared norm motivates sparse steering and a privileged-basis interpretation.The authors connect this pattern to a heavy-tailed direction whose few large coordinates account for a disproportionate share of its norm.Source: E Choosing Magnitude and Norm for… · p. 20

F Bottom-Magnitude Dimension Ablation

The bottom-magnitude dimension ablation tests whether steering effectiveness depends only on mask size. Comparable-sized masks of low-magnitude dimensions yield near-zero relative ASR, supporting functional concentration in the high-magnitude coordinates.

F Bottom-Magnitude Dimension Ablation

  • Across all four models, bottom-magnitude dimensions produce near-zero relative ASR despite budgets of 45–60% of the residual stream, showing that dimension budget alone does not explain steering effectiveness.Gemma2-2B and Phi4-Mini fall below 2.5% relative ASR; Mistral-3.2-Small and Qwen3-4B remain below 15%.Source: F Bottom-Magnitude Dimension Ablation · p. 20

G Random Rotations Experiment

The experiments test whether sparse steering depends on the native coordinate basis, examine component outputs across harmful and harmless prompts, and validate the pruned mechanisms behaviorally. They also show that first-round components can be redundant when steered together.

G Random Rotations Experiment

  • Rotated directions yield 0.6–11.2% relative ASR versus 96.8–101.5% in the original basis, a drop of 85–101 percentage points, showing that selected native-basis coordinates are not interchangeable.The same number of top coordinates is selected after rotation; the table reports the mean across five rotations.Source: G Random Rotations Experiment · p. 21, Table 4 · p. 21

H Component Projection Displacement: Harmful vs. Harmless

  • MLPs project positively onto the refusal direction across prompt types, while several attention components shift toward zero or negative values on harmless prompts, consistent with content-sensitive modulation.The reported shifts are most visible in Phi4-Mini at layer 13 and Qwen3-4B at layers 11 and 18.Source: H Component Projection Displacement:… · pp. 21–22, Figure 8 · p. 22

I Leave-One-Out Circuit Sufficiency

  • Leave-one-out ASR drops identify critical components in the pruned circuits, validating behavioral steering rather than geometric alignment alone.Figure 9 shows absolute ASR drops when individual components are removed; larger drops indicate greater bottleneck importance.Source: I Leave-One-Out Circuit Sufficiency · p. 22, Figure 9 · p. 23

J First-Round ASR Redundancy

  • First-round components show near-zero leave-one-out ASR drops across models, suggesting redundancy when steered together; iterative pruning reveals the functional bottlenecks.The initial set includes components meeting the projection threshold before greedy elimination.Source: J First-Round ASR Redundancy · p. 23

K Threshold Sensitivity: α and τ

Across threshold settings, mechanism size and held-out ASR remain broadly stable, while changing α can select nearby but non-identical mechanisms. Random tie-breaking likewise yields mechanisms with a shared invariant core and small ASR variation.

K Threshold Sensitivity: α and τ

  • Test ASR falls to 0.69–0.74 when α is loosened to 0.90, indicating Stage 1 filters candidates rather than passing through an arbitrary set.Tightening τ to 0.02 retains two to four additional components for a 3–4-point ASR gain; relaxing it to 0.10 removes one to three at comparable cost.Source: K Threshold Sensitivity: α and τ · p. 23
  • Independent reruns at α = 0.95, τ = 0.05 differ from reported mechanisms by 1.7 points in Gemma2-2B and 3.8 points in Qwen3-4B test ASR.The rerun recovers the reported mechanisms for Phi4-Mini and Mistral-3.2-Small; the two differing models have a one-component substitution or addition.Source: K Threshold Sensitivity: α and τ · p. 23
  • 20–56% of D and held-out test ASR within roughly five points of the operating point span all five threshold settings across four models.The α = 0.95, τ = 0.05 configuration lies in the middle of each sensitivity curve.Source: K Threshold Sensitivity: α and τ · p. 23, Table 5 · p. 24
  • α = 0.95 mechanisms are not nested within α = 0.99 mechanisms in any model, whereas all τ-sweep relations are nested.The α = 0.90 mechanism is nested within α = 0.95 for three of four models; added candidates at α = 0.99 can change tied removals and produce a nearby, non-identical set.Source: K Threshold Sensitivity: α and τ · pp. 24–25, Table 6 · p. 25

L Tie-Breaking and Seed Stability

  • Mean pairwise Jaccard similarity under random tie-breaking is 0.66–0.88, with an invariant core of 8–12 components and within-model test-ASR variation of at most 2.8 points.Components that vary are those whose removal costs less than ϵ; every model has a smaller peripheral set around the shared core.Source: L Tie-Breaking and Seed Stability · pp. 25–26, Table 7 · p. 26

M Capability Evaluation Protocol

The capability protocol compares base, full-steering, and mechanism-steering checkpoints using standard capability and perplexity evaluations. The mechanism condition applies steering only to identified components and retained coordinates, while checkpoints are exact weight-space equivalents of hooked models.

M Capability Evaluation Protocol

  • Both interventions are baked into weights, making served checkpoints exact weight-space equivalents of hooked models rather than approximations.The full condition applies steering to every attention output projection and MLP down projection in layers [0, l∗); the mechanism condition restricts it to identified components and retained coordinates.Source: M Capability Evaluation Protocol · p. 26
  • MMLU (0-shot) and GSM8k use their public Inspect Evals tasks, with full 13,937-sample and 1,319-sample sets, respectively, and generation capped at 2048 tokens.Each checkpoint is served under vLLM.Source: M Capability Evaluation Protocol · p. 26
  • Perplexity is measured on monology/pile-uncopyrighted over 64 batches of 8 sequences, with maximum length 512 and approximately 179k tokens per model.Base, full, and mechanism conditions are evaluated in one pass over a loaded model to eliminate variation from reloading.Source: M Capability Evaluation Protocol · p. 26

N Hyperparameters

  • The hyperparameter table defines target layer l∗, the norm-explained threshold for retained dimensions, and α and τ for the two elimination stages.The table points to Table 1 for the resulting dimensions and ASRs, and Appendix K for threshold sensitivity.Source: Table 8 · p. 27

O Example Responses

The example responses show that full, dense component-wise, and sparse component-wise steering can still produce harmful guidance on the illustrated prompts. The examples include email-account intrusion and deceptive stock-market rumors.

O Example Responses

  • The full residual-stream and component-wise steering examples provide email-hacking guidance, including phishing, social engineering, and brute-force steps.The sparse example also describes brute-force password attempts and account access.Source: O Example Responses · pp. 28–29
  • For a prompt requesting a false Apple-spying story to influence stock prices, full, dense, and sparse steering examples all offer ways to spread the claim.The suggestions include social platforms, fabricated expert endorsements, and stock-focused rumor seeding.Source: O Example Responses · pp. 29–30
  • On the insider-trading prompt, full, dense, and sparse steering examples give advice framed around concealing or disguising unlawful trading.The sparse response describes creating a false trail and timing trades before news breaks.Source: O Example Responses · p. 31

Related papers

Loading 2610.06903v1…