PaperlayerView exact revision on arXiv

Source-linked AI summary

Locating and Steering Refusal Beyond Attention

Preethi Carmel Bosco, Gopalakrishnan Srinivasan

arXiv:2609.04721v1cs.LG

TL;DR

The paper asks whether refusal representations transfer beyond transformers and where they should be read in architectures with different information-routing mechanisms. It aligns representation spaces, locates architecture-specific write sites, and evaluates a detector-triggered gate. Refusal transfers as a representation across four architecture classes, while readout must be re-estimated at each model’s fresh write site and supports lower jailbreak success, including under an adaptive SSM attack.

  • Problem

    Prior refusal findings and safety tooling are transformer-based, leaving unclear whether refusal is architecture-specific and where it resides in models using recurrent or hybrid routing.

  • Method

    The paper aligns model representation spaces, estimates refusal directions and detectors at architecture-specific fresh write sites, and evaluates them with matched controls and a detector-triggered gate.

  • Results

    Refusal transfers across SSM, transformer, recurrent, and hybrid architectures, while write-site readouts outperform accumulated-residual readouts and the gate lowers jailbreak success across all four classes.

  • Takeaways & Limitations

    Porting refusal-based safety tooling requires finding the new architecture’s write site and re-estimating the direction there, not rebuilding the representation from scratch.

  • Takeaways & Limitations

    The defense is bounded by its detector: adaptive or wrapped prompts can evade detection, and the technique requires white-box access, paired contrastive data, and per-architecture calibration.

Abstract

from arXiv · show

Where inside a language model does refusal live, and does that place change when the architecture does? In a transformer, refusal is governed by a single direction in the residual stream, a finding that safety and interpretability tooling now depend on. State-space models (SSMs) route information through a recurrent update instead of attention, sharing no token-mixing mechanism with a transformer. Does the same safety representation survive this shift, or must it be rediscovered per architecture? It survives. A single rigid rotation, which can only reorient a space and not reshape it, aligns one model's representation space with another's, so the two genuinely share the representation. A harm probe trained on a transformer then flags an SSM's harmful inputs, and removing the aligned direction makes a model answer attacks it would otherwise refuse, while a random direction of the same size does far less. What is architecture-specific is not where the direction is steered but where it must be read. Each layer computes a fresh output that is then added into the residual stream, and harm is cleanly readable at this output, the write site, before the addition. A control that holds the intervention's strength fixed shows that what matters is where the direction is estimated, not where it is applied. Applied through a detector-triggered gate, this direction lowers jailbreak success in all four architecture families we test (SSM, transformer, recurrent, hybrid), and on the SSM it holds against an attacker that tunes its prompt against the defense. The gate only matches a trivial rule that returns a fixed refusal whenever the same detector fires, so what transfers across architectures is the direction itself, not defense strength. Safety tooling built on refusal therefore ports to a new architecture by re-estimating the direction at that architecture's write site, not by rebuilding it.

1 Introduction

The paper finds that refusal is a shared representation across architectures, while its clean readout location depends on each architecture’s fresh write site. A transported direction and write-site readout support refusal control across transformers, SSMs, recurrent models, and hybrids.

  • Shared refusal representation: Refusal is shared across attention, selective-scan, token-shift recurrent, and hybrid architectures.A rigid rotation aligns representation spaces and transports the refusal direction; removing it jailbreaks target models, while a random direction has a much smaller effect.
  • Architecture-specific readout: Harm is most cleanly readable at each architecture’s fresh write site, before the residual accumulates it.The relevant sites are the attention output in transformers and the selective-scan block output in SSMs.
  • Architecture-specific readout: AUROC 0.99 at the write site separates harmful from benign prompts, versus 0.54 for a surface-only probe.A matched-magnitude control shows that the direction estimated at the write site controls refusal, whereas one estimated in the accumulated residual degenerates; intervention location is irrelevant.
  • Cross-architecture defense: The detector-triggered gate lowers Falcon-Mamba prefix-injection attack success from 98.0% to 5.0% and reduces attacks across three further architecture classes.On Falcon-Mamba, attack success remains 1.0% from 15.0% under a suffix optimized through the gate.
  • Cross-architecture defense: The gate’s results match a fixed-refusal baseline using the same detector, so the transferable object is the refusal direction rather than defense strength.The gate serves as a causal test of the readout claim across the tested architecture classes.

2 Method

The method locates refusal at architecture-specific write sites, trains a detector and refusal direction there, and closes the loop with a gated intervention. It compares write-site and residual readouts while controlling detector, magnitude, coherence, and baseline conditions.

  • Detector and gating: The detector makes one prompt-level decision from a linear probe at the selected routing layer, then gates the whole generation when its score exceeds θ.Harmful and benign prompts are both wrapped in jailbreak templates during training, preventing the probe from relying on wrapper text.
  • SSM write site: In Falcon-Mamba, the block output y_t is the selective-scan contribution written before addition to the residual, while Δ is a readout-only location.The refusal direction is amplified at the block output, not at Δ; gain and routing layers are selected by a held-out sweep below the degeneration threshold.
  • Transformer reference: For transformers, the method estimates refusal at the attention output projection and compares it with estimation in the accumulated residual.The attention-output direction defends in both tested transformers, whereas the residual-estimated direction degenerates under a matched-absolute-magnitude control.
  • Write-site comparison: The position-wise MLP output also yields a usable direction, ruling out cross-position mixing as the source of the effect.This supports locating refusal in a fresh contribution rather than requiring attention-based token mixing.
  • Recipe: The deployment recipe finds the write site, selects layers by held-out decodability, estimates a mean refusal direction and linear trigger, calibrates gain, and deploys closed-loop.Layer selection and gain calibration remain model-specific; naively transferring constants between models fails.
  • Baselines: The baselines compare the gate with residual-subspace projection, residual-based conditional activation steering, and an always-on push.All cells share the critical layers and detector threshold, while erasure rank and gate strength are calibrated for coherent generation.

3 Experimental Setup

The experiments span multiple models, corpora, attacks, judges, metrics, and baselines, with probes trained on harmful versus benign-but-spicy prompts. Attack success, over-refusal, capability, and cross-architecture transfer are evaluated using the specified datasets and controls.

  • The setup summarizes models, corpora, attacks, judges, metrics, and baselines across the experiments.
  • The detector is trained on harmful and benign-but-spicy prompts, while decodability uses AUROC and attack-success uncertainty uses Wilson 95% confidence intervals.
  • Attack success is the Llama-Guard-3-8B-judged unsafe fraction, over-refusal uses benign-but-spicy XSTest prompts, and capability is accuracy with and without the gate.
  • Transferred refusal-direction removal is compared with native removal and an orthogonalized random control across six transfer directions.

4 Locating refusal across architectures

Refusal is a shared representation across transformer, SSM, recurrent, and hybrid architectures, but it is most cleanly read at each architecture’s fresh write site. Rigid-rotation transport and causal ablations show that the direction transfers across models, while intervention strength and location are separable from where the direction is estimated.

  • Write-site readout: Harm peaks at the selective-scan mixer output, with AUROC 0.99 versus 0.95 at ∆ and 0.54 for a surface-only probe on matched prompts.The step size exposes harm through input-dependent selection, but the mixer output is the fresh write where decodability peaks and steering is effective.
  • Write-site readout: The same harm locus remains decodable across 130M–7B Mamba models, with ∆ AUROC 0.89–0.99, and reaches AUROC 1.00 at the Mamba-2 mixer output and transformer residual stream.This supports stability across model scale, SSM generation, and the transformer reference.
  • Shared representation: Category-held-out transport tracks each target’s own probe across XSTest categories, supporting shared harm structure beyond saturated aggregate scores.The evaluation fits probes on seven categories and tests the eighth, comparing transported and target-native AUROC category by category.
  • Causal transfer: Across non-transformer architectures, removing the transported direction raises attack success more than matched random controls, with decisive effects at Mamba write sites and narrower effects on RWKV-6.RWKV-6 provides an additional token-shift recurrence test: its time-mix write point steers attack success to 17.5% versus 82.5% for a matched random direction, while the residual remains inert at 47.5%.
  • Shared representation: A rigid Procrustes rotation transports a transformer refusal direction into an SSM, preserving its causal refusal role rather than merely fitting a flexible target map.Removing the transported direction jailbreaks Falcon-Mamba comparably to removing its native direction, while orthogonalized random controls have much smaller effects.

5 Dissecting the mechanism on the anchor

On Falcon-Mamba, refusal is controlled by a direction estimated at the block-output write site, not by the residual direction or intervention location. A detector-triggered gate reduces attacks while preserving selectivity, but its transferable contribution is the direction rather than defense strength.

  • The write-read direction: A matched-magnitude control shows that pushing at the residual or write site performs similarly; failure comes from estimating the direction in the accumulated residual.The residual-estimated direction degenerates at deployed gains, whereas the write-site direction remains usable.
  • Selectivity and scope: The gate matches a same-detector fixed-refusal baseline, so its causal evidence concerns the portable direction rather than superior defense strength.Open-loop steering also over-refuses, blocking 86.0% of benign prompts, whereas the gate remains at the 12.0% undefended rate.
  • Closed-loop defense: 98.0% to 5.0% attack success under prefix injection demonstrates the block-output gate’s main Falcon-Mamba result.The gate applies the refusal direction after a detector decision and lowers attacks across additional jailbreak conditions.
  • Selectivity and scope: The single-direction gate does not fully close persona roleplay, and the persona-specific two-direction fix does not generalize across models.The limitation is explicitly scoped to the attack family rather than to prefix injection results.
  • Closed-loop defense: CAST and residual projection barely defend, while the block-output gate reaches the lowest attack success among compared defenses.The figure contrasts write-site reading with residual-based steering and randomized smoothing.

6 Related Work

The paper extends activation monitoring and refusal steering across architectures by reading and controlling signals at architecture-specific write sites. It positions rigid cross-architecture transport as a new bridge beyond prior within-architecture work.

  • Activation monitors and gating: A detector-triggered gate closes the detect-then-intervene loop at the write site, where fresh contributions enter the residual stream.The trigger is a linear probe, while the steered refusal direction is separately estimated at that site.
  • Activation monitors and gating: Both the monitored signal and steered direction port across architectures after re-estimation at each write site.The resulting monitoring-and-control stack need not be rebuilt for every architecture.
  • Residual directions: Residual directions remain readable but brittle: the paper finds analogous structure in a pure SSM while distinguishing readability from controllability.The SSM result extends a transformer-centered account of refusal directionality.
  • Cross-architecture transfer: A rigid rotation transports refusal across the attention-to-pure-SSM gap, validated with category-held-out cross-validation and a working SSM defense loop.The paper identifies this as, to its knowledge, a previously unaddressed transport setting.

7 Scope and Limitations

The study tests broad attack coverage but has uneven replication and several scope boundaries. Its adaptive robustness depends on detector coverage, calibrated gains, baseline alignment, and limited human validation on newer architectures.

  • Experimental scope: Seven attack types are tested, with replication across two SSM checkpoints and two transformers but only one checkpoint each for RWKV-6 and Zamba2-7B.The recurrent and hybrid results therefore lack second-release replication.
  • Attack-specific limits: The two-direction persona fix lowers Falcon-Mamba attack success from 81.5% to 17.5% but fails differently at RWKV and Zamba2 write sites.It is a model-calibrated persona augmentation, not a general recipe step.
  • Detector dependence: The detector bounds the defense: optimizing against the deployed gate raises attack success from 3.7% to 15.3%, and wrapped-benign confounds persist at Zamba2’s Mamba site.A detector that misses or misclassifies inputs limits the resulting defense regardless of steering.
  • Calibration and comparison: Gain calibration is necessary because excessive gains can degenerate outputs or backfire, while defense margins vary with each model’s baseline alignment.Cross-architecture comparisons therefore concern the site-versus-direction pattern, not absolute margins.
  • Evaluation limits: Human validation supports the LLM judge, but agreement is measured on Falcon-Mamba and newer architectures have few human-rated completions.The reported judge agreement is 94.8% with consensus on n=97 doubly rated completions.

8 Conclusion

Refusal transfers across architectures as a shared representation, while clean readout occurs at architecture-specific write sites. Porting safety tooling therefore requires re-estimating the direction at each new architecture’s write site rather than rebuilding the stack.

  • Conclusion: A transformer refusal direction mapped into Mamba remains causally meaningful, while a random control does not produce the same jailbreak effect.The shared representation survives despite different token-mixing mechanisms.
  • Conclusion: Porting the safety stack follows five steps: find the write site, select layers by held-out decodability, fit probe and direction, calibrate gain, and deploy closed-loop.The recipe is demonstrated across SSM, transformer, recurrent, and hybrid architectures.

Ethics Statement

The paper argues that its interpretability and defensive value outweigh its dual-use risk, while acknowledging that the rotation-transport technique can aid open-weight safety removal.

  • The rotation-transport method can reconstruct and ablate refusal directions across unfamiliar architectures, including SSM, recurrent, and hybrid models.The technique requires white-box access, paired contrastive data, and per-architecture rotation fitting.
  • The same write site that helps defenders steer refusal is also available to attackers seeking to remove safety training.The paper describes this procedure as symmetric by construction.
  • The access barrier does not protect against actors who already possess open model weights and want to redistribute uncensored forks.The limitation specifically concerns attackers with legitimate white-box access.

A Technical Appendix

The technical appendix formalizes the paper’s detector, refusal direction, and write-site gate, and collects the supporting results and robustness analyses referenced by the main text.

  • The appendix provides formal definitions for the detector, refusal direction, and write-site gate.It is organized as a self-contained technical reference for the main paper.
  • It gathers full result tables, figures, supporting experiments, ablations, and robustness checks cited in the main text.Each subsection identifies the main-text claim it supports.
  • The main paper remains self-contained for every headline result, while appendix cross-references point back to sections of the main paper.

A.1 Formal definitions

The formal definitions specify how detectors, refusal directions, gates, cross-architecture maps, and unequal-width representations are constructed and applied.

  • Detector: A routing-layer detector standardizes pooled activations and scores them with an L2-regularized linear probe.The detector is evaluated at a selected routing layer.
  • Detector: The gate fires once per prompt when the harm score exceeds threshold θ.
  • Refusal direction: The refusal direction is the unit mean-difference direction, distinct from the detector’s discriminative direction.
  • Multi-layer gate: On a positive decision, the gate adds αû_ℓ at every selected layer throughout generation.The selected set L contains the top-k layers by held-out AUROC, and α is calibrated separately.
  • Cross-architecture maps: The structural cross-architecture test uses an orthogonal Procrustes map, while steering transfer uses a ridge map.The orthogonal map can rotate but cannot synthesize a new direction.
  • Harm-weighted pool: The harm-weighted pool weights token scores by each token’s harm logit.
  • Unequal widths: For unequal hidden widths, the thin-SVD map is semi-orthogonal and norm-preserving on its row space.It remains a partial isometry rather than a full square rotation.

A.2 Recipe for siting the gate on a new model

The recipe for siting a gate on a new architecture first identifies its write site, then selects layers whose harm signals are decodable without over-steering.

  • Procedure: The procedure is summarized as a step-by-step method for applying the gate to a new model.
  • Find the write site: Identify where the architecture moves information between positions and writes it into state, such as SSM mixer outputs or transformer attention projections.The selective-scan parameters and block output define relevant SSM sites.
  • Select decodable layers: Probe each layer for harm and select the top k by held-out AUROC.The ranking criterion is model-specific, and Falcon-Mamba’s mixer-output ranking is usable across positions.

A.3 Cross-architecture transfer, ablation test

Removing a mapped refusal direction raises target attack success like removing its native direction, whereas an orthogonal random control has little effect across model pairs and directions.

  • Transferred-direction ablations raise attack success like native-direction ablations across three model pairs and both transfer directions.The transferred-minus-random gap holds across all pairs.
  • 60.0% against 27.0% on Mamba→Mistral preserves the transfer effect with an orthogonalized random control.This pair previously made a raw mapped-random control appear non-specific.
  • Transferred-direction jailbreak completions pass the coherence screen on 77.0% to 100.0% of prompts.Native-direction completions pass on 93.0% to 100.0% of prompts.

A.4 The locus across model scale

Figure 6 shows that the locus holds as the model grows.

  • A.4 The locus across model scale: Figure 6 reports that the locus holds as the model grows.
  • A.4 The locus across model scale: The reported locus remains stable across increasing model scale.
  • A.4 The locus across model scale: Model growth does not displace the locus shown in Figure 6.

A.5 Gate-strength calibration on a second SSM

Figure 6 calibrates gate strength on a second state-space model to characterize the defense operating point.

  • Gate strength is calibrated on the second SSM, Falcon3-Mamba.

A.6 Geometric similarity is unreliable for cross-architecture transfer

Geometric similarity between mapped and target refusal directions is unreliable because the mapping can bias even random directions toward the target axis. Orthogonalized controls are therefore needed to test behavioral specificity.

  • Cosine similarity between mapped and target refusal directions is not a reliable transfer metric.
  • Figure 6 compares harm decodability across Mamba scale and gate-strength effects on Falcon3-Mamba.The left panel contrasts step-size decodability with a near-chance surface probe; the right panel compares calibrated gate strengths against undefended rates.
  • A ridge map can give a random source direction cosine up to 0.515 with the target refusal axis.Such mapped-random directions are not always behaviorally inert.
  • The ablation control orthogonalizes the mapped random direction to the target refusal axis, yielding cosine approximately 0.This removes the leaked refusal component before testing specificity.

A.7 A matched random-rotation null for the ceiling transport

Cross-validation and causal tests show that rigidly transported refusal structure remains aligned across architectures, while defense performance depends on the target’s learned refusal behavior and intervention site. The evidence also exposes distinct failure modes, including degenerate transformer-target ablations and steering limits in lightly safety-tuned models.

  • Transport validity: 0.95 and 0.95 Spearman correlations show that transported and native probes track one another across category-held-out folds.After removing the discrimination fold, Pearson correlation remains 0.85 for Llama and 0.99 for Mistral, with exact permutation tests clearing 0.015 and 0.001.
  • Transport validity: 0.998-to-0.998 cosine alignment into Falcon-Mamba shows that standardized rigid rotation closely recovers the target refusal direction.This alignment is obtained by fitting the rotation on standardized activations.
  • Causal transport: 44.0% and 45.0% attack success after transported ablation in the SSM matches native-direction ablation at 41.5% and exceeds the random control at 9.0%.The resulting completions are 100.0% coherent and constitute harmful instructions rather than degeneration.
  • Causal transport: 92.5% transformer-target attack success is unreliable evidence because only 3.0% of completions pass the coherence screen.The paper excludes this cell because over-suppression produces degenerate output that the harm judge miscounts as compliance.
  • Scope and limits: AUROC 1.00 on Codestral-Mamba-7B’s mixer output shows that harm remains decodable across the Mamba-2 generation, despite the model’s different developer.The mean AUROC is 0.98 across layers, with 61 of 64 layers above 0.95.
  • Scope and limits: 97.0%-to-28.5% and 97.0%-to-29.5% attack-success reductions show mapped-direction steering helps safety-tuned SSM targets, whereas lightly safety-tuned Codestral-Mamba has little refusal behavior to amplify.The paper therefore scopes the write-site defense to safety-tuned pure SSMs; defense transfer also depends on applying steering at an effective intervention site.
Loading 2609.04721v1…