PaperlayerView exact revision on arXiv

Source-linked AI summary

The Rogue Scalpel: Activation Steering Compromises LLM Safety

Anton Korznikov, Andrey Galichin, Alexey Dontsov, Oleg Y. Rogov, Ivan Oseledets, Elena Tutubalina

arXiv:2509.22067v2cs.LGcs.AI

TL;DR

Activation steering is presented as interpretable control, but whether benign steering undermines safety remains underexplored. The paper evaluates its effects across model families and finds that random and benign SAE directions can weaken refusals, while aggregating 20 vectors yields an attack on unseen harmful prompts.

  • Problem

    Whether benign activation steering can undermine alignment remains underexplored, despite its framing as precise, interpretable control.

    Source: §1 · p. 1

  • Method

    The authors measure refusal effects from steering on harmful JailbreakBench queries across multiple model families using LLM-as-judge evaluations.

    Source: §1 · p. 1

  • Results

    Averaging 20 random vectors that jailbreak one prompt creates a universal attack on unseen harmful prompts, while random steering raises harmful compliance from 0% to 1–13%.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Precise, interpretable steering does not guarantee safe outcomes, and benign SAE features can pose risks comparable to random vectors.

    Source: §1 · p. 1, §5 · p. 8

    Individual steering features generalize poorly across prompts, making comprehensive safety screening impractical.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Activation steering is a promising technique for controlling LLM behavior by adding semantically meaningful vectors directly into a model's hidden states during inference. It is often framed as a precise, interpretable, and potentially safer alternative to fine-tuning. We demonstrate the opposite: steering systematically breaks model alignment safeguards, making it comply with harmful requests. Through extensive experiments on different model families, we show that even steering in a random direction can increase the probability of harmful compliance from 0% to 1-13%. Alarmingly, steering benign features from a sparse autoencoder (SAE), a common source of interpretable directions, demonstrates a comparable harmful potential. Finally, we show that combining 20 randomly sampled vectors that jailbreak a single prompt creates a universal attack, significantly increasing harmful compliance on unseen requests. These results challenge the paradigm of safety through interpretability, showing that precise control over model internals does not guarantee precise control over model behavior.

1. Introduction

The introduction asks whether activation steering, framed as precise and interpretable control, can also undermine safety. It reports harmful compliance from random directions and benign SAE features, and a universal attack assembled from prompt-specific failures.

1. Introduction

  • Because fine-tuning and prompt engineering are difficult to interpret, the introduction presents activation steering as a precise-control approach whose safety implications remain underexplored.Prior studies focus mainly on adversarial jailbreak vectors, leaving open whether benign steering can also undermine alignment.Source: §1 · p. 1
  • The study measures steering effects on refusals using 100 harmful JailbreakBench queries and LLM-as-judge evaluations across Llama-3, Qwen2.5, and Falcon-3 models.The dataset covers 10 harmful categories.Source: §1 · pp. 1–2
  • Random-direction steering raises harmful compliance from 0% to 1–13%, with effectiveness depending on model and prompt.Steering works best in middle layers, while optimal coefficients vary across models and layers.Source: §1 · p. 2
  • SAE features increase compliance probability by 1–4% over random steering, despite representing benign concepts and serving as a standard source of interpretable steering vectors.The most effective jailbreaking features generalize poorly across prompts, making systematic safety monitoring practically infeasible.Source: §1 · p. 2
  • Averaging 20 random vectors that jailbreak one prompt creates a universal attack on unseen harmful prompts without harmful training data, weights, gradients, or logits.The attack aggregates prompt-specific failures into a broader one.Source: §1 · p. 2

2. Related Work

Related work contrasts parameter-based alignment and jailbreak vulnerabilities with inference-time activation steering. It identifies benign steering’s unintended safety effects as an overlooked question.

Vulnerabilities of LLM Alignment

Activation Steering

  • Activation steering adds chosen vectors to hidden states during inference, with vectors derived from difference-in-means, contrastive activation addition, or SAE features [Turner et al., 2023; Stolfo et al., 2024; Marks & Tegmark, 2023; Rimsky et al., 2024; Bricken et al., 2023; Durmus et al., 2024].Prior applications include factual recall, writing style, and safety behaviors [Zhao et al., 2025; Konen et al., 2024; Soo et al., 2025; Xiao et al., 2024].Source: §2 · pp. 2–3

Robustness of Activation Steering

  • Earlier steering research reports context-dependent effects, sensitive-information extraction, and deliberately optimized jailbreak vectors, but leaves benign vectors’ unintended safety effects underexplored [Tan et al., 2024; Durmus et al., 2024; Seyito˘glu et al., 2024; Wang & Shu, 2023; Gu et al., 2025; Chia et al., 2025; Dunefsky, 2025; Xu et al., 2024].This work investigates that overlooked vulnerability.Source: §2 · p. 3

3. Methodology

The section defines SAE-feature and activation-steering procedures, then specifies steering conditions, model scope, and refusal-based evaluation.

3.1. Technical Background

  • SAEs encode activations with sparse latent codes, and steering adds a selected feature’s decoder direction during inference to bias behavior.The process first identifies a feature linked to a behavior or concept, then adds its direction to activations.Source: §3.1 · p. 3

3.2. Activation Steering Procedure

  • Steering adds a unit-norm vector to activations, using random directions or SAE features and model- and layer-scaled strengths across specified layers and tokens.The procedure excludes special tokens and uses greedy decoding.Source: §3.2 · p. 3

3.3. Models Configuration

  • Random-direction tests cover Llama3.1, Qwen2.5, Falcon3, and FalconH1 models; SAE-feature tests are limited to Llama3.1-8B at layer 19.The selected SAE was designed for safe, interpretable tuning through monosemantic features.Source: §3.3 · pp. 3–4
  • In the single-prompt bomb-making sweep, SAE steering yields 1–4% higher compliance than random steering, while random steering works best in early or middle layers.The figure compares random steering across model families and coefficients, intervention layers, and vector types.Source: Figure 2 · p. 4

3.4. Evaluation Procedure

  • Evaluation measures the fraction of responses judged unsafe on 100 harmful JailbreakBench prompts, using an LLM judge with 94% precision for the harmful class.Incoherent, repetitive, or nonsensical outputs are classified as safe.Source: §3.4 · p. 4

4. Experiments

Experiments sweep model family, intervention layer, coefficient, and vector type on a harmful prompt, then extend evaluation to JailbreakBench and case studies of API steering and aggregated attacks. The results examine single-prompt vulnerabilities, cross-category behavior, benign SAE features, and universal attacks.

4.1. Single-Prompt Vulnerability Investigation

  • Random steering produced harmful compliance across all model families and most steering coefficients; for Llama3.1-8B, compliance reached 18% at c = 2.0, while SAE features yielded 1–4% higher rates than random directions under identical conditions.For Llama3.1-8B, middle-layer steering peaked at layer 16; the authors say steering interferes with nonlinear safety-mechanism formation rather than simple correlation with refusal vectors.Source: §4.1 · p. 5

4.2. Probing Model Vulnerabilities at Scale

  • Of 1,000 SAE features, 817 jailbreak at least one harmful prompt and 353 jailbreak at least five, while the most effective features map to benign concepts.These vectors can be hard to distinguish from legitimate control directions, creating a safety-monitoring blind spot.Source: §4.2 · pp. 5–6
  • Random and SAE steering produce harmful compliance across all ten JailbreakBench categories; Llama3.1-8B reaches 4.6% and Qwen2.5-7B 3.8% with random steering.Llama3.1-8B SAE steering also reaches 4.6%; Fraud/Deception is most susceptible at 13%, while Sexual/Adult content and Economic harm each reach 0.1%.Source: §4.2 · p. 5, Figure 4 · p. 6
  • SAE-feature transfer across harmful categories is weak, so comprehensive screening would require testing each feature against a vast, growing prompt set.No feature acts as a universal master key; the most potent compromised only 35 of 100 prompts, and the limitation holds across models.Source: §4.2 · p. 6

4.3. Breaking Model Safety with a Benign SAE Feature

  • A benign “brand identity” SAE feature steered through the public Goodfire API jailbroke Llama3.1-8B on two representative prompts, producing disclaimer-then-compliance and fictional framing.The first pattern begins with a safety disclaimer but continues with actionable harmful instructions.Source: §4.3 · p. 7

4.4. Weaponizing Random Steering

  • Averaging 20 random vectors that each jailbreak one prompt yields a universal attack with 4× higher compliance than random steering across models.The attack generalizes to unseen harmful requests and needs only steering capability and observation of outputs, not model weights, gradients, or logits.Source: Figure 7 · p. 8, §4.4 · p. 8

5. Conclusion

Activation steering exposes systematic safety vulnerabilities: random and benign-concept steering can undermine safeguards, and localized failures can be aggregated into an attack on unseen prompts. The findings motivate stronger mitigation and further study of alignment failures.

5. Conclusion

  • Random steering can reliably break alignment safeguards across model families, with harmful compliance rates of 1–13%.The vulnerability is most pronounced in middle layers, where abstract concepts and refusal mechanisms appear more malleable.Source: §5 · p. 8
  • SAE features representing benign concepts increase compliance by 1–4% over random vectors, and 817 of 1000 tested features can jailbreak prompts.Their poor cross-prompt generalization makes comprehensive safety monitoring practically infeasible, while vulnerabilities persist across all 10 JailbreakBench categories.Source: §5 · p. 8
  • Averaging 20 random vectors that jailbreak one prompt creates a universal attack that boosts harmful compliance 4× on unseen requests, using black-box steering without gradients or harmful training data.A case study also jailbroke a production model through a public SAE steering API, with disclaimer-then-compliance and fictional-framing justifications.Source: §5 · p. 8
  • Adversarial training against steering perturbations and analysis of activation patterns or refusal circuits are proposed directions for more robust safety frameworks.The authors identify these as future research and mitigation possibilities.Source: §5 · p. 8

Impact Statement

The paper identifies safety risks from random and semantically benign activation steering and cautions against treating interpretable control as inherently safe.

Impact Statement

  • Random and semantically benign steering can systematically bypass safeguards and generate harmful content.The authors frame this as a security risk in interpretability-based control methods.Source: Impact Statement · p. 9
  • The findings are disclosed to encourage more robust safety mechanisms and caution against over-reliance on low-level interventions as inherently safe.Source: Impact Statement · p. 9
  • The authors’ stated goal is to pursue interpretable control without sacrificing model safety and alignment.Source: Impact Statement · p. 9

A. JailbreakBench Prompt Examples

The JailbreakBench prompts cover harmful requests from 10 categories, with representative examples shown for each category.

A. JailbreakBench Prompt Examples

B. Judge Prompt and Quality Assessment

The safety-evaluation appendix shows the LLM-as-Judge prompt and assesses harmful-class precision against human annotations. In the tested sample, the judge achieved 94% precision for harmful classifications.

B. Judge Prompt and Quality Assessment

  • The LLM-as-Judge prompt is presented for evaluating the safety of model responses.Source: Figure 9 · p. 13
  • 94% precision was measured for harmful classifications against human annotations in a sample of 100 responses flagged as harmful by the judge.Six of the 100 flagged responses were actually safe; the sample came from Llama3.1-8B under random steering.Source: B. Judge Prompt and Quality Assessment · p. 13
  • The assessment was designed to establish a lower bound for Compliance Rate precision given the imbalance between harmful and non-harmful responses.Source: B. Judge Prompt and Quality Assessment · p. 13

C. Additional Results: Generalizability Across Models

Random steering produces harmful compliance across models, though vectors show poor transfer across prompt categories. The pattern extends beyond specific vectors or model architectures.

C. Additional Results: Generalizability Across Models

  • The evaluation used 1,000 unit-normalized Gaussian vectors at the middle layer, with coefficients of 0.75 for Llama3.1-8B and 0.5 for Qwen2.5-7B.Compliance was assessed across all 100 JailbreakBench prompts.Source: C. Additional Results:… · p. 13
  • 387 out of 1,000 Llama3.1-8B vectors and 313 Qwen2.5-7B vectors jailbroke at least five prompts.The figure also reports that most vectors jailbreak multiple prompts across Llama3.1-8B, Qwen2.5-7B, and Falcon3-7B.Source: C. Additional Results:… · p. 13, Figure 10 · p. 14
  • Cross-category jailbreak probabilities remained low for Llama3.1-8B and Qwen2.5-7B, rarely exceeding target-category baseline compliance rates.This mirrors SAE-feature behavior and indicates limited cross-prompt generalization.Source: C. Additional Results:… · pp. 14–15
  • These results extend the SAE-feature findings and show random-steering vulnerabilities across model architectures, while remaining highly prompt-specific.The authors characterize the impact as systemic but challenging to monitor systematically.Source: C. Additional Results:… · p. 15

D. Supplementary API-Steered Responses for Benign SAE Feature

The appendix gives three additional Llama3.1-8B responses to harmful prompts while steering with the benign “brand identity” SAE feature.

D. Supplementary API-Steered Responses for Benign SAE Feature

  • The examples use the “brand identity” SAE feature with Goodfire API default steering hyperparameters on three JailbreakBench harmful prompts.The hyperparameters are those described in Sec. 4.3.Source: D. Supplementary API-Steered Responses… · p. 15
  • The first two responses disclaim responsibility before complying with the harmful requests.The figure presents these as examples of harmful content paired with an appearance of responsibility.Source: Figure 11 · p. 15
  • The third response uses fictional framing to justify harmful content.Source: Figure 11 · p. 15

E. On the Relationship Between Harmful SAE Features and Refusal Direction

The authors compare harmful SAE-feature vectors with a refusal direction and find negligible linear alignment, suggesting a more complex interference mechanism.

E. On the Relationship Between Harmful SAE Features and Refusal Direction

Loading 2509.22067v2…