PaperlayerView exact revision on arXiv

Source-linked AI summary

A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models

Hamid Kazemi, Atoosa Chegini, Maria Safi

arXiv:2605.08513v1cs.CLcs.AIcs.LG

TL;DR

Prior work had not established whether one individual neuron could be causally sufficient for safety. This paper intervenes on individual MLP neurons across seven models and reports 91.7% average JailbreakBench attack success after suppressing refusal neurons, alongside harmful-content induction by amplifying concept neurons.

  • Problem

    Prior work had not shown whether an individual neuron could be causally sufficient for safety, leaving the individual-neuron scale of safety mechanisms unestablished.

    Source: §1 · p. 2

  • Method

    The authors causally intervene on individual MLP neurons across seven models, suppressing refusal neurons and amplifying concept neurons without training or prompt engineering.

    Source: §1 · p. 2

  • Results

    91.7% average attack success rate on JailbreakBench across seven models followed suppression of individual refusal neurons; amplifying suicide-related concept neurons induced related content from innocuous prompts.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The findings motivate alignment strategies that are more robust to single-component interventions.

    Source: §6 · p. 10

    Concept-neuron evidence is limited to suicide-related content, with a broader survey left for future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

Safety alignment in language models operates through two mechanistically distinct systems: refusal neurons that gate whether harmful knowledge is expressed, and concept neurons that encode the harmful knowledge itself. By targeting a single neuron in each system, we demonstrate both directions of failure -- bypassing safety on explicit harmful requests via suppression, and inducing harmful content from innocent prompts via amplification -- across seven models spanning two families and 1.7B to 70B parameters, without any training or prompt engineering. Our findings suggest that safety alignment is not robustly distributed across model weights but is mediated by individual neurons that are each causally sufficient to gate refusal behavior -- suppressing any one of the identified refusal neurons bypasses safety alignment across diverse harmful requests.

1 Introduction

The paper challenges the view that safety is diffusely distributed across model weights, finding that individual neurons can gate refusal or encode harmful knowledge. Across seven models, suppressing a refusal neuron bypasses safety, while amplifying a concept neuron can induce harmful content from innocent prompts.

1 Introduction

  • Unlike prior work on distributed directions or neuron sets, this paper identifies a single neuron as causally sufficient for bypassing safety alignment.Prior studies include Arditi et al. [2024] and Joad et al. [2026] on residual-stream directions, and Zhao et al. [2025] and Zhao et al. [2026] on neuron sets.Source: §1 · pp. 1–2
  • Amplifying a single suicide neuron injects suicide-related content into otherwise innocent prompts across multiple model scales, demonstrating a second, concept-encoding system.The paper presents this as a proof of concept; surveying the full space of harmful concepts is left for future work.Source: §1 · p. 2
  • The identified refusal neurons are present in base models before alignment training, suggesting alignment modulates these neurons rather than creating them de novo.Source: §1 · p. 2
  • A single refusal neuron's activation detects harmful prompts with AUROC comparable to Llama-Guard-3-8B on XSTest.Source: §1 · p. 2
  • 91.7% average attack success rate on JailbreakBench was achieved by suppressing one refusal neuron across seven Qwen3 and Llama-3.1 models, without training, fine-tuning, or prompt engineering.The models span 1.7B to 70B parameters; the method requires only white-box access to model activations.Source: §1 · p. 2, Figure 2 · p. 3

2 Finding Refusal Neurons

The section tests whether targeting one MLP neuron at one layer can suppress refusal, then evaluates single-neuron interventions against an all-layer refusal-direction ablation. Across seven models, the anchor variant preserves capability better than constant intervention while maintaining comparable attack success rates.

2.1 Background

  • Against Arditi et al. [2024]’s all-layer residual-stream direction ablation, the section tests whether one MLP neuron at one layer can suppress refusal.Arditi et al. show that ablating a single residual-stream direction at every layer suppresses refusal across models.Source: §2.1 · p. 3

2.2 Models, Data, and Evaluation

  • Evaluation spans seven instruction-tuned Qwen3 and Llama-3.1 models, with base checkpoints for four Qwen3 models used to test for pre-alignment refusal neurons.Training uses 128 harmful and 128 harmless prompts; HarmBench’s 191-prompt subset is used for validation, and disjoint JailbreakBench’s 100 behaviors for held-out testing. Attack success is assessed by an LLM judge and Llama-Guard-3-8B.Source: §2.2 · p. 3

2.3 Feature Selection and Reranking

  • Candidate refusal neurons are ranked using gradient and activation signals, then the top five are empirically reranked by validation attack success rate because their initial ranking can diverge.The constant intervention pins the selected neuron at every token position; the anchor variant scales the intervention to the prompt’s natural activation, addressing coherence costs from large fixed assignments.Source: §2.3 · pp. 4–5

2.4 Attack Effectiveness and Capability Preservation

  • On JailbreakBench, single-neuron constant and anchor interventions achieved attack success rates comparable to Arditi et al. [2024] under both judges across seven models.Under Llama-Guard-3-8B, averages were 91.9% (constant), 90.1% (anchor), and 91.6% (Arditi); under the LLM judge, they were 91.7%, 90.4%, and 93.1%, respectively.Source: §2.4 · pp. 5–6, Figure 5 · p. 6
  • The anchor variant largely eliminated constant intervention’s capability cost while maintaining on-par attack success rates.Average changes were −8.8% MMLU and −1.2% GSM8K for constant intervention, versus −0.6% MMLU and −0.1% GSM8K for anchor; Arditi et al. [2024] had −0.3% MMLU and −0.5% GSM8K.Source: §2.4 · p. 6

3 Properties of Refusal Neurons

Refusal-neuron activations distinguish harmful from harmless prompts, and the safety-relevant signal is present before alignment training. Their MLP activations also support harmful-prompt detection and reveal varied content themes.

3.1 Token-Level Interpretability

  • On harmful prompts, refusal neurons fire at post-instruction tokens, while harmless prompts remain silent there even when they contain a harmful word in a benign context.Some models also fire on harmful content tokens, whereas Qwen3-14B fires exclusively at the post-instruction boundary; the pattern points to semantic intent rather than the word itself.Source: §3.1 · p. 6

3.2 Refusal Neurons Emerge During Pretraining

  • Before alignment training, refusal-neuron activations largely separated held-out harmful from harmless prompts, indicating that safety-relevant signals were already present.The shift from harmful prompt tokens in base models to post-instruction tokens in instruct models suggests alignment modulates when these neurons fire rather than creating their safety-relevant content de novo.Source: §3.2 · p. 7, Figure 6 · p. 7

3.3 Why MLP Neurons, Not Residual-Stream Features?

  • 45% ASR on Llama-3.1-8B and 39% on Qwen3-8B were the best residual-stream results under the optimal constant, below their MLP counterparts.Residual-stream activations overlapped heavily for Qwen3-8B and partially for Llama-3.1-8B, unlike the clean harmful/harmless separation of MLP neurons.Source: Figure 7 · p. 7, §3.3 · p. 7

3.4 Single-Neuron Harmful Prompt Detection

  • 0.950 recall versus 0.834 for LlamaGuard-3-8B, but 0.848 precision versus 0.949, was achieved by the Llama-3.1-8B refusal neuron on XSTest.Its AUROC and accuracy were comparable to the classifier’s; across other models, AUROC exceeded 0.9 except for Qwen3-1.7B (0.853).Source: §3.4 · pp. 7–8, Table 2 · p. 8

3.5 What Refusal Neurons Respond To

  • Refusal-neuron poles respond to distinct themes, including restricted material, warning language, and coordinated criminal agency.Qwen3-14B:17:2154’s harmful pole responds to adult material and its safe pole to warnings and disclaimers; Meta-Llama-3.1-70B-Instruct:25:10201 responds to coordinated criminal agency on its harmful pole and victim- or witness-perspective incidents on its safe pole. Qwen3-14B also has two additional independently sufficient refusal neurons: Qwen3-14B:16:15515 and Qwen3-14B:14:10112.Source: §3.5 · p. 8

4 Suicide Neurons

Amplifying an identified individual MLP neuron can make Qwen3 produce suicide-related content from benign prompts, supporting a two-system picture in which refusal neurons gate expression and concept neurons provide a substrate for harmful knowledge.

4 Suicide Neurons

  • The authors identified one suicide-related MLP neuron in each of three Qwen3 models, with positive activation in Qwen3-1.7B and Qwen3-14B but negative activation in Qwen3-8B.The neurons were identified by inspecting The Pile examples that maximally activate individual neurons [Gao et al., 2020].Source: §4 · p. 8, Figure 8 · p. 9
  • They added a constant to each candidate neuron's activation at every token during prefill and generation, then evaluated outputs on 20 benign prompts using concept mention, coherence, and prompt relevance.For each intervention strength, they tracked cumulative counts meeting M, M+C, and M+C+P.Source: §4 · p. 9
  • Increasing amplification caused suicide-related content to appear in responses to benign prompts; at high strength, nearly all 20 mentioned it and most remained coherent and prompt-relevant.Figure 10 illustrates a suicide-themed story generated from an innocent creative-writing prompt.Source: §4 · p. 9, Figure 9 · p. 9, Figure 10 · p. 10
  • The findings frame refusal neurons as a gate and concept neurons as a substrate, with a single neuron in each system causally sufficient, while generalization to other harmful concepts and model families remains open.Source: §4 · pp. 9–10

5 Related Work

Prior work studies refusal through residual-stream directions and safety through neuron subsets, while this work localizes suppression to one MLP activation or neuron. Other related approaches either interpret or preserve safety mechanisms, or attack through prompts.

Refusal as a linear feature

  • Arditi et al. [2024] and Joad et al. [2026] frame refusal as shared residual-stream directions, whereas this work achieves comparable suppression through one scalar MLP activation at one layer.Joad et al. describe geometrically distinct directions for different refusal categories that act as a shared behavioral control knob.Source: §5 · p. 10

Neuron-level attacks

  • Compared with pruning ∼3% of safety-critical neurons (Wei et al. [2024]) or NeuroStrike’s under 0.6% for 76.9% average ASR across 20+ open-weight LLMs, this work intervenes on one neuron at one layer.NeuroStrike also transfers to fine-tuned, distilled, and multimodal variants.Source: §5 · p. 10

Mechanistic interpretability of safety

  • Chen et al. [2024] attribute over 90% of safety behavior to ∼5% of MLP neurons, consistent with this work’s localization to one causally sufficient element (Section 3.2).Chen et al. also provide evidence that these neurons exist in base models before alignment.Source: §5 · p. 10
  • Lee et al. [2025] identify refusal-upstream features via sparse autoencoders, while SafeNeuron [Wang et al., 2026] freezes safety neurons during fine-tuning to distribute safety behavior more broadly.These are complementary interpretive and constructive approaches, respectively.Source: §5 · p. 10

Prompt-level jailbreaks

  • Adversarial suffixes, paired-LLM red-teaming, and tree-of-attacks [Mehrotra et al., 2024] target alignment through the input without changing model internals.Source: §5 · p. 10

6 Conclusion and Limitations

The findings show that individual refusal and concept neurons can respectively gate harmful responses and induce harmful content. The authors note limits in the concept-neuron evidence and multiplier selection, and that misuse requires white-box access.

6 Conclusion and Limitations

  • Suppressing one refusal neuron bypasses safety alignment across diverse harmful requests, while amplifying one concept neuron induces harmful content from benign prompts.The identified refusal neurons are already discriminative in base models, suggesting alignment modulates preexisting neurons rather than creating them de novo.Source: §6 · p. 10
  • Concept-neuron evidence covers only suicide-related content; broader surveys and more principled selection of the empirically swept multiplier m∗ remain open.The authors identify broader concept-neuron investigation and principled multiplier selection as future work.Source: §6 · p. 10
  • The findings motivate alignment strategies more robust to single-component interventions, while the authors acknowledge potential misuse to bypass safety in open-weight models.Such misuse requires white-box access; the authors say open-weight models are already jailbreakable via fine-tuning and do not believe this work substantially alters their risk profile.Source: §6 · p. 10

A Additional Experimental Setup

The appendix specifies the datasets, neuron-selection procedure, intervention settings, and evaluation protocols used in the experiments. It also explains why attack success is judged from full responses rather than refusal phrases alone.

Models and chat templates

  • Experiments use bfloat16 models, a single NVIDIA B200 GPU, and model-family chat templates without an explicitly passed system prompt.Qwen3 runs with enable_thinking=False; Llama-3.1’s tokenizer automatically injects a default system preamble, and base checkpoints use their official Qwen3-*-Base chat template.Source: A Additional Experimental Setup · p. 13, Table 3 · p. 13

Datasets

  • The study uses matched training and validation sets and a disjoint held-out test set for comparing its method with Arditi et al. [2024].Training uses 128 harmful and 128 harmless prompts; validation is a 191-prompt HarmBench subset, and JailbreakBench’s 100 behaviors form the held-out test set. Visualization prompts are 100 held-out Alpaca prompts.Source: A Additional Experimental Setup · p. 13

Evaluation

  • Attack success requires a non-refusing, coherent response under the LLM judge, with Llama-Guard-3-8B as a secondary unsafe-content verifier.Classical refusal-phrase detection is unreliable because models may silently pivot to benign content without an explicit refusal; Figure 12 illustrates this pattern.Source: A Additional Experimental Setup · pp. 13, 15, Figure 12 · p. 15

Neuron discovery

Attack generation

  • Attack evaluations use greedy decoding, and intervention scales selected on HarmBench transfer to JailbreakBench without retuning.Generation is capped at 512 tokens; constant interventions sweep from 0 to 20 by 2 and 20 to 200 by 5 opposite the neuron’s harmful-prompt activation, while anchor scales are k ∈{1, 2}.Source: A Additional Experimental Setup · p. 14

Capability evaluation

  • MMLU uses next-token probabilities over answer choices, while GSM8K uses greedy generation and answer extraction; baselines use the same evaluation harness.Both capability evaluations use the attack-generation chat template.Source: A Additional Experimental Setup · p. 14

Suicide neuron evaluation

  • Benign-prompt suicide-neuron evaluation uses additive activation hooks, and a Claude-based judge checks target-concept mention, coherence, and relevance.The 20 benign prompts are evaluated with temperature 0 and a 500-token maximum; unlike refusal-neuron direct assignment, the intervention adds m to the neuron activation.Source: A Additional Experimental Setup · p. 14, Figure 11 · p. 14

Implementation

B Extended Related Work

Prior work locates refusal in activation directions or groups of safety neurons, and knowledge in identifiable model components. This work positions its single-neuron interventions as a finer-grained way to alter refusal and harmful-content expression.

Refusal as a linear feature

  • Refusal research describes harmfulness and refusal as distinct residual-stream directions, with later work testing whether a single refusal direction suffices [Zheng et al., 2024; Arditi et al., 2024; Joad et al., 2026].Zou et al. [2023a] introduced representation engineering, while Siu et al. [2025] and Joad et al. [2026] extended the refusal-direction framework.Source: B Extended Related Work · p. 15

Neuron-level attacks on safety alignment

  • NeuroStrike [Wu et al., 2025] reports 76.9% average ASR after pruning under 0.6% of neurons, whereas this work modifies one neuron; prompt-side jailbreaks instead alter inputs.The related work contrasts this single-neuron, single-layer intervention with attacks on neuron sets and input-side methods.Source: B Extended Related Work · pp. 15–16

Safety neurons from a mechanistic interpretability perspective

  • Mechanistic studies locate safety in pre-existing neurons, upstream SAE features, or neurons used to redistribute safety during fine-tuning [Chen et al., 2024; Lee et al., 2025; Wang et al., 2026].Chen et al. [2024] report that patching roughly 5% of neurons restores over 90% of safety behavior; this work reports that a single neuron can be causally sufficient.Source: B Extended Related Work · p. 16

Knowledge and skill localization

  • Prior work localizes factual associations, task skills, and linguistic or dangerous-content features, contextualizing suicide neurons as a safety-relevant instance of localized knowledge [Geva et al., 2021; Dai et al., 2022; Meng et al., 2022; Wang et al., 2022; Gurnee et al., 2023; Templeton, 2024].The paper describes its suicide neurons as MLP units whose amplification induces suicide-related content from innocent prompts.Source: B Extended Related Work · p. 16

C Detection

Single refusal-neuron activations detect harmful prompts with AUROC above 0.9 for six of seven models on XSTest, and the Llama-3.1-8B neuron matches LlamaGuard 3 accuracy while achieving higher recall at lower precision.

C Detection

  • AUROC exceeds 0.9 for six of seven models; Qwen3-1.7B is the exception at 0.853.Table 6 evaluates harmful-prompt detection on XSTest, with 200 harmful and 250 safe prompts.Source: Table 6 · p. 16, C Detection · p. 16
  • The Llama-3.1-8B refusal neuron matches LlamaGuard 3 accuracy (90.2%) and exceeds its recall (0.950 vs. 0.834), with lower precision.This uses a single scalar activation at layer 11 rather than a full forward pass through an 8B classifier.Source: Table 6 · p. 16, C Detection · p. 16
  • Detection uses minimum or maximum refusal-neuron activation aggregated across post-instruction tokens, depending on the sign of harmful-prompt activation.Figure 14 contrasts single-token results with multi-token aggregation near the assistant boundary.Source: C Detection · p. 17, Figure 14 · p. 17

D Refusal Neuron Activation Distributions

Across seven instruction-tuned models, top refusal neurons activate strongly on harmful prompts and near-silently on harmless prompts. Four Qwen3 base models show this separation before alignment training.

D Refusal Neuron Activation Distributions

  • Across all seven instruction-tuned models, top refusal neurons activate strongly on harmful prompts and near-silently on harmless prompts, clearly separating the distributions.The pattern holds across both model families and all parameter scales.Source: D Refusal Neuron Activation… · p. 18, Figure 15 · p. 18
  • In four Qwen3 base models, refusal neurons already separate harmful from harmless prompts before alignment training, suggesting the signal emerges during pretraining.For some models, the refusal signal appears at the beginning of the response.Source: D Refusal Neuron Activation… · p. 18, Figure 16 · p. 18

E Residual-Stream vs. MLP Features

For Llama-3.1-8B and Qwen3-8B, individual residual-stream features are weaker attack targets than MLP neurons, with lower ASR and poorer harmful/harmless separation.

E Residual-Stream vs. MLP Features

  • Residual-stream features show overlapping harmful and harmless activations, while MLP neurons separate them cleanly because SwiGLU makes neuron coordinates semantically meaningful.The rotation-invariant residual stream instead distributes safety-relevant information across directions rather than individual dimensions.Source: E Residual-Stream vs. MLP Features · p. 19, Figure 17 · p. 19
  • Residual-stream attacks reach at most 45% ASR on Llama-3.1-8B and 39% on Qwen3-8B, below their MLP counterparts (Figure 17).These are the best top-5 residual-stream candidates under optimal-constant and cumulative evaluation.Source: Figure 17 · p. 19, E Residual-Stream vs. MLP Features · p. 19

F Refusal Neuron Activations

Refusal-neuron activations distinguish harmful from harmless prompts, while Qwen3 base and instruct models differ in when the neuron fires. In instruct models it activates at the assistant turn boundary; in base models it is already active on harmful prompt tokens.

F Refusal Neuron Activations

  • Across four Qwen3 sizes, instruct models activate the refusal neuron strongly at the assistant turn boundary, while base models activate it on harmful prompt tokens.The comparison indicates that alignment training rewires when the neuron fires rather than whether it fires.Source: F Refusal Neuron Activations · p. 20
  • The refusal neuron fires selectively on harmful prompts and remains near-silent on structurally similar harmless prompts, across four models from both families.Figure 19 contrasts harmful prompts with harmless prompts using the same suffix patterns.Source: Figure 19 · p. 20, F Refusal Neuron Activations · p. 20
  • On harmful prompts, refusal-neuron activation concentrates at harmful content tokens and the assistant turn boundary.Figure 19 presents per-token activations normalized per model.Source: F Refusal Neuron Activations · p. 20, Figure 19 · p. 20

G Geometric Alignment with the Refusal Direction

The refusal neurons identified by the gradient–activation method align with the refusal direction more strongly than expected under a random-directions baseline. In two models, this method and an independent cosine-similarity method identify the same neuron.

Geometric Alignment with the Refusal Direction

  • Across all seven models, |s1| exceeded the random expected maximum, with union-bound p-values from 3 × 10−8 to 1 × 10−47.The cosine comparison ranks MLP down-projection neurons at the refusal-direction layer; the random baseline assumes uniformly distributed weight rows.Source: G Geometric Alignment with the Refusal… · pp. 20–21, Table 7 · p. 21
  • The gradient–activation and cosine methods selected the same neuron in Qwen3-1.7B (L13:F3270) and Llama-3.1-70B (L25:F10201).In both models, the shared neuron is at the refusal-direction layer and ranks first there by cosine similarity to the refusal direction.Source: G Geometric Alignment with the Refusal… · p. 21, Table 7 · p. 21
  • In the remaining five models, the methods selected neurons at different layers because the cosine search is restricted to the refusal-direction layer, while the gradient method searches across all layers.Source: G Geometric Alignment with the Refusal… · p. 21
  • The authors interpret convergence of the independent contrastive-signal and weight-geometry methods on one neuron as evidence that these neurons are structurally encoded mediators of safety behavior.Source: G Geometric Alignment with the Refusal… · p. 21

H Attack Success Rate and Capability Preservation

Across seven models, constant, anchor, and Arditi refusal-direction ablation [Arditi et al., 2024] achieved similarly high attack success rates on HarmBench-191 and held-out JailbreakBench. Anchor-based intervention preserved MMLU and GSM8K capability substantially better than constant scaling.

Per-model ASR

  • 91.9% ASR for constant and 90.1% for anchor were on par with Arditi’s 91.6% on JailbreakBench under LlamaGuard.All three methods achieved high ASR across seven models, with averages within roughly one point on every metric; HarmBench-191-selected multipliers transferred to held-out JailbreakBench without re-tuning. Arditi refers to Arditi refusal-direction ablation [Arditi et al., 2024].Source: H Attack Success Rate and Capability… · p. 22, Table 8 · p. 22

Capability preservation

  • −8.8% average MMLU degradation under constant scaling fell to −0.6% with anchor-based scaling, while GSM8K degradation fell from −1.2% to −0.1%.Llama-3.1-70B had the largest constant-method MMLU drop (−18.2%); GSM8K was more resilient, which the passage suggests may reflect less dependence on disrupted safety circuitry.Source: H Attack Success Rate and Capability… · p. 22, Table 9 · p. 22
  • −0.6% MMLU degradation under constant scaling and +0.1% under anchor scaling made Qwen3-1.7B minimally affected by either method.The passage suggests Qwen3-1.7B’s refusal neuron is more cleanly separable from general capability circuits than in other models.Source: H Attack Success Rate and Capability… · p. 22

I What Refusal Neurons Respond To

Inspection of the strongest corpus activations in both directions shows that refusal neurons often respond to explicit harmful content at the harmful pole, while coherent safe poles respond to restriction-related language rather than safe content itself.

I What Refusal Neurons Respond To

  • Harmful poles most often activate on explicit sexual and pornographic material, including adult content and sexual assault.Qwen3-8B and Qwen3-32B show similar patterns; Meta-Llama-3.1-8B-Instruct responds more broadly to age-restricted and legally regulated content across harm categories.Source: I What Refusal Neurons Respond To · p. 23
  • Safe-pole responses vary: Qwen3-4B:14:5590 activates on ordinary lifestyle writing, while Qwen3-8B:14:7924 has no coherent safe-pole theme.The lifestyle examples include travel recommendations, food blogs, seasonal events, and game releases.Source: I What Refusal Neurons Respond To · p. 23
  • Some safe poles respond to warnings and restriction language, distinguishing notices about harmful content from the harmful material itself.Qwen3-14B:17:2154 responds to warnings and disclaimers, while Qwen3-1.7B:13:3270 responds to information-restriction language; their harmful poles activate on the adult, regulated, or restricted material itself.Source: I What Refusal Neurons Respond To · p. 23
  • Qwen3-14B:14:10112 distinguishes rule circumvention at its harmful pole from institutional enforcement at its safe pole.Its harmful pole responds to piracy, DRM bypassing, trafficking, jailbreaking, and illegal markets, while its safe pole responds to prosecutions, court proceedings, and investigations.Source: I What Refusal Neurons Respond To · p. 23

J Intervention Examples

The intervention examples show that changing one neuron can alter model behavior in both directions: suppressing a refusal neuron bypasses safety on a harmful prompt, while amplifying individual neurons can induce refusal or harmful content on benign prompts.

J Intervention Examples

  • Suppressing one refusal neuron in Qwen3-32B bypasses refusal on an explicitly harmful prompt and elicits a harmful completion.Without intervention, the model refuses.Source: Figure 20 · p. 24
  • Amplifying single neurons on benign creative-writing prompts induces either a refusal citing fabricated guidelines or unprompted suicide-themed content.The refusal example uses Qwen3-14B, F2154-L17, with α = −60; the suicide-content example uses Qwen3-1.7B, F4256-L20, with multiplier 500, while preserving the prompt’s poetic structure and ocean imagery.Source: Figure 21 · p. 24, Figure 22 · p. 24

K Prompts

The appendix provides two LLM-judge prompts: one evaluates harmful-request responses for refusal, coherence, and helpfulness, while the other assesses whether feature amplification steers responses toward a target concept. Both prompts define evaluation criteria and return structured judgments.

K.1 Safety Bypass Evaluation Prompt (LLM Judge)

  • The safety-bypass judge evaluates refusal, semantic coherence, and actionable helpfulness, returning binary judgments, confidence scores, and an explanation in JSON.Source: §K.1 · p. 25
  • A response counts as a refusal when it declines, warns, redirects to safe alternatives, or says it cannot help; attempts to comply count as non-refusals.Source: §K.1 · p. 25
  • Coherence is judged by grammatical correctness, logical structure, and semantic sense; gibberish, severely broken grammar, or incomprehensible responses are incoherent.Source: §K.1 · p. 25
  • Helpful information includes actionable steps, technical details, or practical knowledge advancing the harmful goal, even when the response technically refuses.Source: §K.1 · pp. 25–26
  • Generic warnings, gibberish, vague or incorrect information, and coherent text that does not advance the harmful goal count as unhelpful.Source: §K.1 · p. 26

K.2 Concept Steering Evaluation Prompt

  • The concept-steering judge checks whether amplification introduces or steers toward a target concept, including indirect mentions, and rates its prominence from none to dominant.Source: §K.2 · pp. 26–27
  • The concept-steering evaluation separately records linguistic coherence and prompt relevance, so a coherent response about the wrong topic can still be irrelevant.Source: §K.2 · p. 27

Related papers

Loading 2605.08513v1…