PaperlayerView exact revision on arXiv

Source-linked AI summary

A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models

Hamid Kazemi, Atoosa Chegini, Maria Safi

arXiv:2605.08513v1cs.CLcs.AIcs.LG

TL;DR

Safety alignment is assumed to be distributed across model weights, but prior work had not isolated a single causally sufficient neuron. This paper uses causal interventions on individual MLP neurons across seven models and finds that suppressing refusal neurons bypasses safety while amplifying concept neurons induces harmful content.

  • Problem

    Prior work had not established whether a single neuron could be causally sufficient for bypassing safety alignment, beyond distributed directions or sets of neurons.

  • Method

    The paper performs systematic causal interventions on individual MLP neurons across seven Qwen3 and Llama-3.1 models spanning 1.7B–70B parameters, without training or prompt engineering.

  • Results

    91.7% average attack success rate was achieved by suppressing individual refusal neurons across seven models, while amplifying single concept neurons induced harmful content from benign prompts.

  • Takeaways & Limitations

    Safety-critical bottlenecks can exist at the level of individual neurons, with refusal and harmful-knowledge expression mediated by distinct neuron types.

  • Takeaways & Limitations

    Concept neurons were demonstrated only for suicide-related content, broader concept-neuron coverage remains future work, and optimal intervention strength requires empirical sweeping.

Abstract

from arXiv · show

Safety alignment in language models operates through two mechanistically distinct systems: refusal neurons that gate whether harmful knowledge is expressed, and concept neurons that encode the harmful knowledge itself. By targeting a single neuron in each system, we demonstrate both directions of failure -- bypassing safety on explicit harmful requests via suppression, and inducing harmful content from innocent prompts via amplification -- across seven models spanning two families and 1.7B to 70B parameters, without any training or prompt engineering. Our findings suggest that safety alignment is not robustly distributed across model weights but is mediated by individual neurons that are each causally sufficient to gate refusal behavior -- suppressing any one of the identified refusal neurons bypasses safety alignment across diverse harmful requests.

1 Introduction

The introduction challenges the view that safety is diffusely distributed across model weights, arguing instead that individual MLP neurons can form safety-critical bottlenecks. Across seven Qwen3 and Llama-3.1 models spanning 1.7B–70B parameters, suppressing a single refusal neuron bypasses safety alignment with 91.7% average JailbreakBench attack success, without training or prompt engineering.

  • Motivation: Safety alignment is conventionally assumed to emerge from a broad reorganization of weights, making it robust to small local perturbations.Under this assumption, no single component should decisively control safety behavior.
  • Motivation: Knowledge in transformer models can concentrate in identifiable neuron coordinates because MLP gating makes individual post-activation dimensions semantically meaningful.This motivates investigating individual neurons as potential carriers of safety-relevant information.
  • Prior limitations: Prior safety studies identified relevant attention or MLP neuron sets and residual-stream directions, but did not isolate an individual causal unit.The introduction frames single-neuron intervention as a finer-grained test of safety mechanisms.
  • Contribution: Single-neuron intervention is sufficient to bypass safety alignment across seven Qwen3 and Llama-3.1 models spanning 1.7B to 70B parameters.The targeted unit is one MLP neuron among hundreds of thousands to over two million, depending on model scale.
  • Contribution: 91.7% average attack success rate on JailbreakBench is achieved by suppressing individual refusal neurons across seven models and two model families.The method requires only white-box access to model activations, with no training, fine-tuning, or prompt engineering.

2 Finding Refusal Neurons

The study identifies refusal neurons by ranking single MLP neurons using gradient–activation signals and tests interventions across seven instruction-tuned models. Pinning one neuron at one layer achieves attack success rates comparable to full refusal-direction ablation, while the anchor variant substantially reduces capability degradation.

  • Evaluation setup: Seven instruction-tuned models from Qwen3 and Llama-3.1 are evaluated, alongside four publicly available Qwen3 base checkpoints.The instruction-tuned models range from 1.7B to 70B parameters; base checkpoints test whether refusal neurons precede alignment training.
  • Neuron identification: Candidate refusal neurons are ranked using gradient–activation signals measured on harmful and harmless prompts, then filtered for stronger harmful-prompt activation.The method registers pre-down-projection MLP activations at monitored layers and selected post-instruction token positions.
  • Neuron identification: The top-5 candidates are empirically reranked because the highest local score does not always produce the highest attack success rate.The selected candidate is evaluated on held-out JailbreakBench without further tuning.
  • Results: 91.9% constant and 90.1% anchor ASR versus 91.6% for Arditi’s full-direction ablation under Llama-Guard-3-8B show that one neuron can match the distributed baseline.The comparison modifies one scalar activation at one layer rather than ablating an entire direction across every layer.
  • Results: −8.8% MMLU and −1.2% GSM8K capability changes for the constant intervention are reduced to −0.6% and −0.1% with the anchor variant.The anchor costs are comparable to Arditi et al.’s −0.3% MMLU and −0.5% GSM8K while maintaining on-par attack success rates.

3 Properties of Refusal Neurons

Refusal neurons exhibit token-level, harmfulness-sensitive activation patterns that largely predate alignment training and can serve as effective detectors. Their semantic profiles suggest multiple refusal gates, while MLP intermediate features support stronger single-feature interventions than residual-stream dimensions.

  • Token-level activation patterns: Refusal neurons fire on post-instruction tokens of harmful prompts, with some also activating on harmful-content tokens and others only at the post-instruction boundary.These token-level patterns differ across models, including Qwen3-32B, Llama-3.1-8B, and Qwen3-14B.
  • Pre-alignment presence: Refusal-neuron activations largely separate harmful from harmless prompts in base models, indicating that the safety-relevant signal is present before alignment training.The results support alignment training as modulating the downstream effect of a pre-existing signal by connecting neurons to refusal behavior.
  • Feature basis: 45% ASR is achieved by the best residual-stream feature on Llama-3.1-8B, compared with 39% on Qwen3-8B, showing weaker performance than single-neuron MLP targeting.The authors attribute this gap to semantically meaningful coordinates in MLP intermediate activations versus rotation-invariant residual-stream dimensions.
  • Detection capability: AUROC remains above 0.9 across all other tested models, with Qwen3-1.7B reaching 0.853, demonstrating that a single refusal-neuron activation can detect harmful prompts.The detector uses one scalar activation, such as the layer-11 activation of a 32-layer model, rather than a full dedicated-classifier forward pass.
  • Neuron semantics and multiplicity: Safe-pole activations respond coherently to restrictions, warnings, disclaimers, and regulatory language, while Qwen3-14B has two additional independently sufficient refusal neurons.The reported neurons include Qwen3-14B:16:15515 for explicit content and Qwen3-14B:14:10112 for rule circumvention versus enforcement, suggesting multiple refusal gates.

4 Suicide Neurons

Suicide neurons are individual MLP neurons whose amplification can induce suicide-related content from entirely benign prompts. Together with refusal neurons, they support a two-system view in which harmful knowledge and its expression are each localized enough for a single neuron to be causally sufficient.

  • Neuron identification: Individual MLP neurons with suicide-related top activations were identified in three Qwen3 models as candidate suicide neurons.Qwen3-1.7B:20:4256 and Qwen3-14B:32:9115 activate positively, whereas Qwen3-8B:26:4061 activates negatively on suicide-related content.
  • Causal intervention: Adding a constant m to each candidate neuron during prefill and generation tested whether the neurons were causally responsible for producing suicide-related content.The intervention was evaluated on 20 entirely benign prompts using mention, coherence, and prompt-relevance criteria.
  • Induced harmful content: At sufficiently high intervention strength, nearly all 20 benign prompts produced suicide-related content, with most responses remaining coherent and prompt-relevant.The model generally wove the concept naturally into its output rather than producing incoherent text.
  • Implications: Suicide neurons provide proof of concept that harmful knowledge can be localized to individual neurons, complementing refusal neurons in a two-system gate-and-substrate model.Whether concept neurons generalize to other harmful concepts and model families remains open.

5 Related Work

Prior work characterizes refusal as a distributed linear feature, neuron-level vulnerability, and mechanistically identifiable safety circuitry, while prompt-level jailbreaks operate through inputs. This work narrows refusal intervention to a single causally sufficient MLP neuron at one layer.

  • Refusal as a linear feature: Refusal can be mediated by a shared residual-stream direction, while distinct refusal categories occupy geometrically distinct directions; this work intervenes on one scalar MLP activation.Arditi et al. identify a single residual-stream direction across layers, and Joad et al. identify category-specific directions that share behavioral control.
  • Neuron-level attacks: ∼3% pruning degrades alignment, while under 0.6% targeted-layer pruning achieves 76.9% average ASR across 20+ open-weight LLMs; this work reduces intervention to one neuron.NeuroStrike transfers to fine-tuned, distilled, and multimodal variants.
  • Mechanistic interpretability of safety: ∼5% of MLP neurons causally account for over 90% of safety behavior, and this work localizes safety control to a single causally sufficient element.Related approaches identify refusal-upstream features with sparse autoencoders, while the cited evidence indicates relevant neurons can predate alignment.
  • Prompt-level jailbreaks: Adversarial suffixes, paired-LLM red-teaming, and tree-of-attacks jailbreak alignment through the input without modifying model internals.These methods represent prompt-level attacks rather than neuron-level interventions.

6 Conclusion and Limitations

The study identifies individual neurons as safety-critical bottlenecks: suppressing refusal neurons bypasses alignment, while amplifying concept neurons induces harmful content. The findings motivate robustness to single-neuron interventions but are limited to suicide-related concept neurons and require white-box access for misuse.

  • Conclusion: Suppressing a single refusal neuron bypasses safety alignment across diverse harmful requests, while amplifying a single concept neuron induces harmful content from benign prompts.These results demonstrate safety-critical bottlenecks at the level of individual neurons.
  • Limitations: Refusal neurons are already discriminative in base models, suggesting alignment modulates preexisting neurons rather than creating them de novo.Concept neurons were demonstrated only for suicide-related content.
  • Broader Impacts: The findings motivate alignment strategies robust to single-component interventions, although misuse requires white-box access to open-weight models.The authors state that open-weight models are already jailbreakable via fine-tuning and that the work does not substantially alter their risk profile.

A Additional Experimental Setup

The experiments use balanced harmful and harmless training prompts, a 191-behavior HarmBench validation set, and independent judges for attack success. Interventions are evaluated under fixed decoding and transfer settings, with silent pivots motivating semantic rather than refusal-phrase-based assessment.

  • Datasets: 128 harmful and 128 harmless prompts train neuron discovery, while validation uses 191 non-overlapping HarmBench behaviors.Harmful prompts come from AdvBench, MaliciousInstruct, and TDC2023; harmless prompts come from Alpaca.
  • Evaluation: Attack success requires both a coherent, non-refusing response under Claude and an unsafe flag from Llama-Guard-3-8B.Classical refusal-phrase detection is not used because it can misclassify silent pivots.
  • Neuron discovery: Neuron discovery searches the first two-thirds of layers using gradient–activation scores from one forward–backward pass per training prompt.Post-instruction tokens provide scoring locations, and model-family refusal phrases define the refusal log-odds targets.
  • Attack generation: Attack evaluations use greedy decoding at temperature = 0 with maximum lengths of 512 tokens, selecting interventions on 191 HarmBench prompts before transfer to JailbreakBench.The constant sweep spans 0 to 20 in steps of 2 and 20 to 200 in steps of 5; anchor scales use k ∈{1, 2}.

B Extended Related Work

Prior work characterizes refusal as a low-dimensional activation feature and attacks safety by intervening on neuron sets or prompts. This work differs by testing whether a single MLP neuron at one layer is sufficient to bypass alignment across seven models.

  • Refusal as a linear feature: Refusal has been characterized as a low-dimensional feature, including a single direction whose layerwise ablation mediates refusal behavior.Representation engineering identifies and steers concept directions, while later work distinguishes harmfulness and refusal directions in the residual stream.
  • Neuron-level attacks on safety alignment: Prior neuron-level attacks prune safety-critical neuron sets, with one method removing under 0.6% of targeted-layer neurons and achieving 76.9% average ASR.The reported attack transfers to distilled, fine-tuned, and multimodal variants.
  • Neuron-level attacks on safety alignment: This work intervenes on one MLP neuron with a scalar at one layer, asking whether the lower bound for safety-alignment attacks is one neuron.The intervention is evaluated across seven models spanning two families.
  • Safety neurons from a mechanistic interpretability perspective: Mechanistic studies locate safety neurons through activation contrasting and causal patching, reporting that patching roughly 5% restores over 90% of safety behavior.These studies also indicate that safety neurons preexist alignment in base models, with alignment training serving a later role.
  • Knowledge and skill localization: Related work finds transformer knowledge concentrated in identifiable components, including MLP key-value memories, factual-association neurons, task-skill neurons, and linguistic features.A parallel prompting literature attacks alignment through adversarial suffixes, automated red-teaming, and tree-of-attacks search without modifying model internals.

C Detection

Single-neuron activation detects harmful prompts on XSTest effectively across the evaluated models, with AUROC above 0.9 for six of seven models. The Llama-3.1-8B refusal neuron matches LlamaGuard 3 in accuracy and achieves higher recall using a single scalar activation.

  • Benchmark and setup: 450 prompts comprise XSTest’s detection benchmark, including 200 harmful and 250 safe prompts.Metrics compare multi-token single-neuron activation against LlamaGuard 3, an 8B dedicated safety classifier.
  • Detection performance: AUROC remains above 0.9 for six of seven models, with Qwen3-1.7B the lone exception at 0.853.Neuron metrics are computed at the optimal threshold.
  • Detection performance: The Llama-3.1-8B refusal neuron matches LlamaGuard 3 on accuracy at 90.2% and exceeds its recall, 0.950 vs. 0.834.This comes at the cost of lower precision for the refusal-neuron detector.
  • Detection method: Multi-token aggregation takes the minimum or maximum refusal-neuron activation across post-instruction tokens, depending on the activation sign for harmful prompts.The main-text detection results use this aggregation rather than a single token position.

D Refusal Neuron Activation Distributions

Across seven instruction-tuned models, top refusal neurons strongly activate for harmful prompts while remaining nearly silent for harmless ones, yielding clear distributional separation. The same discriminative signal appears in four Qwen3 base checkpoints before alignment training, suggesting it emerges during pretraining.

  • Instruction-tuned models: Top refusal neurons clearly separate harmful and harmless prompt activations across all seven instruction-tuned models.They activate strongly on harmful prompts and near-silently on harmless ones across both model families and all parameter scales.
  • Base checkpoints: Four Qwen3 base checkpoints already show clean refusal-neuron separation between harmful and harmless prompts before alignment training.This indicates the safety-relevant signal is encoded before fine-tuning and suggests these neurons emerge during pretraining.

E Residual-Stream vs. MLP Features

Residual-stream features are substantially weaker single-feature attack targets than MLP neurons in Llama-3.1-8B and Qwen3-8B. Their lower attack success rates reflect poorer separation between harmful and harmless inputs.

  • Residual-stream vs. MLP features: 45% ASR on Llama-3.1-8B and 39% on Qwen3-8B were the best residual-stream candidate results, below corresponding MLP candidates.These figures apply to the top-5 residual-stream candidates under optimal-constant and cumulative evaluation.
  • Residual-stream vs. MLP features: Residual-stream features showed poor harmful/harmless separation, including near-complete activation overlap for Qwen3-8B.The comparison attributes their weaker attack performance to this limited separation relative to MLP neurons.

F Refusal Neuron Activations

Refusal neurons act as selective gates for harmful content, activating strongly on harmful prompts while remaining near-silent on harmless ones. Across Qwen3 instruct and base models, alignment training changes when these neurons fire rather than whether they respond to harmful content.

  • Activation timing: Across four Qwen3 model sizes, instruct-model refusal neurons activate strongly at the assistant turn boundary on the same harmful prompt.This activation pattern is consistent with a gating role in refusal behavior.
  • Activation timing: In base models, the same refusal neuron is already active on harmful content tokens such as “bomb?”, indicating that alignment training rewires when the neuron fires rather than whether it responds.In instruct models, activation is concentrated at the assistant turn boundary.
  • Harmful/harmless discrimination: Refusal neurons fire selectively on harmful prompts, concentrating activation at harmful content tokens and the assistant turn boundary while remaining near-silent on structurally similar harmless prompts.The contrast appears across four models from both model families.

G Geometric Alignment with the Refusal Direction

The gradient–activation method identifies refusal neurons independently of the refusal direction, yet the identified neurons show highly significant geometric alignment across all seven models. The two methods converge on the exact same neuron in two models, supporting these neurons as structurally encoded mediators of safety behavior.

  • Independent identification: The gradient–activation method uses a contrastive harmful-versus-harmless prompt signal, whereas the geometric method relies purely on weight geometry relative to the refusal direction.The refusal direction is computed from the difference between mean residual activations on harmful and harmless prompts at a chosen layer.
  • Geometric significance: Across all seven models, the observed top cosine exceeds the random expected maximum, with union-bound p-values from 3 × 10−8 to 1 × 10−47.The null treats Wdown rows as uniformly random directions, and the expected maximum over dff cosines provides the baseline.
  • Method convergence: In two of seven models, the gradient–activation and refusal-direction methods converge on the exact same neuron: L13:F3270 in Qwen3-1.7B and L25:F10201 in Llama-3.1-70B.Each convergent neuron lies in the refusal-direction layer and ranks first among that layer’s dff neurons by cosine similarity with the refusal direction.
  • Interpretation: Convergence between these independent strategies provides additional evidence that the identified neurons are structurally encoded mediators of safety behavior rather than artifacts of either method alone.The evidence comes from agreement between gradient–activation identification and weight-geometry alignment.

H Attack Success Rate and Capability Preservation

The constant, anchor, and Arditi interventions achieve similarly high attack success rates across seven models and evaluation settings. Capability costs are larger for constant interventions on MMLU than on GSM8K, while anchor-based interventions substantially reduce the MMLU cost.

  • Attack Success Rate: All three interventions achieve high attack success rates across all seven models, with average metrics within roughly one point of one another.Table 8 evaluates HarmBench-191 and JailbreakBench using both an LLM judge and LlamaGuard.
  • Attack Success Rate: 91.9% and 90.1% versus 91.6%: on JailbreakBench under LlamaGuard, constant and anchor are on par with Arditi.The same intervention selections are used on JailbreakBench without re-tuning.
  • Capability Preservation: −8.8%: constant interventions cause substantial average MMLU degradation, with Llama-3.1-70B showing the largest drop of −18.2%.Table 9 reports MMLU and GSM8K accuracy changes relative to the unmodified baseline.
  • Capability Preservation: −1.2%: average GSM8K degradation under constant interventions is smaller than the corresponding MMLU degradation.The passage attributes GSM8K’s greater resilience to mathematical reasoning depending less on disrupted safety-relevant circuitry.
  • Capability Preservation: Anchor-based interventions substantially reduce the capability cost compared with the constant method, particularly for MMLU.The supplied passage reports this reduction but does not provide the complete anchor-based numerical results.

I What Refusal Neurons Respond To

Refusal neurons show a consistent directional contrast across models: their harmful pole responds to explicit content, while their safe pole responds to restriction and warning meta-language. This pattern indicates that the neurons distinguish harmful material from language describing restrictions around it.

  • What Refusal Neurons Respond To: Inspecting corpus examples at each neuron’s harmful and safe poles reveals two patterns that recur across models.The analysis examines texts that drive activation toward each pole, including harmful-prompt and opposite-direction examples.
  • What Refusal Neurons Respond To: The harmful pole responds to explicit content, especially sexual and pornographic material in most models.Examples include escort services, adult video descriptions, sex product listings, and content moderation disclaimers.
  • What Refusal Neurons Respond To: The safe pole responds to restriction and warning meta-language rather than safe content itself.Qwen3-14B:17:2154’s safe pole fires on cautionary notices, content warnings, and liability disclaimers, while its harmful pole fires on actual adult and explicitly regulated material.

J Intervention Examples

Single-neuron interventions demonstrate both directions of safety failure: suppressing a refusal neuron enables harmful completions, while amplifying feature neurons induces refusal or suicide-themed content from benign creative-writing prompts.

  • Refusal bypass: Suppressing a single refusal neuron bypasses safety on an explicitly harmful prompt, changing refusal into a harmful completion.This intervention uses Qwen3-32B neuron F9168-L46.
  • Induced refusal: Amplifying a single feature neuron causes spurious refusal on an innocuous creative-writing prompt, citing fabricated platform guidelines against romantic content.In Qwen3-14B, neuron F2154-L17 is amplified with multiplier α = −60; without intervention, the model responds fluently and on task.
  • Induced harmful content: 500× amplification of a single suicide neuron induces suicide-themed content unprompted while preserving the requested poem’s structure and ocean imagery.The intervention targets Qwen3-1.7B neuron F4256-L20 on an innocent ocean-writing prompt.

K Prompts

The paper evaluates safety bypasses by judging refusal, coherence, and harmful helpfulness with a structured LLM prompt. It separately evaluates concept steering by testing whether amplified features introduce target concepts, including their prominence, coherence, and relevance to benign prompts.

  • K.1 Safety Bypass Evaluation Prompt (LLM Judge): The safety-bypass judge assesses whether a response refuses, remains semantically coherent, and provides actionable information for the harmful request.These are returned as Boolean judgments alongside confidence scores and a brief explanation.
  • K.1 Safety Bypass Evaluation Prompt (LLM Judge): A response counts as refusal when it declines, warns, redirects to safe alternatives, or says it cannot or will not help.Compliance, step-by-step instructions, facilitation details, or engagement without clear refusal count as non-refusal.
  • K.1 Safety Bypass Evaluation Prompt (LLM Judge): Coherence is judged from grammatical correctness, logical structure, and semantic sense, while gibberish, severely broken grammar, or incomprehensibility indicate incoherence.The criterion applies whether the response refuses or complies.
  • K.1 Safety Bypass Evaluation Prompt (LLM Judge): Helpful information requires specific actionable steps, technical details, procedures, or practical knowledge that advances the harmful goal.Generic warnings, ethical lectures, gibberish, vague or incorrect information, and coherent text that does not advance the goal are unhelpful; helpfulness can remain true despite a technical refusal.
  • K.2 Concept Steering Evaluation Prompt: The concept-steering judge tests whether amplifying an internal feature makes a response introduce or steer toward a specified target concept despite a benign original prompt.The evaluation receives the original prompt, model response, and target concept, then returns a JSON assessment.
  • K.2 Concept Steering Evaluation Prompt: Concept steering is evaluated using concept mention, prominence, linguistic coherence, prompt relevance, confidence, and an explanation.Prominence ranges from none through subtle and moderate to dominant.
  • K.2 Concept Steering Evaluation Prompt: The prompt distinguishes linguistic coherence from prompt relevance: a well-structured response about the wrong topic can be coherent but irrelevant.The specified example is a story about suicide responding to a request to write a love story.

L Feature Profiles

The profiled neurons respond to diverse safety-relevant concepts, including harmful content, privacy and credential risks, rule circumvention, and crime. Several show polarity-specific contrasts, with opposite activations separating related harmful, benign, procedural, or contextual content.

  • Safety-relevant concepts: Feature profiles span explicit sexual content, suicide-related content, politically sensitive material, crime, privacy, credential access, and rule circumvention.Examples include sexual descriptions, suicide-related content, hate speech, phishing, password databases, piracy, and jailbreaking.
  • Activation polarity: Opposite activation polarities distinguish sexual content from animal and pet content, reflecting ambiguity in the token “pussy.”Bottom activations include cats, dogs, horses, pandas, and pet care, while the contrasting profile activates on suicide-related or explicit sexual content.
  • Safety-relevant concepts: One neuron activates broadly across sexual, antisemitic, extremist, threatening, and anti-religious material, suggesting a general safety-relevant signal rather than one concept.Its top activations include explicit sexual material, Charlottesville chanting, a Quranic fighting verse, villain monologues, gun threats, and anti-religious rhetoric.
  • Activation polarity: Several neurons separate harmful or intrusive contexts from benign, contextual, or protective counterparts, including privacy policies, anonymity, investigative security, and publicly reported controversies.Other profiles contrast credential access with identity protection, explicit content with warning language, and direct threats or misuse with investigative or preventive framing.
  • Non-coherent profiles: Some profiled neurons lack a coherent conceptual theme, while others respond to benign content unrelated to safety-relevant material.The profiles include neurons whose activations are explicitly described as non-coherent or benign and unrelated to safety-relevant content.
Loading 2605.08513v1…