PaperlayerView exact revision on arXiv

Source-linked AI summary

Refusal in Language Models Is Mediated by a Single Direction

Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, Neel Nanda

arXiv:2406.11717v3cs.LGcs.AIcs.CL

TL;DR

The paper asks how refusal is implemented in safety-fine-tuned chat models and studies this mechanism across open-source systems. It identifies and intervenes on a single refusal direction, then uses that understanding to construct a white-box jailbreak and examine adversarial suffixes. The results show a broadly shared refusal mechanism that can be disabled through a simple weight modification while retaining model coherence, raising concerns about current alignment robustness.

  • Problem

    Safety-fine-tuned chat models are designed to help with benign requests and refuse harmful ones, but the mechanisms underlying this widespread refusal behavior remain poorly understood.

  • Method

    The paper identifies a refusal direction from harmful–harmless activation differences, intervenes on it, develops a rank-one weight-orthogonalization jailbreak, and examines adversarial-suffix effects on its propagation.

  • Results

    Refusal is consistently mediated by a single direction across 13 open-source chat models up to 72B parameters, and modifying model weights can disable refusal while retaining model coherence.

  • Takeaways & Limitations

    Model-internals interpretability can reveal practical vulnerabilities, and the simplicity of this refusal mechanism raises concerns about the robustness of current alignment techniques.

  • Takeaways & Limitations

    The findings may not generalize to untested models, especially larger proprietary or future systems, and the refusal-direction extraction relies on heuristics.

Abstract

from arXiv · show

Conversational large language models are fine-tuned for both instruction-following and safety, resulting in models that obey benign requests but refuse harmful ones. While this refusal behavior is widespread across chat models, its underlying mechanisms remain poorly understood. In this work, we show that refusal is mediated by a one-dimensional subspace, across 13 popular open-source chat models up to 72B parameters in size. Specifically, for each model, we find a single direction such that erasing this direction from the model's residual stream activations prevents it from refusing harmful instructions, while adding this direction elicits refusal on even harmless instructions. Leveraging this insight, we propose a novel white-box jailbreak method that surgically disables refusal with minimal effect on other capabilities. Finally, we mechanistically analyze how adversarial suffixes suppress propagation of the refusal-mediating direction. Our findings underscore the brittleness of current safety fine-tuning methods. More broadly, our work showcases how an understanding of model internals can be leveraged to develop practical methods for controlling model behavior.

1 Introduction

Safety fine-tuning aims to make chat models helpful on innocuous requests while refusing harmful ones, but the mechanisms underlying refusal remain difficult to understand. This work identifies a single refusal direction across many open-source models and uses that insight to expose practical vulnerabilities.

  • Safety fine-tuning targets both helpful responses to innocuous requests and refusals of harmful or inappropriate ones.
  • Refusal is mediated by a one-dimensional subspace across 13 open-source chat models with up to 72B parameters.
  • A difference-in-means direction extracted from harmful and harmless instruction pairs can be intervened on to bypass harmful-prompt refusal or induce refusal on harmless prompts.The intervention targets residual-stream activations.
  • A rank-one weight edit disables refusal with minimal impact on other capabilities, illustrating a practical white-box jailbreak based on model-internals analysis.
  • The findings raise concerns about the robustness of current alignment techniques because refusal can be easily circumvented in the white-box setting.

2 Methodology

The methodology extracts refusal-related activation directions from harmful and harmless instructions, selects one vector per model, and tests interventions that add or erase it. Evaluations use multiple model families, datasets, and complementary refusal and safety measures.

  • Datasets: Training data comprise harmful instructions from four datasets and harmless instructions from ALPACA, with non-overlapping 128-sample training and 32-sample validation splits.
  • Models: The study evaluates safety-fine-tuned models spanning 1.8 to 72 billion parameters, including preference-optimized and fine-tuned alignment types.
  • Extracting a refusal direction: For every layer and post-instruction position, difference-in-means vectors subtract mean harmless activations from mean harmful activations.The vector’s direction represents how the means differ, while its magnitude measures their distance.
  • Extracting a refusal direction: The method selects one candidate vector using validation performance on refusal bypass, refusal induction, and minimal behavioral change.The selected vector is denoted r, with unit-norm version r̂.
  • Interventions: Activation addition adds a difference-in-means vector at one layer across all token positions to shift harmless inputs toward harmful mean activations.Directional ablation instead removes the component along the unit direction from every residual activation across all layers and positions.
  • Evaluation: Completions are generated greedily for up to 512 tokens and scored separately for refusal and harmfulness.Refusal uses substring matching, while harmfulness is classified by Meta Llama Guard 2; substring matching does not assess coherence or harmful content.

3 Refusal is mediated by a single direction

Across the evaluated models, one extracted direction is reported as both necessary and sufficient for refusal: erasing it reduces refusal on harmful prompts, while adding it induces refusal on harmless prompts.

  • A single difference-in-means vector is extracted for each model and tested as the refusal-mediating direction.
  • Ablating the direction across all layers and token positions reduces refusal and elicits unsafe completions on 100 harmful JAILBREAKBENCH instructions.Without intervention, chat models refuse nearly all harmful requests.
  • Adding the direction to residual-stream activations causes models to refuse even harmless requests, whereas they typically do not refuse without intervention.The activation-addition experiment uses 100 randomly sampled harmless ALPACA instructions.

4 A white-box jailbreak via weight orthogonalization

The section introduces weight orthogonalization, which removes the refusal direction directly from model weights to bypass refusal while preserving most general capabilities. On HARMBENCH, ORTHO performs competitively with other jailbreaks, but results vary with system prompts and TRUTHFULQA accuracy declines.

  • 4 A white-box jailbreak via weight orthogonalization: Weight orthogonalization modifies residual-stream-writing weights so the model no longer represents the refusal direction.The method orthogonalizes embedding, positional embedding, attention-output, MLP-output matrices, and output biases with respect to the refusal direction.
  • 4 A white-box jailbreak via weight orthogonalization: The weight modification is equivalent to inference-time directional ablation and therefore has the same refusal-bypassing performance.Both interventions prevent the model from writing the refusal direction into residual-stream activations.
  • 4.2 Comparison to other jailbreaks: ORTHO performs well against other general jailbreaks and is comparable to prompt-specific GCG across the QWEN family.HARMBENCH evaluates attack success rate on 159 standard behaviors using the provided classifier.
  • 4.2 Comparison to other jailbreaks: System prompts substantially reduce LLAMA-2 attack success but have little effect on QWEN attack success.For LLAMA-2 7B, ASR is 22.6% with versus 79.9% without the system prompt; for QWEN 7B, it is 79.2% versus 74.8%.
  • 4.3 Measuring model coherence: Orthogonalized models generally match baseline performance on MMLU, ARC, and GSM8K, while TRUTHFULQA accuracy consistently decreases.Except for QWEN 7B and YI 34B, evaluation metrics lie within 99% confidence intervals of original performance; the authors also describe directional ablation as more surgical than activation addition.

5 Mechanistic analysis of adversarial suffixes

The section studies how adversarial suffixes bypass refusal in QWEN 1.8B CHAT by comparing them with random suffixes. The suffix suppresses the refusal direction and redirects key attention heads from harmful instructions toward the suffix.

  • 5 Mechanistic analysis of adversarial suffixes: Adversarial suffixes are evaluated against unedited prompts and equal-length random suffixes on 128 harmful instructions.The instructions are sampled from JAILBREAKBENCH and the HARMBENCH test set.
  • 5.1 Adversarial suffixes suppress the refusal-mediating direction: Appending the adversarial suffix heavily suppresses the refusal direction, producing an expression pattern resembling harmless instructions.The comparison uses cosine similarity between last-token residual-stream activations and the refusal direction.
  • 5.2 Adversarial suffixes hijack attention heads: The analysis selects the eight attention heads with the largest direct feature-attribution contributions to the refusal direction.Each component’s contribution is measured by projecting its output onto the refusal direction.
  • 5.2 Adversarial suffixes hijack attention heads: The adversarial suffix significantly suppresses these heads’ direct contributions to the refusal direction relative to no-suffix and random-suffix conditions.This result is shown in Figure 6(a).
  • 5.2 Adversarial suffixes hijack attention heads: The suffix shifts these heads’ attention from the harmful instruction region to the appended suffix region.Figure 6(b) characterizes this shift as attention hijacking.

6 Related work

Related work frames this study within research on safety representations, feature directions, safety fine-tuning, and input-based jailbreaks. The paper distinguishes its weight-based intervention from approaches requiring malicious completion data or prompt modification.

  • Understanding refusal in language models: Prior work has linked safety behavior to critical neurons, ranks, and contrastive representations of harmfulness that may differ from refusal.These studies examine how safety mechanisms and harmfulness representations are encoded in model internals.
  • Features as directions: Contrastive inputs and activation-space feature vectors are established tools for extracting and manipulating represented concepts.Related studies apply these directions to behaviors including truth, sentiment, language, topic, and other features.
  • Undoing safety fine-tuning: Malicious fine-tuning can undo safety guardrails with minimal capability degradation, but it requires harmful instructions and completions.The paper contrasts this data requirement with its own use of harmful instructions alone.
  • Jailbreaks: Input-based jailbreaks include social-engineering prompts and optimized adversarial suffixes that alter model behavior without changing weights.The paper’s method instead requires access to model weights and does not modify the input prompt.

7 Discussion

The discussion presents refusal as a single-direction phenomenon across diverse open-source chat models and uses that finding to motivate a simple weight-based jailbreak. It also bounds the claims by noting uncertain generalization, unclear direction semantics, restricted suffix analysis, and measurement limits.

  • 7 Discussion: Refusal behavior is consistently mediated by a single direction across a diverse set of open-source chat models.The authors use this interpretation to motivate direct weight modification that disables refusal while retaining model coherence.
  • 7 Discussion: The ease of bypassing refusal through a simple weight modification raises concerns about the robustness of current alignment techniques.The discussion characterizes this as a practical use of model-internals interpretability.
  • Limitations: The study may not generalize to untested models, especially larger-scale, proprietary, or future systems.This is identified as a limitation despite evaluation across a broad range of open-source models.
  • Limitations: The refusal-direction extraction uses heuristics, and the authors frame the study as an existence proof rather than an optimized extraction methodology.Methodological improvements are left for future work.
  • Limitations: The adversarial-suffix analysis is restricted to one model and one adversarial example, so it is not comprehensive.The authors also note that chat-model coherence is difficult to measure and that the metrics used are flawed in various ways.
  • Limitations: The semantic meaning of the refusal directions remains unclear and may correspond to harm, danger, or no straightforward concept.“Refusal direction” is used as a functional description rather than a settled semantic interpretation.
  • Ethical considerations: The method lowers the barrier to jailbreaking open-source weights, including a 70B model with less than $5 of compute.The authors state that it requires neither gradient-based optimization nor a dataset of harmful completions.
  • Ethical considerations: The paper contributes to concerns that current safety mechanisms are fragile and insufficient to prevent misuse of open-source LLMs.This conclusion is situated within the rapid advancement of state-of-the-art model capabilities.

B Refusal metric: an efficient proxy for measuring refusal

The paper defines an efficient refusal proxy from next-token probabilities, then uses it to select residual-stream directions that bypass or induce refusal while limiting behavioral changes. It also evaluates refusal with complementary refusal and safety metrics because either metric alone can misclassify ambiguous completions.

  • Refusal metric: Greedy completion assessment is accurate but expensive, motivating a no-generation proxy based on next-token probabilities.Models tend to begin refusals with characteristic phrases, enabling assessment at the completion’s first token.
  • Refusal metric: The refusal probability sums next-token probabilities over model-specific refusal tokens R, and its log-odds form yields the refusal metric.The metric is used to filter harmful and harmless prompts and to evaluate interventions efficiently.
  • Direction selection: Candidate directions are scored by bypass, induction, and KL metrics, then filtered for refusal induction, limited harmless-prompt change, and distance from unembedding directions.The selected direction minimizes bypass_score subject to these conditions.
  • Direction selection: Directional ablation suppresses refusal, activation addition induces refusal, and the selected direction’s source position, layer, and metrics are reported per model.The search excludes directions too close to unembedding directions and takes about an hour for 72-billion-parameter models.
  • Evaluation metrics: Refusal_score detects refusal substrings, while safety_score uses Llama Guard 2 to assess harmfulness, because refusals and harmful content can diverge.Examples include refusal_score=0 with safety_score=1 and refusal_score=1 with safety_score=0.

F.2 Effect of system prompts on evaluation

System prompts substantially affect orthogonalized LLAMA-2 attack success but have little effect on QWEN, indicating model-family-dependent sensitivity. Across diverse prompts, LLAMA-2 shows much greater variability than QWEN.

  • Effect of system prompts on evaluation: Including the default system prompt substantially reduces HARMBENCH ASR for LLAMA-2 models but has minimal effect on QWEN models.The evaluation compares attack success with and without each family’s default system prompt.
  • Effect of system prompts on evaluation: LLAMA-2’s default system prompt contains explicit safety guidelines, whereas QWEN’s default prompt is minimal and lacks specific safety or ethics directives.The prompts are shown in Figures 17 and 18.
  • Effect of system prompts on evaluation: Applying the LLAMA-2 system prompt to QWEN models causes no significant ASR change, suggesting that prompt impact may be model-dependent.This test separates prompt wording from model-family responsiveness.
  • Effect of system prompts on evaluation: LLAMA-2 7B’s ASR varies across 12 system prompts with mean 30.0% and standard deviation 23.3%.The evaluation uses the same orthogonalized LLAMA-2 7B model across diverse prompts.
  • Effect of system prompts on evaluation: QWEN 7B’s ASR across the same 12 prompts has mean 76.7% and standard deviation 5.9%, indicating more consistent performance than LLAMA-2 7B.The paper reports this comparison as evidence of different system-level responsiveness.
  • Effect of system prompts on evaluation: The evaluation distinguishes no system prompt from an empty-string system prompt when testing model sensitivity.Figure 19 defines both conditions separately.

G.1 Language model evaluation

The paper evaluates whether orthogonalization preserves general capabilities and examines adversarial suffixes. General benchmark performance usually changes little, but TRUTHFULQA declines consistently, while suffix analysis is restricted to one model and one suffix.

  • General performance: Orthogonalization causes less than 1% average performance drop on general benchmarks, with significant drops mainly for QWEN 7B and YI 34B.The paper describes most baseline differences as indistinguishable from noise.
  • General performance: TRUTHFULQA performance is consistently worse for orthogonalized models than for unmodified models.The benchmark covers truthful and accurate responses on topics including misinformation, stereotypes, and conspiracies.
  • Loss evaluation: Cross-entropy is measured on THE PILE and ALPACA, with chat templates applied for ALPACA and completion-token loss used there.The study also evaluates generated on-distribution ALPACA completions to address out-of-distribution loss concerns.
  • Method comparisons: The study compares directional ablation with activation addition and fine-tuning as alternative weight-modification methodologies.The orthogonalized model is labeled “Ablation,” while activation addition is labeled “Act Add.”
  • Adversarial suffixes: One 20-token suffix was selected from 100 GCG-generated suffixes because it performed well across a wide range of harmful prompts.The suffix is shown in Figure 21.
  • Adversarial suffixes: Suffix-mechanism analysis is restricted to QWEN 1.8B CHAT and a single suffix because universal and transferable suffixes were difficult to find.The authors identify broader suffix and model coverage as desirable but unavailable for this analysis.

I.1 Comparison to activation addition

Activation addition can bypass refusal by shifting activations away from the refusal direction, but it perturbs harmless activations more than directional ablation. Related interventions and analyses indicate that the refusal representation is reused in base models and can be modified without broadly changing harmless behavior.

  • Activation addition: Activation addition in the negative refusal direction reduces refusal and elicits unsafe completions, approximately matching directional ablation.The intervention shifts harmful activations toward harmless activations with respect to refusal-direction projection.
  • Activation addition: Activation addition causes increased loss over harmless data because it pushes harmless activations far out of distribution.Directional ablation also shifts harmful activations toward harmless activations without shifting harmless activations as far off distribution.
  • Comparison with fine-tuning: Fine-tuning disables refusal effectively, but constructing high-quality harmful instruction-completion pairs requires non-trivial effort.Directional ablation requires only harmful instructions, without harmful completions.
  • Base and chat models: The refusal direction is expressed similarly in corresponding base and chat models, suggesting safety fine-tuning repurposes an existing representation.Base models show high similarity on harmful prompts and low similarity on harmless prompts.
  • Behavioral effects: Orthogonalization preserves harmless behavior while retaining standard refusal strings, indicating a conceptual intervention rather than simple suppression of refusal tokens.Completions on harmless prompts often appear indistinguishable from those of the unmodified model, while meta-refusal explanations may remain incoherent.

N Compute statement

The experiments used NVIDIA RTX A6000 GPUs, with smaller models run on one GPU and larger models distributed across four GPUs. Direction generation took approximately 5 minutes for models up to 14B parameters and approximately 1 hour for larger models.

  • Hardware: Models with ≤14B parameters ran on a single 48GB NVIDIA RTX A6000 GPU, while larger models used four GPUs in parallel.Most experiments used a cluster of eight RTX A6000 GPUs.
  • Runtime: Direction generation took approximately 5 minutes for models ≤14B parameters and approximately 1 hour for larger models.
Loading 2406.11717v3…