Source-linked AI summary
The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence
Tom Wollschläger, Jannes Elstner, Simon Geisler, Vincent Cohen-Addad, Stephan Günnemann, Johannes Gasteiger
TL;DR
LLM alignment can be bypassed by adversarial attacks, but the internal mechanisms underlying refusal remain poorly understood beyond a proposed single refusal direction. The paper uses gradient-based representation engineering and introduces representational independence to identify refusal structures. It finds multi-dimensional refusal cones and multiple independently contributing refusal directions, while noting scope limitations in target diversity and undiscovered refusal regions.
Problem
Adversarial attacks can bypass LLM alignment, while the mechanisms underlying refusal and jailbreaks remain poorly understood beyond a proposed single refusal direction.
Method
The paper uses gradient-based representation engineering and representational independence to identify and analyze refusal directions and their intervention relationships.
Results
The study identifies multi-dimensional polyhedral refusal cones and multiple refusal directions that remain independent under intervention.
Takeaways & Limitations
Refusal behavior is governed by complex spatial structures and multiple distinct mechanisms rather than a single refusal direction.
Takeaways & Limitations
The computed refusal directions use the same targets, and the method cannot rule out other undiscovered refusal-mediating regions.
Abstract
from arXiv · showhide
The safety alignment of large language models (LLMs) can be circumvented through adversarially crafted inputs, yet the mechanisms by which these attacks bypass safety barriers remain poorly understood. Prior work suggests that a single refusal direction in the model's activation space determines whether an LLM refuses a request. In this study, we propose a novel gradient-based approach to representation engineering and use it to identify refusal directions. Contrary to prior work, we uncover multiple independent directions and even multi-dimensional concept cones that mediate refusal. Moreover, we show that orthogonality alone does not imply independence under intervention, motivating the notion of representational independence that accounts for both linear and non-linear effects. Using this framework, we identify mechanistically independent refusal directions. We show that refusal mechanisms in LLMs are governed by complex spatial structures and identify functionally independent directions, confirming that multiple distinct mechanisms drive refusal behavior. Our gradient-based approach uncovers these mechanisms and can further serve as a foundation for future work on understanding LLMs.
1. Introduction
LLMs remain vulnerable to adversarial attacks that bypass alignment safeguards, while the mechanisms behind refusal and jailbreaks remain poorly understood. This work introduces gradient-based representation engineering and argues that refusal is mediated by multi-dimensional cones and multiple independent directions rather than one linear direction.
- Adversarial attacks can bypass LLM alignment mechanisms and induce harmful outputs.
- Prior work suggests that harmful-query refusals may be mediated by a single refusal direction in activation space.
- The paper introduces gradient-based representation engineering to investigate refusal mechanisms in LLMs.
- Multi-dimensional cones can contain infinitely many refusal directions, challenging the assumption of a single linear refusal direction.
- Representational independence identifies directions that remain mutually unaffected under intervention, including linear and non-linear dependencies.
2. Background
The background describes how LLM representations are formed and how refusal directions can be extracted and manipulated. Prior approaches use harmful-versus-harmless prompt representations, while directional ablation can reduce refusal probability.
- A language model embeds input tokens, processes them through layers, and converts final logits into token probabilities.
- Extracting refusal directions: Difference-in-means extracts a directional feature by comparing residual-stream representations for harmful and harmless prompts.
- Adversarial steering attacks: With white-box access, suppressing activations in the harmfulness direction reduces the model’s probability of refusal on harmful queries.
- Adversarial steering attacks: Directional ablation removes the component of an activation aligned with a unit direction, while activation subtraction subtracts a scaled direction.
3. Related Work
Related work spans adversarial prompt attacks, mechanistic interpretability, refusal mechanisms, and over-refusal benchmarks. These studies motivate analyzing how internal representations mediate refusal and how safety systems reject benign queries.
- Adversarial attacks for LLMs: Adversarial-prompt research includes handcrafted strategies, search and genetic algorithms, discrete token optimization, and gradient-based perturbations.
- Interpretability of LLMs: Interpretability research studies activation features and circuits that implement specific functions or behaviors.
- Understanding Refusal Mechanisms: Refusal-mechanism studies examine safety-critical neurons, harmfulness directions, sparse-autoencoder features, and related internal representations.
- Over-refusal: Over-refusal research measures false refusals on benign queries and investigates lexical shortcuts and over-defense in safety filters.
4. Gradient–based Refusal Directions
The paper introduces Refusal Direction Optimization, a gradient-based method for identifying directions that control refusal while preserving desired behavior. Evaluations show that RDO improves refusal manipulation, precision, and safety–over-refusal trade-offs relative to DIM.
- Method: RDO identifies refusal directions by optimizing refusal behavior with gradients rather than relying solely on paired-prompt activation differences.The method encodes monotonic scaling, surgical ablation, and retention objectives into loss functions optimized over a direction vector.
- Method: RDO uses activation addition to induce refusal on harmless inputs and directional ablation to elicit answers to previously refused harmful prompts.A KL-based retain loss constrains ablation effects on harmless instructions.
- Evaluation: On average, RDO outperforms DIM for jailbreaking with refusal-direction subtraction and induces refusal more effectively when the direction is added to harmless inputs.RDO remains competitive with DIM under directional ablation despite not being explicitly optimized for subtraction-based attacks.
- Evaluation: RDO reduces the average TruthfulQA error from directional ablation by 40% and generally has less impact on ARC, GSM8K, and MMLU scores than DIM.These results support more precise manipulation of refusal behavior with fewer benchmark side effects.
- Evaluation: On XSTest, RDO consistently achieves higher refusal rates for harmful inputs while maintaining or reducing benign over-refusal compared with DIM.This yields a uniformly better safety–over-refusal trade-off at any given level of benign over-refusal.
- Flexibility: Tuning the retain-loss weight allows RDO to balance attack success and side effects, while choosing the target layer freely.The authors report that many Qwen2.5 DIM directions have excessive side effects, whereas RDO limits these effects through retain loss.
5. Multi-dimensional Refusal Cones
The paper investigates whether refusal is governed by one direction or a more complex geometry, finding multi-dimensional cones whose directions mediate refusal. It evaluates these cones through optimized orthonormal bases and sampled directional interventions across models and dimensions.
- Cone construction: Refusal concept cones are polyhedral regions whose nonzero directions all satisfy the same refusal properties.The paper treats every direction in such a region as expressing the same refusal concept.
- Cone construction: Nonnegative cone coefficients ensure directions consistently strengthen refusal, while orthonormal bases prevent co-linear directions.Arbitrarily large scaling remains constrained because excessive activation changes can cause model degeneration.
- Optimization: Refusal cones are optimized with projected gradient descent using losses on Monte Carlo samples and basis vectors, followed by Gram–Schmidt orthogonalization.Basis-vector losses improve stability and the lower bounds of attack success rate because basis vectors define cone boundaries.
- Empirical findings: Refusal-mediating cones reach dimensions up to five across all tested models, indicating refusal is encoded in multi-dimensional regions rather than a single direction.Cone effectiveness is measured by sampling 256 vectors and computing directional-ablation attack success rates.
- Empirical findings: Larger Qwen 2.5 models appear to support higher-dimensional cones, although lower-bound cone performance degrades as dimensionality increases.The degradation means more sampled directions have low attack success rates; larger residual streams may permit more distinct directions.
- Empirical findings: Sampling multiple directions improves low-sample attack success over temperature sampling, while gains increase with dimensionality and plateau at four dimensions.The low-sample advantage suggests distinct directions capture complementary refusal aspects; the plateau may reflect model support limits.
6. Mechanistic Understanding of Directions
The paper tests whether refusal directions represent independent mechanisms rather than merely orthogonal vectors. It finds multiple representationally independent directions whose interventions produce additive refusal-related effects and can be accessed through adversarial inputs.
- Representational Independence: Orthogonality alone does not establish independence because ablating one refusal direction can indirectly reduce the other’s later-layer representation.The analysis tracks cosine similarities across layers to capture linear, nonlinear, and cumulative interactions.
- Representational Independence: Representational independence requires that ablating one direction does not change how strongly the other is expressed in model activations.The framework evaluates this condition across layers using cosine similarity.
- Independent Directions: A direction can be representationally independent from DIM while retaining the refusal properties targeted during training.For Gemma 2 2B, the RepInd and DIM directions barely affect each other’s representations under directional ablation.
- Independent Directions: Multiple directions remain effective when each is constrained to be independent of DIM and all previously identified RepInd directions.Attack success rate declines as more directions are sought, possibly because optimization becomes harder or independently contributing directions are limited.
- Compositional Effects: Ablating top-k RepInd directions increases attack success rate monotonically, surpassing the DIM baseline for k≥4 before diminishing returns appear.Ablating multiple DIM directions from different layers does not improve performance, supporting distinct mechanisms for the RepInd directions.
- Input Manipulation: Adversarial GCG suffixes reduce RepInd–1 representation and jailbreak the model 36% of the time, similar to RepInd–1’s attack success rate.This connects independently identified refusal directions to representations altered through token-space input manipulation.
7. Limitations
The study identifies important limitations in how refusal directions and refusal-mediating regions are discovered.
- Limitations: Optimizing all refusal directions on the same targets may limit their ability to capture fully distinct mechanisms.The authors suggest diverse targets or judge-based reinforcement learning as possible extensions.
- Limitations: Although higher-dimensional refusal cones are established, other undiscovered refusal-mediating regions may still exist.
8. Conclusion
The paper uses gradient-based representation engineering to analyze refusal directions and argues that refusal is organized by complex, high-dimensional structures. It further identifies multiple representationally independent directions corresponding to distinct mechanisms.
- Conclusion: Gradient-based representation engineering identifies and analyzes refusal directions with fewer side effects and greater modeling flexibility.The approach can be extended to other concepts by changing the optimization targets.
- Conclusion: Refusal behavior is better characterized by high-dimensional polyhedral cones in activation space than by a single linear direction.
- Conclusion: Multiple refusal directions within representationally independent spaces correspond to distinct underlying mechanisms.
Impact Statement
The impact statement frames the work as defensive research on refusal mechanisms, while the experiments use established harmful and harmless instruction benchmarks and chat models. The evaluation includes implementation details, activation interventions, and cross-dataset refusal results.
- Impact Statement: Understanding refusal mechanisms could aid stronger attacks, but the stated aim is to support more robust and reliable safety systems.The work focuses on open-source models requiring white-box access, limiting its primary applicability to defensive improvement rather than deployed-system compromise.
- Dataset and Models: The experimental dataset combines harmful and harmless instructions from established benchmarks, including 1,184 harmful instructions sampled from remaining SALADBENCH sources.The Multilingual and ToxicChat sources were excluded as unsuitable harmful-instruction sources.
- Dataset and Models: Experiments exclusively use chat models with each model’s default chat template, while omitting IT and INSTRUCT from model names.
- Implementation and Evaluation: The method selects refusal directions after convergence from the final 20 training steps and balances attack success against side effects through retain-loss weighting.For RepInd directions, multiple candidate vectors and repeated selection address high variance between runs.
- Results: Activation addition with RDO directions induces refusal across all evaluated models for DIM and RDO, with RDO slightly outperforming DIM for most models.
- Results: Directional-ablation attack success results on STRONGREJECT, SORRY-BENCH, and JAILBREAKBENCH confirm that the findings transfer across datasets.
B.3. Benchmarks
The benchmarks compare RDO and DIM directions by their side effects on general capabilities and jailbreaking effectiveness. RDO generally preserves benchmark performance better while maintaining strong attack effectiveness.
- RDO directions have significantly lower side effects than DIM across most evaluated models and benchmarks.
- The evaluation compares jailbreaking effectiveness with changes in general capability benchmark scores under directional ablation.
- The ablation studies assess the importance of the addition, ablation, and retain losses in the RDO algorithm.
C.1. Addition and Ablation Loss
Both addition and ablation losses are needed for strong attack performance, while retain-loss weighting trades off refusal-direction effectiveness against benchmark preservation. RDO configurations robustly improve on DIM across these metrics.
- Addition and Ablation Loss: Above 80% ASR is maintained across λabl and λadd balances from 0.2–0.8 for both directional ablation and activation subtraction.Using only one loss component significantly reduces performance.
- Retain Loss: Higher λretain improves benchmark preservation with minimal ASR loss up to λretain = 4, after which ASR drops significantly.
- Retain Loss: The retain loss can impede learning effective refusal directions when weighted too strongly.
- Comparison with DIM: Multiple hyperparameter configurations Pareto-dominate the DIM baseline across jailbreaking effectiveness and benchmark preservation.This indicates that the improvements are not limited to one specific hyperparameter choice.
- Comparison with DIM: For Qwen 2.5 7B, only one evaluated token-layer direction is effective both for inducing refusal and maintaining low side effects under the DIM selection procedure.
E. Extended Results for Refusal Cones
Extended experiments show that refusal cones support refusal induction across model families, while higher-dimensional cones improve attack success only up to a model-dependent plateau. The results also examine interactions between directions and the safety–over-refusal trade-off.
- Cone Dimensionality: For Gemma 2 2B, the lower bounds of refusal-cone performance degrade significantly at dimension 5.
- Refusal Induction: Most directions in high-dimensional Qwen 2.5 refusal cones remain effective for inducing refusal, which is easier than disabling refusal through directional ablation.
- Cone Dimensionality: ASR increases with Gemma 2 2B cone dimensionality but plateaus at four dimensions, consistent with complementary mechanisms captured beyond a single direction.The authors hypothesize that the plateau reflects difficulty encoding higher-dimensional refusal cones.
- Direction Interactions: Ablating orthogonal refusal directions substantially influences the DIM direction across additional models and the larger SORRY-BENCH dataset.
- Safety and Over-Refusal: RDO provides better or equivalent safety–over-refusal trade-offs than DIM across activation-addition strengths on harmful and benign instruction datasets.Safety is measured on SORRY-BENCH, while over-refusal is measured on XSTEST.