Source-linked AI summary
There Is More to Refusal in Large Language Models than a Single Direction
Faaiz Joad, Majd Hawasly, Sabri Boughorbel, Nadir Durrani, Husrev Taha Sencar
TL;DR
Prior work proposed a single activation-space direction for refusal, leaving unclear whether broader non-compliance behaviors share that mechanism. This paper analyzes eleven refusal categories with activation-space steering and sparse autoencoders, finding geometrically distinct directions but nearly identical refusal trade-offs. The main difference among directions is refusal style rather than refusal frequency.
Problem
Prior work centered on a single refusal direction for harmful requests, leaving open whether qualitatively different refusal and non-compliance behaviors use distinct mechanisms.
Method
The paper studies eleven refusal and non-compliance categories using activation-space interventions and sparse autoencoders to analyze directions and latent features.
Results
Refusal behaviors correspond to geometrically distinct directions, yet steering along them produces similar refusal and over-refusal trade-offs across inputs.
Takeaways & Limitations
Different refusal directions act as a shared linear control knob for whether the model refuses, while shared and specialized latent features shape how it refuses.
Takeaways & Limitations
The study evaluates two instruction-tuned models and does not establish whether the observed mechanisms persist in larger, base, or differently aligned models.
Abstract
from arXiv · showhide
Prior work argues that refusal in large language models is mediated by a single activation-space direction, enabling effective steering and ablation. We show that this account is incomplete. Across eleven categories of refusal and non-compliance, including safety, incomplete or unsupported requests, anthropomorphization, and over-refusal, we find that these refusal behaviors correspond to geometrically distinct directions in activation space. Yet despite this diversity, linear steering along any refusal-related direction produces nearly identical refusal to over-refusal trade-offs, acting as a shared one-dimensional control knob. The primary effect of different directions is not whether the model refuses, but how it refuses.
1 Introduction
Refusal in LLMs extends beyond safety and appears in diverse styles, challenging the view that one activation-space direction explains all non-compliance. The paper studies whether distinct refusal directions differ geometrically, mechanistically, and behaviorally, finding shared linear control alongside style-specific structure.
- Motivation: Prior work identified a single activation-space direction whose ablation suppresses refusal and whose amplification induces it, but mainly for harmful requests.This mechanism motivates testing whether refusal generalizes beyond safety-related behavior.
- Motivation: Models refuse for varied reasons and in different styles, creating a tension between an apparently simple mechanism and diverse observed behavior.The open question is whether qualitatively different refusal and non-compliance behaviors rely on distinct mechanisms.
- Research scope: The study covers incomplete, unsupported, indeterminate, humanizing, professional-advice, contextual, safety, and over-refusal behaviors beyond conventional safety cases.These categories broaden refusal into a wider class of non-compliant behaviors observed in real-world interactions.
- Main findings: Linear steering along geometrically distinct refusal directions produces similar refusal efficacy, increasing refusal of both harmful and benign prompts as strength rises.The directions therefore act like a shared control knob for refusal frequency across datasets.
- Main findings: Different directions primarily change refusal style, ranging from policy-based refusals to clarification, missing-context responses, and softer deflection.SAE analysis links this pattern to shared reusable features combined with specialized features associated with particular styles.
2 Methods
The methods construct refusal directions from contrasts between non-compliant and benign prompts, intervene on residual-stream activations, and use sparse autoencoders to identify and test latent refusal features. Steering and ablation provide causal tests of refusal behavior in activation and latent spaces.
- Activation-space analysis: A refusal direction is the difference between mean residual-stream activations for non-compliant and benign prompts at a fixed token position.One direction is computed for each evaluation split and used in steering and ablation experiments.
- Activation-space analysis: Steering adds a scaled normalized refusal direction, while ablation removes its projection from the residual-stream activation.The interventions quantify effects through refusal rate, over-refusal rate, and overall accuracy.
- Activation-space analysis: Steering strength is selected by validation as the smallest tested value reaching at least 90% refusal on harmful prompts while keeping benign over-refusals below a chosen threshold.The search uses a small discrete grid such as α ∈ {5, 10, 20, 30, 60}.
- SAE analysis: SAEs decompose activation-space refusal directions into sparse latent components, whose causal involvement is tested by amplifying their decoder directions during generation.The analysis also examines alignment, reuse across splits and layers, and semantic themes.
- SAE analysis: Refusal-associated latents are identified by comparing their firing rates across refusal and non-refusal examples, then ranked for constructing SAE-based refusal directions.The selected latent sets support subsequent causal steering, ablation, and cross-split reuse analyses.
3 Data and Models
The evaluation spans eleven refusal and non-compliance splits built from four complementary datasets, pairing labeled non-compliant prompts with benign prompts. Experiments use two instruction-tuned language models and fixed residual-stream hooks, with SAEs for selected layers.
- Evaluation data: Four datasets provide complementary coverage for eleven evaluation splits spanning safety, contextual non-compliance, unsupported or incomplete requests, and over-refusal.The splits include SafetyCore–WGM, four SorryBench domains, five CoCoNot categories, and OverRefusal–XST.
- Evaluation data: Each split pairs at least 32 prompts labeled to elicit non-compliant behavior with an equal number of benign prompts where compliance is appropriate.Multiple independent 32/32 splits are constructed when data allow, enabling within-category geometric stability checks.
- Models and representations: Experiments use Gemma-2-9B-IT and Llama-3.1-8B-Instruct, with activation-space directions computed at fixed mid-layer residual-stream hooks.SAE analyses use residual-stream SAEs at selected layers, focusing on later layers where refusal-related features are more pronounced.
4 Findings
Across 11 refusal splits, refusal directions are geometrically distinct, yet steering them produces similar refusal–over-refusal behavior. SAE analyses reveal a shared refusal core alongside specialized features that mainly shape refusal style.
- Activation-space directions: Typical cosine similarity is 0.4–0.6, with several near-orthogonal pairs, indicating distinct activation-space directions across refusal categories.
- Activation-space steering: Increasing steering strength raises refusal on harmful prompts while also increasing over-refusal on benign prompts across refusal directions and models.
- Refusal style: Refusal performance is largely invariant across directions, while expression varies by category, including confusion, non-human framing, impossibility claims, moral judgments, and safety disclaimers.
- SAE steering: A small SAE-derived direction recovers the same one-dimensional refusal knob: increasing β monotonically raises both harmful-prompt refusals and benign-prompt over-refusals.
- SAE steering: Random SAE latents produce much weaker, non-monotonic refusal changes, supporting specificity to identified refusal latents.
- Latent overlap: Only about 8–10% of 16,384 latents appear in any top-1000 list, while 2.5–4% appear across all splits, indicating sparse but consistent refusal structure.
- Latent overlap: Strict latent cores contain 591, 517, and 421 latents at layers 9, 20, and 31, while deeper layers show broader but smaller shared cores.
- Semantic structure: The latent organization combines a reusable refusal core with a long tail of features specialized for particular refusal styles or domains.
5 Related Work
Prior work established low-dimensional activation-space control of refusal and related methods for behavioral steering. This paper extends that line with sparse-autoencoder interpretability to analyze refusal features.
- Refusal directions and ablation: Prior work found that suppressing a dominant residual-stream direction inhibits refusal, while amplifying it induces refusal even for benign inputs.
- Activation steering: Representation engineering and activation addition introduced contrastive concept directions that can modulate behavior at inference time without optimization.
- Sparse autoencoders: The paper uses sparse autoencoders because their features are more interpretable than individual neurons and have been applied to safety-relevant representations.
6 Conclusion
The analysis finds multiple stable, geometrically distinct refusal directions, yet linear interventions collapse them into similar behavioral trade-offs. Shared latent features explain how rich internal structure becomes a single effective control dimension.
- Refusal corresponds to multiple stable and geometrically distinct directions in activation space.
- Different refusal-related directions produce nearly identical trade-offs between refusing and over-refusing, while visibly changing refusal style.
- Shared reusable latent features, together with style- and domain-specific features, underlie refusal across datasets.
- Distinct refusal directions are different linear combinations of shared latent structure, explaining their collapse into one effective control dimension under linear steering.
- These findings reconcile single-direction refusal control with more complex internal structure and expose a limitation of linear interpretability for alignment-relevant mechanisms.
Limitations
The study evaluates refusal mechanisms in two instruction-tuned models but does not establish whether the findings generalize beyond this model regime. Its sparse-feature analysis is also constrained by the limited availability of publicly released SAEs.
- The analysis covers two instruction-tuned language models spanning different architectures and training pipelines.
- The study does not evaluate whether the observed refusal mechanisms persist in larger, base, or differently aligned models.
- The findings characterize refusal within the studied model regime rather than establishing universality across all large language models.
- Sparse-feature analysis depends on publicly released SAEs, whose computational cost restricts coverage to a limited set of layers and models.
- Limited SAE coverage reduces feature-resolution and may bias analysis toward later layers where SAEs are more common.
Potential Risks and Ethics Statement
The work presents low-dimensional refusal control as dual-use research: the same techniques could suppress refusals and facilitate harmful content. The authors therefore emphasize analysis, established benchmarks, and the brittleness of linear control.
- Low-dimensional interventions can modulate refusal behavior, creating potential for misuse to suppress refusals and facilitate harmful content.
- The authors mitigate this risk by focusing on analysis rather than deployment and using established safety benchmarks.
- Linear steering collapses rich internal refusal structure into a coarse behavioral knob and is therefore not a principled safety mechanism.
- The authors use existing public datasets and moderation tools without introducing new harmful content, while motivating feature-level alignment approaches.
- The steering demonstrations use prompts about stealing money from an older woman and being hit.
- Different refusal directions reliably refuse both illustrative prompts but produce systematically different styles.
B Refusal splits
The refusal splits pair prompts that should elicit non-compliance with benign prompts, using dataset labels and balanced subsamples to learn and evaluate category-specific directions. Within-category directions are highly stable, while cross-category similarity is lower.
- Split construction: Each refusal split contains prompts expected to elicit non-compliant behavior and benign prompts for which straightforward compliance is appropriate.
- Split construction: Dataset safety labels and categories determine whether prompts belong on the non-compliant or benign side.
- Split construction: The source pools support balanced subsampling and held-out evaluation of refusal behavior and direction stability.
- Datasets: WildGuard-Mix contributes 915 vanilla prompts: 430 benign and 485 expected to elicit refusal-style behavior.
- Datasets: XSTest contributes 450 prompts, comprising 200 unsafe prompts that should be refused and 250 benign prompts.
- Datasets: CoCoNot supplies 6,526 non-compliant prompts across categories, paired with benign prompts from WildGuard-Mix.
- Datasets: SorryBench supplies 440 non-compliant prompts from 44 fine-grained categories, also paired with WildGuard-Mix benign prompts.
- Balanced subsamples: Direction learning typically uses 32 non-compliant and 32 benign prompts, with smaller sides matched when necessary.
D Oracle experiments for refusal directions
Oracle experiments show that refusal-related directions are stable and geometrically distinct, while their causal effects remain robust across intervention choices. SAE analyses further indicate that these directions engage different semantic themes despite sharing a common refusal-control profile.
- Held-out gold oracles: Gold-oracle comparisons likewise separated refusal contrasts: HR–BC and HR–HC reached 0.70±0.11 and 0.68±0.05 for a gold HR–BC oracle, while BR–BC reached 0.39±0.11.For a gold HR–HC oracle, HR–HC reached 0.81±0.02, HR–BC 0.51±0.03, and BR–BC 0.24±0.13.
- Stability under subsampling: 100 resampled HR–BC directions had mean cosine 0.96±0.01 to the oracle, whereas BR–BC directions were nearly orthogonal at approximately 0.10±0.07.The resamples used 32/32 examples from pools containing 11,872 HR and 8,048 BC examples.
- Family-level structure: Across layers and label families, HR–BC and HR–HC formed stable, separated refusal-related directions distinct from BR–BC and other harm/compliance contrasts.The family-level analysis reported high within-family cosine similarities and a cross-family HR–BC versus BR–BC cosine of approximately 0.10.
- Robustness of SAE interventions: Varying SAE layer, hook timing, and injection frequency preserved the smooth refusal/over-refusal trade-off, with deeper or more frequent hooks requiring smaller α.The overall curve shape remained essentially unchanged across layers 9, 20, and 31 and across prompt-only, generation-only, and combined interventions.
- Semantic structure: Different refusal directions emphasized different latent themes, including self-identity, illegality, professional qualifications, and capability or access limits.The semantic taxonomy also grouped top refusal-moving latents into recurring macro-classes spanning harm, exploitation, privacy, copyright, and capability limitations.
E Latent Semantic Annotation
The latent-annotation pipeline extracts highly activating examples, labels individual SAE latents, and groups them into semantic categories. The resulting annotations reveal both recurring refusal themes and context-dependent interpretations.
- Annotation procedure: The annotation pipeline selected the top-10 refusal-moving latents and used their 20 highest-activation prompts to generate semantic labels and characteristic patterns.GPT-4 then organized the latent annotations into at most four coherent semantic categories.
- Evaluation context: The controlled test set evaluated steered responses using WildGuard judgments of refusal or compliance across harmful and benign prompt types.Each split contained 50 harmful-refusal, 50 harmful-compliance, 50 benign-compliance, and 50 benign-refusal prompts.
- Context dependence: Latents received different semantic interpretations across data contexts, showing that a single latent can reflect multiple refusal-related themes.Latent 550 shifted from “Commissioned public-harm communications” to “Unsupported modality request” across analyses.
- Context dependence: Latent 7137 likewise represented missing-input requests, malware and doxxing requests, or attached-media translation depending on the analyzed subcategory.These context-specific labels support interpreting refusal latents as polysemous rather than fixed categorical detectors.
F Refusal Latent Overlap between Splits
The overlap analyses compare refusal-associated SAE latents across splits and layers, while accompanying tables report steering performance for full directions and single SAE directions. The materials distinguish shared latent structure from direction-specific behavior.
- Latent overlap analysis: Tables 16–19 measure overlap among refusal latents across directions and layers, using top-3000 latents overall and top-1000 latents for layers 9, 20, and 31.The comparisons cover latent overlap between directions from different splits.
- Steering evaluation: Table 12 reports Gemma steering performance with TP, TN, FP, and FN-based metrics, while the unsteered base model reaches 50% on all three metrics.The metrics use WildGuard judgments of refusal or compliance.
- SAE steering evaluation: Tables 13 and 14 report controlled-test performance for SAE-steered Llama and Gemma models using a single refusal direction.The Gemma table notes that its safety-tuned base model already exhibits substantial refusal and over-refusal at α = 0.
- Common and distinct latents: Table 15 distinguishes common SAE latents shared across settings from distinct latents specialized to harm types or capability limitations.The table frames common latents as domain-general refusal features and distinct latents as subcategory-specific detectors.
- Dataset coverage: The evaluated subcategories include hate speech, crimes and torts, humanizing requests, and unsupported visual, audio, artwork, and language requests.These categories span safety, anthropomorphization, and capability-related non-compliance.