PaperlayerView exact revision on arXiv

Source-linked AI summary

Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation

Xinpeng Wang, Chengzhi Hu, Paul Röttger, Barbara Plank

arXiv:2410.03415v2cs.CL

TL;DR

False refusal makes it difficult for language models to remain helpful on safe requests while refusing harmful ones. The paper uses single vector ablation with orthogonalized false-refusal vectors to reduce false refusal, preserving safety and general capability while enabling fine-grained calibration.

  • Problem

    Language models can refuse clearly safe queries that superficially resemble unsafe ones, making refusal behavior difficult to calibrate without compromising helpfulness or harmlessness.

  • Method

    The method extracts true-refusal and false-refusal vectors, orthogonalizes them, and ablates the false-refusal direction from model activations or weights.

  • Results

    The method reduces false refusal while maintaining low compliance on harmful queries and preserving general capabilities, with improvements also appearing on out-of-distribution false-refusal tests.

  • Takeaways & Limitations

    Single vector ablation provides a cheap, surgical, training-free way to calibrate refusal behavior, including through partial orthogonalization and adjustable projection removal.

  • Takeaways & Limitations

    Vector quality could improve with more diverse pseudo-harmful samples and better data curation, while the refusal-score metric could also be improved.

Abstract

from arXiv · show

Training a language model to be both helpful and harmless requires careful calibration of refusal behaviours: Models should refuse to follow malicious instructions or give harmful advice (e.g."how do I kill someone?"), but they should not refuse safe requests, even if they superficially resemble unsafe ones (e.g. "how do I kill a Python process?"). Avoiding such false refusal, as prior work has shown, is challenging even for highly-capable language models. In this paper, we propose a simple and surgical method for mitigating false refusal in language models via single vector ablation. For a given model, we extract a false refusal vector and show that ablating this vector reduces false refusal rate while preserving the model's safety and general capabilities. We also show that our approach can be used for fine-grained calibration of model safety. Our approach is training-free and model-agnostic, making it useful for mitigating the problem of false refusal in current and future language models.

1 INTRODUCTION

False refusal creates a difficult calibration problem: models must reject harmful requests without rejecting safe requests that superficially resemble them. The paper proposes single vector ablation as a cheap, flexible, and surgical training-free solution.

  • False refusal occurs when models reject clearly safe queries because they superficially resemble unsafe requests, complicating the balance between helpfulness and harmlessness.
  • Prior training-based methods are inflexible after training, while training-free methods require costly inference-time computation and can be imprecise.
  • Single vector ablation extracts and separates true-refusal and false-refusal vectors, then removes the false-refusal direction from model activations.
  • The method enables fine-grained safety calibration through partial orthogonalization and adjustable refusal-removal strength while preserving safety and general capability.

2 BACKGROUND

The background describes how refusal vectors are extracted from activation differences and selected using refusal-score changes. Vector ablation removes a selected refusal direction across activations, while vector addition increases refusal behavior.

  • 2.1 REFUSAL VECTOR EXTRACTION: Difference-in-means extracts candidate refusal vectors by subtracting mean activations for harmless prompts from mean activations for harmful prompts at each layer and token position.
  • 2.1 REFUSAL VECTOR EXTRACTION: Candidates are ranked by the drop in refusal score after ablation, where refusal score compares probabilities of refusal-related and non-refusal-related tokens at the first response position.
  • 2.2 VECTOR ABLATION AND ADDITION: A selected refusal vector is ablated by projecting each residual-stream activation onto its direction and removing that projection.
  • 2.2 VECTOR ABLATION AND ADDITION: Ablation is applied across layers and token positions, whereas adding a refusal vector can increase refusal behavior at a chosen layer with controllable strength α.
  • 2.2 VECTOR ABLATION AND ADDITION: Both vector removal and addition can be applied directly to model weights without adding computation during inference.

3 MITIGATING FALSE REFUSAL VIA SINGLE VECTOR ABLATION

The method disentangles false-refusal features from true-refusal features by orthogonalizing their vectors before ablating the false-refusal direction. Partial orthogonalization and adjustable removal strength provide finer control over responses to ambiguous queries.

  • 3.1 EXTRACTING A FALSE REFUSAL VECTOR: The method extracts false-refusal candidates from pseudo-harmful prompts, selects the most effective vector using validation refusal-score drops, and then orthogonalizes it against the true-refusal vector.
  • 3.1 EXTRACTING A FALSE REFUSAL VECTOR: Raw difference-in-means vectors are insufficient because true-refusal and false-refusal vectors are not independent, so direct ablation can remove refusal behavior for both harmful and harmless prompts.
  • 3.1 EXTRACTING A FALSE REFUSAL VECTOR: Ablating the orthogonalized false-refusal vector preserves low harmful-query compliance while substantially increasing harmless-query compliance, unlike raw difference-in-means ablation.
  • Orthogonalization sets a boundary intended to retain refusal responses to harmful features while reducing excessive caution on ambiguous examples.
  • Lowering the partial-orthogonalization coefficient modifies the false-refusal vector less, producing stronger false-refusal mitigation and granular control over ambiguous-query sensitivity.

4 EXPERIMENTAL SETUP

The experiments evaluate chat-tuned models on safety, false refusal, and general capability. They use harmful, harmless, and pseudo-harmful data, including in-distribution and out-of-distribution false-refusal tests.

  • Models: The study evaluates GEMMA-7B-IT, LLAMA2-7B/13B/70B-CHAT, and LLAMA3-8B-INST with greedy decoding.
  • Datasets for Vector Extraction: Vector extraction uses harmful, harmless, and pseudo-harmful datasets, with OR-BENCH-HARD chosen because its challenging examples better control refusal behavior and minimally affect true refusal.
  • Evaluation: The evaluation covers safety, false refusal, and general capability using harmful tests, ORB-H, OKTEST, XSTEST, MMLU, ARC-C, and WIKITEXT.
  • Evaluation: False refusal is assessed on one in-distribution pseudo-harmful dataset and two out-of-distribution datasets, including a harder high-refusal safe subset of XSTEST.
  • Evaluation: Responses are classified using refusal-string matching, while main harmful-query experiments additionally use Wildguard to determine full compliance.

5 RESULTS

Single-vector ablation increases compliance on pseudo-harmful queries while preserving refusal of harmful queries and general capabilities. Partial orthogonalization and λ provide fine-grained control over safety sensitivity, whereas vector addition improves safety at a cost to general-task performance.

  • 5.1 MITIGATING FALSE REFUSAL: 26.7% compliance on ORB-H followed ablation from 5.5% for LLAMA2-13B-CHAT, while true refusal and general capability changed minimally.Across models, vector ablation increased compliance on pseudo-harmful data while preserving harmful-query refusal and general performance.
  • 5.1 MITIGATING FALSE REFUSAL: 42.4% versus 13.6% compliance on XSTest-S(H), and 65.0% versus 59.0% on OKTest, show generalization beyond the extraction data.The false refusal vector was extracted using ORB-H data, yet improvements also appeared on the two out-of-distribution tests.
  • 5.1 MITIGATING FALSE REFUSAL: Less than 1.0 absolute change in general-task metrics indicates that ablation minimally affects capabilities while restoring answers to harmless queries and preserving harmful-query refusal.Figure 1 illustrates this behavior for LLAMA2-7B-CHAT, including the distinction between a harmful cybersecurity request and a harmless “kill the lights” query.
  • 5.2 FINE-GRAINED CONTROL WITH PARTIAL ORTHOGONALIZATION: Lowering λ makes responses more compliant, enabling fine-grained control over the boundary between safe and unsafe concepts.For the fish-head example, λ values of 1, 0.8, and 0.6 correspond to refusal, cautionary compliance, and full compliance, respectively.
  • 5.2 FINE-GRAINED CONTROL WITH PARTIAL ORTHOGONALIZATION: MMLU accuracy remains unaffected as λ changes, while compliance rates on both harmful and pseudo-harmful data increase as λ decreases.The selected vectors have minimal effect on the output distribution, supporting calibration without measurable general-capability loss in this evaluation.
  • 5.4 DOES ADDING TRUE REFUSAL VECTOR HELP KEEPING MODEL SAFE?: Vector addition improves safety only by sacrificing ARC-C, MMLU, and Wikitext performance, unlike the more surgical ablation approach.The comparison motivates ablation as a way to reduce false refusal without the general-performance degradation observed for addition.

6 RELATED WORK

Prior work evaluates and mitigates false refusal through test suites and training-free or training-based methods, while representation editing offers a route to steer refusal behaviour. The paper positions its approach as a more surgical way to target false refusal without harming other capabilities.

  • Testing for False Refusal: XSTest introduced a dedicated false-refusal evaluation suite, while OR-Bench expanded testing with large collections of seemingly toxic prompts.XSTest contains 250 safe and 200 contrasting unsafe prompts; OR-Bench contains 80k seemingly toxic prompts.
  • Mitigating False Refusal: Training-based mitigation is effective but only calibrates safety during training, whereas training-free methods are flexible but add inference-time computation.Prior training-free methods are also described as imprecise because they can have unintended effects.
  • Representation Editing: Representation editing steers behaviour through hidden representations and has been used to control truthfulness, sentiment, and refusal behaviour.Arditi et al. (2024) extracted a refusal direction that controls general refusal regardless of whether refusal is true or false.
  • Representation Editing: This work distinguishes false from true refusal to target the former more surgically and avoid harming other model behaviour.The paper contrasts its fine-grained approach with prior general refusal directions.

7 CONCLUSION

The paper concludes that ablating a single orthogonalized false refusal vector can mitigate false refusal while preserving safety, general capabilities, and inference efficiency. It also supports fine-grained safety calibration, but notes that better sample curation and refusal metrics could improve the method.

  • 7 CONCLUSION: Single-vector ablation targets false refusal while preserving harmful-query safety through orthogonalization against the true refusal vector.The false refusal vector is extracted from pseudo-harmful and harmless data, and its projection onto the true refusal vector is removed before ablation.
  • 7 CONCLUSION: Partial orthogonalization and adjustable projection-removal strength enable fine-grained control over compliance with ambiguous and sensitive queries.The paper frames this flexibility as allowing response compliance levels to be adjusted according to user judgment.
  • 7 CONCLUSION: Direct model-weight modification preserves the modified model’s original general capabilities and requires no additional inference cost.This is presented as a consequence of the ablation operation’s surgical character.
  • Limitations: The method’s effectiveness could improve with more diverse pseudo-harmful samples, better data curation, and refusal metrics based on sequence probabilities.The current work mainly uses pseudo-harmful samples from OR-Bench-Hard and selects vectors using refusal-related tokens at the first token position.

C EVALUATION ON SAFETY AND REFUSAL OUTPUT

The evaluation uses keyword matching to detect refusal, with WildGuard and human annotation added where string matching is less reliable. Weight-space vector ablation prevents representations of the targeted refusal vector without changing matrix dimensions or inference cost.

  • Evaluation: Refusal detection uses string matching against keywords shown in Figure 8 for harmful and pseudo-harmful datasets.The keywords are used to judge whether the model refuses.
  • Evaluation: WildGuard evaluates safety in the main experiments because keyword matching has a minor gap from human annotations on harmful datasets.String matching is retained for other experiments and refusal-trend analysis because it shows similar general trends to WildGuard.
  • Weight ablation: The fixed refusal vector can be ablated before inference by directly transforming model weights rather than intervening during generation.The edited model therefore requires no inference-time intervention.
  • Weight ablation: Applying the transformation to Attention or Feedforward outputs prevents those blocks from representing the vector targeted for ablation.The output of either block is written to the residual stream, so the intervention acts on that representation.
  • Weight ablation: Because Xoutput = WXpre, ablating the output is equivalent to ablating the weight matrix W.The transformed matrix W′ preserves the original dimensions, so the edited model adds no memory or inference cost.

E REFUSAL VECTOR ADDITION

Adding a refusal vector makes GEMMA-7B-IT safer and more conservative, but increasing its strength sacrifices general capabilities. A particular setting raises TruthfulQA substantially while degrading usefulness on other tasks.

  • Refusal vector addition: TruthfulQA rises from 56.5 to 90.0 for GEMMA-7B-IT at α = 0.8 and λ = 0.8.The improvement comes with a substantial usefulness cost on OR Bench, ARC-C, and Wikitext.
  • Refusal vector addition: Increasing α degrades general model capacity, while adding a refusal vector makes the model safer and more conservative.The reported capability losses include Wikitext PPL and ARC-C.

F COMPARISON TO MORE BASELINE METHODS

The section compares the proposed method with training-based and training-free safeguards using reported baseline results. Table 4 covers LLAMA2-7B-CHAT compliance on unsafe and safe XSTest prompts.

  • Baseline comparison: The comparison includes DRO, Self-CD, SafeDecoding, and SCANS as training-based or training-free baselines.The baseline performance numbers are taken directly from Cao et al. (2024).
  • Baseline comparison: Table 4 reports LLAMA2-7B-CHAT compliance rates after intervention on XSTest-Unsafe and XSTest-Safe.

G FINE-GRAINED CONTROL ON RESPONSE OPENNESS

Varying λ provides fine-grained control over how directly models address sensitive prompts. Full orthogonalization favors refusals, while lower values increase engagement but can risk unsafe responses or produce benign apparent compliance.

  • FINE-GRAINED CONTROL ON RESPONSE OPENNESS: Lowering λ from 1 to 0 makes responses more direct, enabling control over model openness to sensitive or harmful prompts.λ = 1 denotes full orthogonalization and λ = 0 denotes none.
  • FINE-GRAINED CONTROL ON RESPONSE OPENNESS: Pseudo-harmful prompts shift from literal interpretations at λ = 1 to more nuanced metaphorical explanations at lower λ values.The examples show tuning toward contextually appropriate responses while maintaining ethical boundaries.
  • FINE-GRAINED CONTROL ON RESPONSE OPENNESS: High λ values produce clear refusals for harmful prompts, whereas lower values increase engagement and may retain warnings or factual corrections.At low λ, responses can become unsafe, although some apparent compliance remains benign and counter-argues the unsafe prompt.
  • FINE-GRAINED CONTROL ON RESPONSE OPENNESS: The λ trade-off balances overly restrictive behavior against potentially unsafe responses across harmful and jailbreaking prompts.This calibration can tailor refusal behavior to use cases while maintaining overall safety and ethical standards.
Loading 2410.03415v2…