PaperlayerView exact revision on arXiv

Source-linked AI summary

Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models

Minji Kim, Hyounghun Kim

arXiv:2609.04714v1cs.CLcs.AI

TL;DR

False refusals arise when language models reject benign prompts that resemble harmful requests, creating a need to understand their training-data causes. The paper decomposes safety responses into refusal statements and rationales, finding that rationale-only supervision reduces false refusals while preserving comparable safety and generalizing across evaluated settings.

  • Problem

    False refusals undermine usefulness and user trust because models can reject benign requests that superficially resemble harmful ones.

  • Method

    The paper decomposes safety-tuning responses into boilerplate refusal statements and rationales, then compares supervision and analyzes their behavioral effects.

  • Results

    Rationale-Only supervision reduces false refusals with rare safety compromises, while its benefits persist in in-context learning and complement inference-time mitigation methods.

  • Takeaways & Limitations

    Structurally curated, fine-grained safety supervision can better balance helpfulness and safety by emphasizing semantically informative rationales over boilerplate refusal statements.

  • Takeaways & Limitations

    Experiments exclude models larger than 80B parameters because tuning scale was constrained by limited resources.

Abstract

from arXiv · show

Striking a balance between helpfulness and safety remains a fundamental challenge in aligning large language models. To achieve this balance, models should refuse harmful queries (e.g., "How do I shoot someone?") while remaining responsive to benign inputs, even those superficially resembling harmful queries (e.g., "Where can I shoot a good photo?"). However, models often struggle to distinguish genuinely harmful queries from benign queries that contain superficially risky language, resulting in false refusals. In this paper, we address the issue by decomposing a response in the safety-tuning dataset into two distinct components: (i) a boilerplate refusal statement and (ii) a rationale explaining the refusal. Our experiments and analyses show that refusal statements impede accurate discrimination between harmful and benign queries by inducing reliance on superficial cues. In contrast, training solely on rationales reduces false refusals while maintaining a comparable level of safety performance. Rationale-Only benefits also appear in our ICL configuration and remain compatible with the evaluated inference-time mitigation methods. The results emphasize the necessity of precisely curated, fine-grained safety supervision datasets and outline directions for constructing aligned agents that better reconcile helpfulness with safety.

1 Introduction

The paper examines false refusals as a data-side alignment problem, identifying refusal statements as a source of superficial cue reliance. It finds that rationale-only supervision reduces false refusals while preserving safety and generalizing to additional settings.

  • Motivation: Safety tuning can reduce harmful assistance but also cause false refusals of benign prompts that resemble harmful requests.These errors diminish model utility and user trust.
  • Approach: The paper decomposes safety responses into a boilerplate refusal statement and a rationale explaining the refusal.This decomposition supports a data-centric analysis of response structure.
  • Findings: Rationale-Only supervision reduces false refusals with rare safety compromises.Paired outputs show refusals for harmful inputs while maintaining compliance for benign, pseudo-harmful inputs.
  • Applicability: Rationale-Only benefits persist in in-context learning and remain compatible with evaluated inference-time mitigation methods.The paper reports consistently low false refusal rates under these applicability settings.
  • Findings: Refusal statements drive false refusals by weakening models’ distinction between harmful and benign prompts and encouraging superficial cue reliance.The analyses attribute this behavior to reduced reliance on semantically informative signals.

2 Related Work

Prior work studies safety alignment, false refusals, and template-induced behavior, but the paper focuses on which components of safety-tuning responses precipitate false refusals. It frames refusal statements and rationales as distinct response structures for investigating this gap.

  • Safety alignment: Safety alignment methods aim to reject harmful requests without sacrificing general usefulness, using training-time and post-hoc interventions.The related work spans supervised fine-tuning, reinforcement learning, decoding changes, parameter steering, and representation interventions.
  • False refusal: False refusals are linked to overly cautious alignment and reliance on superficial linguistic cues rather than deeper contextual understanding.Existing mitigation approaches include prompting, activation manipulation, structured reasoning traces, and safety-boundary-aware data curation.
  • Paper’s focus: This paper addresses the remaining training-data gap by decomposing safety responses into refusal statements and rationales.The decomposition targets the response structure itself rather than only applying inference-time corrections.
  • Template-induced effects: Templated or highly regularized formats can encourage models to use surface patterns instead of underlying semantics.Examples in related work include word overlap, negation, regular answer formats, and positional biases.

3 Method

The method decomposes safety-tuning responses into boilerplate refusal statements and request-specific rationales, then manipulates their structure, position, and specificity to isolate their effects.

  • Problem formulation: The conceptual prompt representation separates semantic intent from superficial lexical patterns without claiming explicit neural disentanglement.
  • Response decomposition: Safety responses are modeled as a boilerplate refusal statement followed by a rationale explaining request-specific concerns.
  • Base formatting: Responses are normalized by placing one concise refusal first, followed by a contiguous rationale of at least two sentences while preserving substantive content.
  • Experimental conditions: Statement position is varied by placing the refusal at the beginning, middle, or end while keeping the remaining response nearly unchanged.
  • Experimental conditions: Component conditions compare Statement-Only, Rationale-Only, and Statement and Rationale responses, while specificity conditions compare Generic and Request-Specific rationales.
  • Rationale specificity: The study uses tabled Generic and Request-Specific rationale variants for the same harmful request to examine rationale content.

4 Experimental Setup

Experiments fine-tune base language models under matched conditions and evaluate false refusal, harmful-query safety, general capabilities, and jailbreak robustness across specified benchmarks.

  • Models: Experiments use base Llama-3.1-8B, Mistral-7B-v0.3, Gemma-2-9B, and Qwen2.5-7B models to reduce confounding from prior instruction tuning.
  • Training data: Training combines 1,024 Alpaca instruction-response pairs with 256 safety-oriented instances filtered from Safety-Tuned LLaMAs.
  • Training setup: All conditions use QLoRA with identical optimization settings, a maximum sequence length of 2,048 tokens, and greedy decoding during inference.
  • Evaluation: Core capabilities are evaluated across multiple benchmarks, and Table 2 reports compliance with lower values preferred on harmful sets and higher values on pseudo-harmful sets.
  • Evaluation: Safety is evaluated on AdvBench and MaliciousInstruct, while jailbreak robustness is assessed on HarmBench, JailbreakBench, WILDJAILBREAK, and SORRY-Bench.
  • Evaluation: False refusal is measured on XSTest-Safe and OKTest, whose lexically similar but semantically harmless queries test compliance with pseudo-harmful prompts.

5 Results

The results identify refusal statements as a source of false refusals, while rationale-only supervision lowers false refusals without reducing safety or general capabilities; request-specific rationales provide an additional benefit.

  • Refusal statements: Refusal statements impede discrimination between harmful and pseudo-harmful queries, with their position influencing model sensitivity.
  • General capabilities: Altering refusal statements and rationales produces negligible differences in overall model performance across core capability benchmarks.
  • Response components: Including a refusal statement elevates false refusal rates whether it appears alone or alongside a rationale.
  • Response components: Rationale-Only models reduce false refusals while maintaining comparable safety performance to models trained with statements and rationales.
  • Rationale specificity: Request-Specific rationales consistently outperform Generic rationales on pseudo-harmful benchmarks by naming the requested action and its risk.
  • Robustness checks: The component-level performance gap persists under length and reasoning-first controls.

6 Analyses

The analyses test whether response decomposition preserves capability and safety while explaining how refusal statements contribute to false refusals. Results indicate that boilerplate templates increase reliance on superficial cues, whereas Rationale-Only remains more semantically grounded.

  • Evaluation scope: The evaluation examines whether response decomposition preserves core capabilities and robustness against jailbreak-style inputs.The analyses cover six capability benchmarks and four jailbreak-style safety benchmarks.
  • Core capabilities: Across established capability tasks, no response condition shows a consistent advantage or degradation.Performance varies slightly across tasks and model families, but removing refusal statements does not impair unrelated capabilities.
  • Safety robustness: Improved responsiveness to pseudo-harmful prompts does not produce a consistent loss in safety on the evaluated jailbreak-style inputs.The jailbreak-style evaluation reports harmful-compliance rates, where lower values indicate greater robustness.
  • Internal effects: Statement and Rationale produces lower first-token entropy on harmful and pseudo-harmful prompts, while Rationale-Only distributes sensitivity across more semantically informative spans.These patterns are examined through decoding entropy and token-level attribution, primarily for Llama-3.1-8B.
  • Internal effects: Manual inspection finds meaningful attributions in over 97% of Rationale-Only cases, compared with less content-related cues in approximately 90% of Statement and Rationale instances.The comparison links boilerplate statements with surface-level associations and less flexible refusal patterns.
  • Structural effects: A fixed neutral prefix still raises false refusal rates, stylistic diversity improves compliance, and both remain below the Rationale-Only baseline.The structural experiments vary response regularity, stylistic form, and keyword placement.
  • Structural effects: Keyword-conditioned refusal statements reduce compliance for their targeted keyword while leaving other keyword categories comparatively similar.The experiment uses five risky keywords and evaluates pseudo-harmful queries for each category.

7 Applicability Study

The applicability study tests Rationale-Only beyond fine-tuning, including in-context learning and inference-time mitigation. Its advantage persists across these settings, with request-specific combinations achieving the lowest false-refusal rates.

  • Study design: The study evaluates whether Rationale-Only benefits persist in in-context learning without fine-tuning and alongside inference-time mitigation methods.The applicability experiments use modified demonstrations and existing mitigation procedures.
  • In-context learning: Rationale-Only findings consistently hold under the evaluated in-context learning settings.The ICL setup modifies URIAL demonstrations under Statement-Only, Rationale-Only, and Statement and Rationale conditions.
  • Inference-time mitigation: Under both evaluated mitigation methods, Rationale-Only retains lower false-refusal rates than Statement-Only and Statement and Rationale.These experiments use Llama-3.1-8B only.
  • Inference-time mitigation: Combining Rationale-Only with the evaluated methods further reduces false-refusal rates, with Request-Specific models achieving the lowest rates across benchmarks.The result supports tailoring rationales to the specific request.

8 Conclusion

The paper concludes that decomposing safety responses identifies refusal statements as contributors to false refusals, while Rationale-Only mitigates them with rare safety compromises. These benefits generalize to ICL and complement inference-time mitigation methods.

  • Conclusion: Decomposing safety responses into refusal statements and rationales identifies refusal statements as contributors to false refusals.The paper frames this as a data-level analysis of safety-response structure.
  • Conclusion: Rationale-Only supervision mitigates false refusals with rare safety compromises and encourages reliance on semantically meaningful rather than superficial signals.The reported benefits generalize to in-context learning and complement existing inference-time mitigation methods.

Limitations

The experiments are constrained to models no larger than 80B parameters because of limited tuning resources, leaving larger-scale effects for future study.

  • Scope boundary: The study does not include models larger than 80B parameters because the scale of tuning was constrained by limited resources.Future work could investigate curated safety training datasets at larger scale.

Ethics Statement

The study documents annotation safeguards and describes the source data and training setup used to construct experimental variants.

  • Annotators could decline participation or withdraw without penalty, and reviewers screened sensitive examples before distribution.
  • The authors analyzed the original safety dataset to identify responses combining refusal statements with explanatory rationales.
  • Dataset-generation prompts and examples are documented in supplementary sections and tables.
  • Models were fine-tuned with QLoRA using LoRA adapters, 4-bit quantization, a 1e-4 learning rate, batch size 64, and 10 epochs.

C Evaluation

The evaluation combines automatic, human, and external-model judgments across safety, false-refusal, capability, and uncertainty analyses.

  • Metrics: Compliance rates distinguish harmful benchmarks, where lower compliance is preferred, from pseudo-harmful benchmarks, where higher compliance is preferred.
  • Automatic evaluation: The safety judge achieved 98.33% accuracy on 120 validation examples balanced between compliance and refusal.
  • False-refusal evaluation: False-refusal evaluation uses stricter semantic-compliance criteria to detect partial refusals, deflections, and ethical caveats that surface-level judges may accept.
  • Human evaluation: Human evaluation covered 550 pseudo-harmful examples distributed equally among five graduate-student evaluators without annotation overlap.
  • Robustness checks: GPT-5.1 judgments closely matched the primary judge, while qualitative audits found no systematic difference in harmful-query failure patterns for Rationale-Only.
  • Benchmark coverage: Capability testing spans factual knowledge, commonsense and logical reasoning, while additional benchmarks probe harmful behavior and jailbreak robustness.
  • Uncertainty analysis: Next-token uncertainty is measured with entropy, and token contributions are probed through embedding-layer ablations reporting mean signed entropy shifts with bootstrap confidence intervals.

F.5 Neutral Prefix Experiment

The neutral-prefix experiment tests whether a consistent response template can affect false refusals even when the template contains no explicit safety language.

  • A fixed “Thank you for asking!” prefix was prepended to every Rationale-Only response while preserving the original rationale and other training data.
  • The neutral template increased false refusal rates relative to Rationale-Only, suggesting that consistent structure can become associated with refusal behavior.
  • The study also constructs variants with 15 refusal phrasings spanning formal, concise, empathetic, and principle-driven styles.
  • Keyword-conditioned variants inject refusal statements only for responses whose harmful queries contain one of five selected risky tokens.
  • The in-context-learning experiments compare Statement and Rationale, Statement-Only, and Rationale-Only templates under greedy decoding.

H Results for Different Model Sizes

Across model sizes and control analyses, Rationale-Only retains higher pseudo-harmful compliance while preserving low harmful-query compliance; comparisons with instruction-tuned models remain limited.

  • Model-size results: The same pattern holds across smaller and larger models: refusal statements increase false refusals, whereas Rationale-Only training reduces them.
  • Length control: A one-sentence Rationale-Only condition retains higher pseudo-harmful compliance than Statement-Only across all four model families while harmful-query compliance remains low.
  • Reasoning-first control: A generic reasoning-first cue does not close the Rationale-Only advantage, and its effects are small and mixed across models and benchmarks.
  • Instruction-tuned comparison: Instruction-tuned comparisons are constrained because their proprietary alignment datasets and training pipelines are undisclosed.
  • Dataset-size control: Increasing the safety dataset from 256 to 512, 1024, or 2048 examples does not change the relative advantage of rationale-only training.
  • Evaluation materials: The evaluation materials compare response components, refusal placement, rationale explicitness, compliance judgments, and example outputs across harmful and pseudo-harmful benchmarks.
Loading 2609.04714v1…