PaperlayerView exact revision on arXiv

Source-linked AI summary

FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning

Zhehao Zhang, Weijie Xu, Fanyou Wu, Chandan K. Reddy

arXiv:2505.08054v2cs.CLcs.AI

TL;DR

Safety alignment can lead LLMs to refuse benign but ambiguous requests, limiting helpfulness in sensitive scenarios. FalseReject combines graph-informed adversarial query generation with structured responses and training sets for standard and reasoning-oriented models. Benchmarking found persistent over-refusal, while supervised finetuning reduced unnecessary refusals without harming general language capabilities.

  • Problem

    Over-refusal of benign instructions and ambiguity in natural language make it difficult to calibrate informative responses that remain safe.

    Source: §1 · pp. 1–2

  • Method

    FalseReject uses graph-informed adversarial multi-agent query generation and structured safety reasoning to build training sets and a human-annotated benchmark.

    Source: §1 · p. 2

  • Results

    Across 29 SOTA LLMs, benchmarking found variable over-refusal; FalseReject finetuning reduced unnecessary refusals without harming general language capabilities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FalseReject provides a resource for evaluating and calibrating over-refusal across safety scenarios in both standard and reasoning-oriented LLMs.

    Source: §1 · p. 2

Abstract

from arXiv · show

Safety alignment approaches in large language models (LLMs) often lead to the over-refusal of benign queries, significantly diminishing their utility in sensitive scenarios. To address this challenge, we introduce FalseReject, a comprehensive resource containing 16k seemingly toxic queries accompanied by structured responses across 44 safety-related categories. We propose a graph-informed adversarial multi-agent interaction framework to generate diverse and complex prompts, while structuring responses with explicit reasoning to aid models in accurately distinguishing safe from unsafe contexts. FalseReject includes training datasets tailored for both standard instruction-tuned models and reasoning-oriented models, as well as a human-annotated benchmark test set. Our extensive benchmarking on 29 state-of-the-art (SOTA) LLMs reveals persistent over-refusal challenges. Empirical results demonstrate that supervised finetuning with FalseReject substantially reduces unnecessary refusals without compromising overall safety or general language capabilities.

1 Introduction

The introduction frames over-refusal as a calibration problem: ambiguous prompts can invite needless refusal even when benign interpretations exist. FalseReject combines structured training data with a benchmark to address this challenge.

1 Introduction

  • The introduction identifies gaps in prior resources: synthetic queries lack diversity and complexity, and prior datasets focus on evaluation rather than training.Manually written prompts have also become less challenging for state-of-the-art models.Source: §1 · p. 1
  • Ambiguous prompts complicate calibration because models must provide informative answers for safe interpretations while maintaining safety.The introduction describes over-refusal as unnecessary rejection of benign instructions, which can diminish user experience in sensitive scenarios.Source: §1 · pp. 1–2
  • FalseReject uses graph-informed adversarial multi-agent interactions and structured safety reasoning to generate challenging queries and responses for safe-versus-unsafe distinctions.Its resource includes 15k training examples and 1.1k test examples across 44 safety-related categories, with sets tailored to standard and reasoning-oriented LLMs.Source: §1 · p. 2
  • FalseReject training substantially reduces unnecessary refusals without harming general language capabilities, with benchmarking across 29 SOTA LLMs showing substantial variation in over-refusal.The introduction also reports improved distinction between safe and unsafe contexts and deeper, more consistent alignment than standard instruction-tuned models.Source: §1 · p. 2

2 Related Work

Related work covers datasets for evaluating over-refusal and inference-time or training-based approaches to mitigation. The paper positions FalseReject as a training resource with both queries and responses.

Over-Refusal Dataset

  • Existing over-refusal datasets include safe prompts resembling harmful ones, but some are now too simple for state-of-the-art models and many provide queries without training responses.FalseReject instead includes query-response data to support instruction-tuning calibration.Source: §2 · pp. 2–3

Over-Refusal Mitigation

  • Figure 1 contrasts direct refusal of a benign prompt and unsafe compliance with the contextual distinction attributed to FalseReject-finetuned models.The figure presents these examples as a helpfulness-and-safety contrast.Source: Figure 1 · p. 3
  • Prior mitigation methods largely intervene at inference time, while FalseReject targets refusal behavior during post-training.Other training approaches cited focus on jailbreak prevention rather than over-refusal explicitly.Source: §2 · p. 3

3 FalseReject: A Large-Scale Over-Refusal Training and Evaluation Resource for LLMs

FalseReject provides 16k seemingly harmful but benign prompts across 44 safety-related categories, with training responses and a human-annotated test set. Its pipeline combines graph-based query generation, quality controls, and context-aware response construction.

3 FalseReject: A Large-Scale Over-Refusal Training and Evaluation Resource for LLMs

  • FalseReject contains 16k prompts across 44 safety-related categories, including 15k training query-response pairs and a 1.1k-sample human-annotated test set.The training sets target non-reasoning and reasoning LLMs, respectively.Source: §3 · p. 3

3.1 Entity Graph Extraction

  • Entity graphs built from safety-related datasets encode relationships among extracted people, locations, objects, and concepts to ground diverse query generation.An LLM extracts candidate entities, and an LLM-driven vote selects a representative list for the graph.Source: §3.1 · p. 4

3.2 Iterative Adversarial Multi-Agent Interaction for Synthetic Queries

  • A Generator and Discriminator iteratively develop seemingly harmful but benign prompts; validation LLMs retain prompts that trigger at least one refusal, and an Orchestrator checks their validity.The adversarial process uses feedback to refine prompts and confirms they are sensitive-looking yet objectively benign.Source: §3.2 · pp. 4–5

3.3 Quality Control and Human Annotation

  • Quality control removes duplicates, balances coverage across 44 categories, and uses human annotation to select sensitive-looking queries that can be safely answered with sufficient context.FalseReject-Test contains 1.1k data points.Source: §3.3 · p. 5

3.4 Context-Aware Safety Response Generation via Structured Reasoning Traces

  • Structured responses distinguish safe from potentially unsafe interpretations, explain the safe interpretation, address risks with clarification or guidance, and close with a cautionary summary.FalseReject-Train-Instruct uses a standard LLM for regular responses, while FalseReject-Train-CoT uses DeepSeek-R1 for reasoning-intensive responses.Source: §3.4 · pp. 5–6

4 Benchmarking LLMs with FalseReject

The section benchmarks 29 LLMs using Compliance Rate and Useful Safety Rate, finding that over-refusal remains a challenge even for widely used models. Results also vary across reasoning models and model families, with neither model size nor general language ability consistently predicting refusal behavior.

4.1 Results and Findings

  • GPT-4.5 and Claude-3.5-Sonnet both have compliance rates below 50%, showing that over-refusal remains challenging across SOTA models.Figure 3 compares Compliance Rate and USR across models.Source: Figure 3 · p. 6, §4.1 · p. 7

4 Benchmarking LLMs with FalseReject

  • The benchmark evaluates 29 closed- and open-source models, reporting Compliance Rate and a three-class Useful Safety Rate that distinguishes direct refusal, safe partial compliance, and full compliance.The evaluation uses Claude-3.5-Sonnet as an LLM judge; higher USRToxic reflects better safety, while higher USRBenign indicates improved helpfulness with fewer refusals on benign prompts.Source: §4 · pp. 6–7

4.1 Results and Findings

  • DeepSeek-R1 reaches 87.53% compliance and 99.66% USR, while QwQ and o1 have substantially lower compliance than the average of non-reasoning models.The reported results show inconsistent over-refusal behavior among reasoning-oriented models.Source: §4.1 · p. 7
  • Phi-4 models have a 30–40% gap between Compliance Rate and USR, while GPT models such as GPT-4o have gaps below 15%.The paper interprets the gap as approximating the share of safe partial compliance.Source: §4.1 · p. 7, §4 · p. 7
  • No consistent relationship appears between model size and refusal metrics: Llama-3.2-1B has 78.43% compliance and 86.60% USR, versus 56.28% and 74.56% for Llama-3-1-405B.Source: §4.1 · pp. 7–8
  • GPT-4.5 and o1 show lower Compliance Rate and USR than Llama-3.2-1B and Phi-4-mini, so general language ability does not necessarily correlate with improved over-refusal behavior.The paper says refusal-targeted alignment requires distinct optimization objectives.Source: §4.1 · p. 8
  • Mistral-7B and DeepSeek-R1 show strong over-refusal metrics compared with GPT-4.5 and the Claude-3 series, indicating that some open-source models can potentially outperform closed-source counterparts.Source: §4.1 · p. 8

5 Finetuning To Mitigate Over-refusal with FalseReject

The section describes FalseReject fine-tuning for non-reasoning and reasoning models, its evaluation setup, and analyses of alignment depth and jailbreak robustness. Reported findings include reduced over-refusal, improved safety for reasoning models, and sustained alignment without compromising general utility.

5 Finetuning To Mitigate Over-refusal with FalseReject

  • Training uses a 90:10 utility-to-safety data ratio, sampling 12,000 utility pairs and 1,300 FalseReject pairs for each target model type.Non-reasoning models use Tulu-3 and FalseReject-Train-Instruct; reasoning models use Open-Thoughts and FalseReject-Train-CoT. Models are fine-tuned from base pretrained checkpoints and compared with utility-only baselines.Source: §5 · p. 8
  • Evaluations cover over-refusal, safety, and general ability using FalseReject-Test, OR-Bench-Hard-1K, four safety datasets, GSM8K, and MMLU.The evaluated model families include Llama, Qwen, and Gemma models of varying sizes.Source: §5 · p. 8

5.1 Results and Findings

  • FalseReject SFT reduces over-refusal in non-reasoning models and improves reasoning-model safety while preserving general utility.The reported results also describe minimal loss in general language ability for non-reasoning models and near-perfect over-refusal benchmark results for reasoning models.Source: §5.1 · pp. 8–9, Table 2 · p. 9
  • Reasoning models trained with FalseReject-Train-CoT improve safety over general-CoT-only training while maintaining strong over-refusal benchmark and general-utility performance.The general-CoT-only models achieve high compliance on over-refusal evaluations but struggle in safety evaluations.Source: §5.1 · p. 9

5.2 In-depth Analysis

  • FalseReject-Train-Instruct produces deeper, more sustained alignment than official instruction-tuned versions across three model families.For Gemma-2-2B and Llama-3.2-1B, KL divergence stays high beyond the first five response tokens; Qwen-2.5-7B also maintains elevated divergence at subsequent positions.Source: §5.2 · pp. 9–10, Figure 4 · p. 10
  • Fine-tuning with FalseReject does not increase vulnerability to GCG, Prefilling, or AutoDAN jailbreak attacks and consistently improves or maintains resilience.The authors connect this result to their hypothesis that structured-reasoning training fosters deeper, more principled safety alignment.Source: §5.2 · p. 10

6 Conclusion

The conclusion presents FalseReject as a resource for benchmarking and mitigating over-refusal, built from seemingly harmful but benign prompts and structured responses. It reports that fine-tuning reduces unnecessary refusals in non-reasoning models and improves reasoning-model safety without compromising general language capabilities.

6 Conclusion

  • FalseReject is a large-scale resource for benchmarking and mitigating over-refusal in LLMs, comprising 16k seemingly toxic queries and structured responses spanning 44 safety-related categories.Source: §6 · p. 10
  • A graph-informed adversarial multi-agent method generates diverse, challenging queries, while structured context-aware responses help distinguish safe from unsafe contexts.Source: §6 · p. 10
  • Benchmarking 29 state-of-the-art LLMs reveals frequent over-refusal; FalseReject SFT reduces unnecessary refusals in non-reasoning models and improves reasoning-model safety without compromising general language capabilities.Source: §6 · p. 10

Ethics Statement

The ethics statement describes safeguards for sensitive examples, including human annotation of test data, content warnings, limits on prompt realism, and oversight under research ethics guidelines.

Ethics Statement

  • All FalseReject-Test examples were human-annotated, and annotators were informed about sensitive content exposure and given resources to manage discomfort.Source: Ethics Statement · p. 10
  • The authors structured responses to reinforce safety reasoning and limited prompt complexity and realism to what they considered necessary for scientific validity.Source: Ethics Statement · p. 11
  • The paper says it includes content warnings and followed research ethics guidelines with continuous oversight by experienced researchers.Source: Ethics Statement · p. 11
  • The authors state that these measures seek to minimize risks while enabling research toward safer and more effective language models.Source: Ethics Statement · p. 11

A Implementation Details

The appendix reports model, training, and inference configurations, and includes a refusal-overlap figure whose row-normalized matrix compares model refusal sets.

A Implementation Details

  • Fine-tuning uses three epochs, batch size 8, bfloat16, AdamW, and a 1 × 10−5 learning rate with warmup and cosine decay.The training cut-off length is 5000 for reasoning models and 2048 for non-reasoning models; experiments use eight NVIDIA A100 40G GPUs.Source: A Implementation Details · p. 16
  • Inference uses temperature 0 for safety evaluations and 0.7 for general-ability evaluations, with maximum output lengths of 1024 tokens for non-reasoning and 8192 for reasoning models.Source: A Implementation Details · p. 16
  • Figure 5’s non-symmetric, row-normalized matrix reports the percentage of each row model’s refused prompts also refused by each column model.Higher cell values indicate stronger agreement for that row model’s refusal set.Source: Figure 5 · p. 16

B Details of FalseReject Dataset

FalseReject draws on safety-related datasets and organizes its coverage around 44 safety topics. The source datasets and their licenses are listed in Table 3.

B Details of FalseReject Dataset

C Refusal Consistency Analysis

The analysis compares refusal sets across representative LLMs using a non-symmetric overlap measure. The reported overlaps vary across model pairs, indicating differences in refusal behavior.

C Refusal Consistency Analysis

  • The analysis computes a normalized overlap for every model pair, producing a non-symmetric matrix of refusal consistency and discrepancies.The models analyzed include GPT-4.5, Claude-3.7-Sonnet, Gemini-2.5-Pro, Llama3.3-70B, Phi-4, Qwen2.5-32B, and Deepseek-V3.Source: C Refusal Consistency Analysis · p. 18
  • Claude-3.7-Sonnet refuses about 67% of prompts refused by GPT-4.5, while GPT-4.5 refuses approximately 77% of prompts refused by Claude-3.7-Sonnet.The overlap measure is directional: it calculates the share of one model’s refused prompts also refused by another.Source: C Refusal Consistency Analysis · p. 18
  • Qwen2.5-32B and Deepseek-V3 have refusal overlaps above 80%, and their refusal sets largely represent subsets of prompts other models recognize as problematic.The passage reports that some models are substantially more consistent with each other than others.Source: C Refusal Consistency Analysis · p. 18

D Detailed Results on Other Over-refusal Benchmarks

This section compares model rejection rates on existing over-refusal datasets. It describes the sampling and evaluation procedure and presents the comparison in Table 4.

D Detailed Results on Other Over-refusal Benchmarks

  • For datasets with more than 1,000 samples, the evaluation randomly samples 1,000 instances.Source: D Detailed Results on Other… · p. 18
  • The section uses the same LLM-as-a-judge framework as Röttger et al. (2024) to compute refusal rates.Source: D Detailed Results on Other… · p. 18
  • Table 4 presents a comparison of model rejection rates across existing over-refusal datasets.The supplied passage identifies the table’s scope but does not provide its cell values.Source: Table 4 · p. 18

E Additional Jailbreak Robustness Evaluation

Fine-tuning with FalseReject is reported to preserve or improve jailbreak robustness, and human labels largely agree with the automatic judge. The appendix also details prompt roles and human curation procedures.

E Additional Jailbreak Robustness Evaluation

F Human Validation Details

  • 92.5% of majority-vote human labels agree with the automatic judge across 500 outputs assessed by nine annotators.The average annotation time was 72.80 seconds per sample.Source: F Human Validation Details · p. 19

G Prompt Design

  • The entity-extraction prompt asks for candidate entity lists, then has a second prompt select the best-fitting list.The prompt requests a JSON candidate index.Source: G Prompt Design · p. 20
  • The Generator crafts benign prompts that appear unsafe, while the Discriminator judges their actual safety and gives feedback for subsequent attempts.The Generator is instructed to use the Discriminator’s prior justification to make later prompts appear more unsafe while remaining harmless.Source: G Prompt Design · p. 21
  • The Orchestrator judges whether a prompt is a good over-refusal example: superficially alarming but ultimately benign and without genuine harm.The prompt requires a yes-or-no judgment with a brief explanation.Source: G Prompt Design · p. 22

H Human Annotation Interface

  • FalseReject-Test retains data points meeting the specified annotation choices for Questions 1–4 and aggregates annotations from three annotators per point.The interface used for this human annotation is shown in Figures 9 and 10.Source: H Human Annotation Interface · p. 23

Related papers

Loading 2505.08054v2…