PaperlayerView exact revision on arXiv

Source-linked AI summary

OR-Bench: An Over-Refusal Benchmark for Large Language Models

Justin Cui, Wei-Lin Chiang, Ion Stoica, Cho-Jui Hsieh

arXiv:2405.20947v5cs.CLcs.AI

TL;DR

Safety alignment can make LLMs reject benign prompts, while systematic measurement is difficult because suitable over-refusal prompts are hard to construct. The paper addresses this gap with an automated generation pipeline and OR-Bench, finding a safety–over-refusal trade-off across evaluated models.

  • Problem

    Safety alignment can cause LLMs to reject benign prompts, but systematically measuring over-refusal is difficult because suitable prompts are hard to construct.

  • Method

    The paper automatically rewrites harmful prompts into benign prompts, moderates them, and uses the resulting datasets to evaluate LLM over-refusal.

  • Results

    Most evaluated models achieve toxic-prompt rejection at the expense of over-refusal, rarely excelling at both safety and helpfulness.

  • Takeaways & Limitations

    OR-Bench provides a testbed for studying and improving the trade-off between safety and helpfulness.

  • Takeaways & Limitations

    Figure 1 results are amplified by ensemble rejections, so OR-Bench-80K results are a better indicator of normal use.

Abstract

from arXiv · show

Large Language Models (LLMs) require careful safety alignment to prevent malicious outputs. While significant research focuses on mitigating harmful content generation, the enhanced safety often come with the side effect of over-refusal, where LLMs may reject innocuous prompts and become less helpful. Although the issue of over-refusal has been empirically observed, a systematic measurement is challenging due to the difficulty of crafting prompts that can elicit the over-refusal behaviors of LLMs. This study proposes a novel method for automatically generating large-scale over-refusal datasets. Leveraging this technique, we introduce OR-Bench, the first large-scale over-refusal benchmark. OR-Bench comprises 80,000 over-refusal prompts across 10 common rejection categories, a subset of around 1,000 hard prompts that are challenging even for state-of-the-art LLMs, and an additional 600 toxic prompts to prevent indiscriminate responses. We then conduct a comprehensive study to measure the over-refusal of 32 popular LLMs across 8 model families. Our datasets are publicly available at https://huggingface.co/bench-llms and our codebase is open-sourced at https://github.com/justincui03/or-bench. We hope this benchmark can help the community develop better safety aligned models.

1. Introduction

The paper introduces OR-Bench to measure over-refusal, addressing the tendency of safety-aligned LLMs to reject benign prompts. It provides large-scale datasets and evaluates models to characterize the safety–helpfulness trade-off.

  • Safety alignment can cause LLMs to reject prompts that should receive helpful answers, creating an over-refusal problem.
  • OR-Bench is presented as the first large-scale benchmark specifically designed to test over-refusal in LLMs.
  • 80,000 safe prompts span 10 harmful categories, supplemented by a challenging OR-Bench-Hard-1K subset.
  • Most models improve toxic-prompt rejection at the expense of rejecting more safe prompts, rarely performing well on both objectives.
  • The study evaluates 32 open-source and black-box LLMs across 8 model families.

2. Related Work

Prior work develops safety alignment, refusal mitigation, and jailbreak defenses, but safety improvements can increase rejection of benign prompts. OR-Bench is positioned as a benchmark for measuring this trade-off.

  • Safety Alignment: Safety-alignment methods aim to produce truthful, helpful, and safe outputs through techniques such as RLHF, self-instruction, and related training procedures.
  • Model Refusal: Existing refusal research includes datasets and strategies for identifying toxic content and reducing harmful or offensive chatbot behavior.
  • Over Refusal: Prior studies report that safety training may make models overly cautious, rejecting safe prompts that resemble unsafe ones and reducing helpfulness.
  • Jailbreak Defense: Jailbreak-defense methods address attacks that induce unsafe behavior in LLMs.
  • Jailbreak Defense: Evaluating defense algorithms on OR-Bench shows that many achieve high defense success while significantly increasing over-refusal rates.

3. Over-Refusal benchmark

OR-Bench is built through an automated pipeline that generates toxic seeds, rewrites them into borderline prompts, and filters them with moderation. The resulting datasets support large-scale evaluation of over-refusal.

  • Benchmark pipeline: The benchmark-generation pipeline consists of toxic-seed generation, toxic-seed rewriting, and moderation-based filtering.
  • Toxic seed rewrite: Few-shot demonstrations improve rewrite quality by encouraging more diverse benign prompts without explicit safety terms or fictional framing.
  • Prompt moderation: An ensemble of GPT-4-turbo, Llama-3-70B, and Gemini-1.5-pro retains only prompts judged safe by majority vote.
  • Prompt moderation: The pipeline uses an additional non-safety-moderated model to answer flagged prompts before reassessing them, helping recover prompts that moderators marked toxic because of keyword sensitivity.
  • Prompt moderation: The moderation ensemble achieves over 98% of human-expert performance, while a fine-tuned Mistral classifier reaches around 90%.
  • Benchmark construction: The released benchmark contains 80K prompts across 10 categories, including a hard subset selected for rejection by multiple large or recent models.

4. Experimental results

The evaluation benchmarks 32 models across 8 families and finds persistent safety–helpfulness trade-offs, with substantial variation across releases, families, and prompt categories. Newer models often reduce over-refusal, but some do so while accepting more toxic prompts.

  • Evaluation setup: 32 models across 8 families were evaluated using OR-Bench-80K, OR-Bench-Hard-1K, and OR-Bench-Toxic, with keyword matching and GPT-4-based evaluation.Keyword matching closely approximated GPT-4 evaluations, with discrepancies of 2.4% for GPT-3.5-turbo-0125 and 1.2% for Llama-3-70b on sampled datasets.
  • Overall trends: Newer models typically show lower over-refusal rates than predecessors, although safety–helpfulness trade-offs remain evident across model families.The figure compares over-refusal on OR-Bench-Hard-1K with toxic-prompt acceptance on OR-Bench-Toxic; smaller regions are better for both measures.
  • Overall trends: 0.89 Spearman correlation between safe- and toxic-prompt rejection rates indicates that safer models generally over-refuse more.The authors characterize this pattern as a trade-off, with few models breaking it.
  • Family comparisons: GPT-3.5-turbo-0301 rejects over 57% of OR-Bench-Hard-1K prompts, while GPT-3.5-turbo-0125 rejects only 62% of toxic prompts.The later GPT-3.5 release reduces rejection of safe prompts but is described as less safe; GPT-4 is safer while maintaining similarly low over-refusal.
  • Family comparisons: Claude models reject many safe prompts while rejecting most toxic prompts, whereas Mistral models reject few safe prompts but answer 20% more toxic prompts than Claude.The authors present Claude as among the safest tested families and Mistral as occupying the opposite trade-off direction.
  • Category analysis: Models exhibit category-specific sensitivities: GPT-3.5-turbo-0125 is sensitive to privacy, Mistral-large-latest to self-harm, and Qwen-1.5-72B to sexual and deception content.Llama-3-70b is sensitive to privacy and self-harm, while Claude-3-opus is comparatively less sensitive to sexual topics.

5. Ablation study

The ablation study examines how jailbreak defenses, system prompts, and temperature affect the balance between safety and over-refusal. Defense and system-prompt interventions generally increase both toxic-prompt rejection and benign-prompt rejection, while temperature has little effect on refusal behavior.

  • Jailbreak Defense: Figure 5 compares defense methods for GPT-3.5-turbo-0125 and Llama-3-70b on OR-Bench-Hard-1K and OR-Bench-Toxic.The comparison is intended to assess defense effects on both toxic-prompt rejection and over-refusal.
  • Jailbreak Defense: Most jailbreak defenses reject more toxic prompts but also increase rejection of benign prompts.In-Context Learning produces the highest rejection of both toxic and over-refusal prompts, while SmoothLLM slightly raises both rates.
  • System Prompt: System prompts instructing models to be helpful and safe substantially increase both safety and over-refusal across four evaluated models.For GPT-3.5-turbo-0125, toxic-prompt rejection rises by around 35% and benign-prompt rejection by around 55%; Mistral-large-latest rises by around 20% and 10%, respectively.
  • System Prompt: Figure 6 evaluates the effect of helpful-and-safe system prompts on OR-Bench-Hard-1K and OR-Bench-Toxic.The plotted changes show how system-prompt instructions alter safety behavior and benign-prompt rejection.
  • Different Temperatures: Temperature has little impact on refusal behavior for the two evaluated models, whose responses remain consistent across settings.The main experiments use temperature 0.0 for reproducibility, while Table 4 reports results at different temperatures on OR-Bench-Hard-1K.

6. Conclusion and future work

The paper concludes by presenting OR-Bench as a large-scale evaluation framework for over-refusal and by identifying future needs for broader coverage and more nuanced refusal assessment.

  • Conclusion: OR-Bench contains 80,000 over-refusal prompts, a 1,000-prompt hard subset, and 600 toxic prompts, and evaluates 32 models across 8 families.The benchmark is designed for ongoing updates as new models emerge to help prevent over-fitting.
  • Future Work: Future work will expand the benchmark with more models and categories and develop finer-grained definitions beyond binary refusal.The paper notes that models can respond without refusing while also withholding fully unethical or harmful answers.

Impact Statement

The impact statement frames OR-Bench as a benchmark intended to support more reliable and safety-aligned AI systems while emphasizing ethical dataset curation.

  • Impact Statement: OR-Bench evaluates over-refusal and highlights the trade-off between safety and helpfulness.The stated goal is to contribute to more reliable and safety-aligned AI systems.
  • Impact Statement: The work reports IRB approval and responsible dataset curation through ethical standards during data collection and moderation.

A. Limitations

The limitations concern evaluator bias, incomplete toxic-prompt filtering, and the scope of the proposed prompt-generation approach.

  • Evaluator Limitations: Evaluation by three LLM moderators may not fully reflect true performance because of fairness-related concerns.The paper states that empirical evidence suggests the bias impact is limited, but does not eliminate this concern.
  • Dataset Curation: Some toxic prompts may remain unidentified because large-scale moderation is difficult.The paper reports that the chance for a moderated prompt to be toxic is very small, while acknowledging possible missed cases.
  • Method Scope: The prompt-generation approach is one useful evaluation method rather than a claimed optimal method.

C. Detailed Experiment Setup

The experiment benchmarks models across families and frames over-refusal as a normal-use safety-alignment issue distinct from red-teaming. It also compares the toxic-prompt dataset with AdvBench.

  • 32 models from 8 model families are benchmarked, including black-box and open-source systems.
  • Existing model guidelines emphasize fulfilling user goals when possible, assuming best intentions, and refusing only when safe compliance is impossible.
  • Over-refusal concerns whether models help with safe, plausible requests in normal use, whereas red-teaming probes worst-case misuse scenarios.
  • Models show similar performance on AdvBench and OR-Bench-Toxic, with a Spearman’s ranking correlation of 0.991.

F. Model response evaluation

The evaluation measures refusal and acceptance behavior across categories and datasets, using model outputs and category breakdowns to characterize safety alignment.

  • Model responses are evaluated primarily through keyword matching and another LLM because the benchmark operates at large scale.
  • OR-Bench-Hard-1K contains more illegal and privacy-related prompts because models more often reject prompts from those categories.
  • Table 2 reports rejection results on OR-Bench-Hard-1K, while Tables 6 and 7 report toxic-prompt acceptance and full-benchmark rejection results.
  • Figure 8 compares rejection rates on over-refusal prompts with acceptance rates on toxic prompts, where smaller areas indicate better safety alignment.

L. Annotation Experiment

The annotation experiment compares independent human and ensemble-LLM judgments on a random sample of 100 tasks.

  • 100 randomly selected tasks are labeled independently by three workers, an LLM ensemble moderator, and one paper author.

M. Keywords Based Evaluation

The evaluation pipeline uses refusal keywords and model-based moderation to assess responses, while dataset construction and category definitions support large-scale prompt analysis.

  • Rejected responses are identified using recurring refusal-opening keywords such as “I’m unable to” and “I cannot assist.”
  • The benchmark evaluates both rejection of over-refusal prompts and acceptance of toxic prompts, with lower values indicating better alignment in the figure’s radar comparison.
  • Multiple rewritten prompts are generated from each toxic seed because single rewrites may remain toxic; temperature 0.7 is used to increase diversity.
  • OR-Bench-80K spans prompts with different toxicity levels, while OR-Bench-Hard-1K contains safe prompts rejected by at least three large models.
  • Moderators categorize prompts only when they involve intended or promoted harmful activity, excluding knowledge-understanding requests without such intent.
  • The category scheme includes hate, self-harm, sexual, violence, harassment, harmful, illegal, unethical, privacy, deception, and none of the above.
  • LLM moderators can be keyword-sensitive, so Mistral-7B-Instruct-v0.3 is used to answer prompts and help determine whether labeled-toxic prompts are genuinely toxic.

V. Experiment with human workers

The study compares human annotation with an ensemble moderator for identifying safe rewritten prompts and uses the results to justify automated moderation. It also evaluates whether audited data can support a classifier and examines difficult over-refusal prompts.

  • Moderation validation: Human workers and an expert labeled sampled rewritten prompts, with the ensemble majority vote defining the ground-truth labels for the confusion matrix.The comparison used 100 sampled tasks, three workers, and one paper author.
  • Moderation validation: 77.3% was the highest safe-prompt identification rate among workers, achieved by Worker 2.The worker results were used in the validation comparison against the ensemble moderator.
  • Moderation validation: 43% inter-worker agreement and a misclassified wealth question illustrate disagreement over whether controversial topics are safe to answer.Workers treated the question as illegal, whereas the paper describes it as asking about a controversial topic.
  • Moderation validation: The authors conclude that a state-of-the-art LLM ensemble performs better than human raters for over-refusal moderation.This conclusion motivates selecting LLMs rather than human workers as moderators.
  • Classifier validation: 0.83 binary safe/toxic accuracy surpassed the best human workers’ 0.78, while fine-grained classification reached 0.84, about 90% of human-expert performance.These results came from fine-tuning Mistral-7b-instruct-v0.2 on 1,200 audited samples for three epochs.
  • Hardness analysis: Prompt hardness varies substantially: stronger wording can trigger refusals without toxicity, and even better-calibrated GPT-4-family models can sometimes withhold safe answers.Examples include requests about confidential information and election tactics that can be answered without endorsing wrongdoing.
Loading 2405.20947v5…