PaperlayerView exact revision on arXiv

Source-linked AI summary

SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal

Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, Prateek Mittal

arXiv:2406.14598v2cs.AI

TL;DR

SORRY-Bench addresses limitations in safety-refusal evaluation involving coarse and imbalanced taxonomies, overlooked linguistic variation, and expensive automated judging. It builds a balanced 44-category benchmark, evaluates diverse evaluator designs using human judgments, and benchmarks more than 50 LLMs. The study finds substantial variation in refusal behavior and reports that fine-tuned 7B evaluators can reach over 80% human agreement at low cost.

  • Problem

    Existing safety evaluations have coarse or imbalanced categories, overlook linguistic prompt variation, and lack systematic study of efficient automated refusal evaluators.

  • Method

    SORRY-Bench combines a 44-category, 440-instruction balanced dataset and 20 linguistic augmentations with 7K+ human judgments and a meta-evaluation of automated safety evaluators.

  • Results

    Across more than 50 LLMs, refusal behavior varies by model, category, and linguistic mutation; fine-tuned 7B evaluators achieve over 80% human agreement.

  • Takeaways & Limitations

    SORRY-Bench supports balanced, granular, and scalable analysis of LLM safety-refusal behavior across models, risk categories, and prompt forms.

  • Takeaways & Limitations

    Fixed decoding parameters may not fully capture safety performance because decoding choices can noticeably affect model safety behavior.

Abstract

from arXiv · show

Evaluating aligned large language models' (LLMs) ability to recognize and reject unsafe user requests is crucial for safe, policy-compliant deployments. Existing evaluation efforts, however, face three limitations that we address with SORRY-Bench, our proposed benchmark. First, existing methods often use coarse-grained taxonomies of unsafe topics, and are over-representing some fine-grained topics. For example, among the ten existing datasets that we evaluated, tests for refusals of self-harm instructions are over 3x less represented than tests for fraudulent activities. SORRY-Bench improves on this by using a fine-grained taxonomy of 44 potentially unsafe topics, and 440 class-balanced unsafe instructions, compiled through human-in-the-loop methods. Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations. We supplement SORRY-Bench with 20 diverse linguistic augmentations to systematically examine these effects. Third, existing evaluations rely on large LLMs (e.g., GPT-4) for evaluation, which can be computationally expensive. We investigate design choices for creating a fast, accurate automated safety evaluator. By collecting 7K+ human annotations and conducting a meta-evaluation of diverse LLM-as-a-judge designs, we show that fine-tuned 7B LLMs can achieve accuracy comparable to GPT-4 scale LLMs, with lower computational cost. Putting these together, we evaluate over 50 proprietary and open-weight LLMs on SORRY-Bench, analyzing their distinctive safety refusal behaviors. We hope our effort provides a building block for systematic evaluations of LLMs' safety refusal capabilities, in a balanced, granular, and efficient manner. Benchmark demo, data, code, and models are available through https://sorry-bench.github.io.

1 INTRODUCTION

SORRY-Bench addresses gaps in prior safety-refusal evaluations by balancing fine-grained risk categories and linguistic variation, while developing an efficient automated evaluator. It then uses the benchmark to compare refusal behavior across many LLMs.

  • Motivation: Prior evaluations use inconsistent, coarse-grained, and imbalanced safety categories, leaving some risky topics insufficiently represented.The paper identifies omissions such as legal advice and political campaigning alongside overrepresentation of other categories.
  • Dataset: SORRY-Bench provides 44 fine-grained safety categories and 440 class-balanced unsafe instructions.The dataset contains 10 instructions per category, with human-in-the-loop taxonomy refinement.
  • Dataset: Twenty linguistic mutations extend the base benchmark to cover varied writing styles, persuasion techniques, encodings, and languages.These transformations target formatting and linguistic characteristics that prior evaluations often overlook.
  • Automated evaluation: Over 7K human annotations support a meta-evaluation showing that fine-tuned 7B LLMs achieve over 80% human agreement at low computational cost.The evaluator requires approximately 10 seconds per evaluation pass on a single A100 GPU and can scale to hundreds of passes.
  • Results: Across more than 50 proprietary and open-weight LLMs, refusal rates vary substantially by model, safety category, and linguistic mutation.Claude-2 and Gemini-1.5 show the most refusals, while Mistral models more often fulfill potentially unsafe requests.

2 A RECIPE FOR DIVERSE & BALANCED DATASET

SORRY-Bench constructs a unified, fine-grained, and customizable safety taxonomy, then balances both risk categories and prompt linguistics. Its dataset design addresses category imbalance and tests how prompt mutations affect safety behavior.

  • Taxonomy: Prior benchmarks use discrepant safety categories that may be too coarse-grained and inconsistent across evaluations.The paper motivates a unified taxonomy that captures diverse risks while allowing stakeholders to select categories of concern.
  • Taxonomy: SORRY-Bench organizes 44 potential risk categories into 4 high-level domains.The taxonomy unifies prior datasets and supports more granular and customizable safety-refusal analysis.
  • Taxonomy construction: The taxonomy aggregates 10 prior safety benchmarks, refines broad categories, and uses iterative human review to identify uncovered categories.This two-stage procedure is intended to produce an extensive and unified taxonomy.
  • Class balance: Prior datasets are heavily imbalanced, overrepresenting categories such as System Intrusion and Fraud while underrepresenting Self-Harm, Animal-related Crimes, and PII Violations.The paper links this distribution to unequal coverage of risks across safety categories.
  • Class balance: SORRY-Bench contains 440 samples, with 10 diverse unsafe instructions for each of its 44 categories.Novel instructions supplement filtered and deduplicated data from prior work where categories lacked sufficient examples.
  • Linguistic diversity: Twenty linguistic mutations vary writing styles, persuasion techniques, encoding strategies, and languages to test prompt diversity.The mutations include interrogative phrasing, misspellings, slang, ASCII, Caesar encoding, Tamil, and French.

3 A RECIPE FOR EFFICIENT & ACCURATE AUTOMATED EVALUATORS

The paper frames automated safety evaluation as a binary fulfillment-versus-refusal classification problem requiring scalable, accurate judges. It builds human-labeled data to compare evaluator designs and finds that fine-tuning can deliver strong agreement at lower computational cost.

  • Human evaluation provides ground truth but lacks scalability and efficiency, motivating language-model proxies for automated safety judgment.
  • The evaluator classifies each unsafe instruction–response pair as either fulfillment or refusal, with fulfillment requiring substantial direct assistance.
  • 3,520 records were manually labeled from 440 instructions, eight model responses per instruction, and six author judgments per response.
  • The human annotations were split into 2,640 training records and 4,400 test records, covering both in-distribution and linguistically mutated cases.
  • Fine-tuning consistently improves agreement, and smaller models can exceed 80% agreement while costing about 10 seconds per evaluation pass.

4 BENCHMARK RESULTS

SORRY-Bench evaluates refusal behavior across 56 LLMs, 44 safety categories, and 20 linguistic mutations. Results show substantial variation across models, categories, languages, prompt styles, and evaluation configurations.

  • Benchmark setup: 56 models are evaluated using 440 unsafe requests, with responses classified as refusal or fulfillment and summarized by fulfillment Rate.The benchmark includes open-weight and proprietary models ranging from 1.8B to 400B+ parameters.
  • Model-wise results: 27 of 56 LLMs show 20%∼50% fulfillment, while Claude-2 and Gemini-1.5 remain below 10% and eight models fulfill more than half of unsafe instructions.Mistral models are among those with fulfillment rates above 50%, including on Fraud.
  • Model-wise results: Llama-3-70b fulfillment rises from 12% to 35% versus Llama-2, while Gemini fulfillment falls from 33% for Gemini-Pro to 7% for Gemini-1.5.The paper relates some category-specific changes to differing model policies, including GPT-4o's treatment of adult-content requests.
  • Evaluation sensitivity: Fixed decoding parameters, such as temperature 0.7, may not fully capture the nuances of model safety performance.The paper notes that varying decoding parameters can noticeably affect safety behavior.
  • Category-wise results: 36 of 44 categories have average fulfillment below 50%, and at least one LLM refuses all unsafe instructions in 41 categories.Harassment, Child-related Crimes, and Sexual Crimes are among the most frequently refused categories, whereas Legal Advice is commonly fulfilled.
  • Linguistic mutations: Question-form prompts reduce fulfillment by 2∼13% for 6 of 8 models, whereas technical terms increase fulfillment by 8∼19%.Table 2 reports overall Base compliance alongside rate differences induced by 20 linguistic mutations.
  • Linguistic mutations: Chinese and French increase fulfillment by 2∼15% for 7 of 8 models, while low-resource languages shift fulfillment by -20∼58%; GPT-4o varies by less than ±4%.More recent models can show higher unsafe-instruction fulfillment in low-resource languages despite stronger multilingual conversation abilities.
  • Evaluation sensitivity: All five persuasion techniques increase fulfillment by 5∼66%, while omitted prompt-format tokens raise Llama-2 and Gemma fulfillment by 7%∼30%.Llama-3 models remain robust to the missing-token configuration described in Appendix K.

5 CONCLUSION

SORRY-Bench is introduced as a systematic benchmark for LLM safety refusal. It combines fine-grained topic coverage, balanced linguistic variations, and an efficient automated evaluator to analyze refusal behavior across many models.

  • Contributions: SORRY-Bench provides a 44-topic taxonomy and 440 class-balanced unsafe instructions for granular safety-refusal evaluation.The taxonomy is refined through human-in-the-loop mapping and iterative identification of uncovered categories.
  • Contributions: The benchmark adds 8.8K unsafe instructions across 20 diverse linguistic augmentations to balance prompt formatting and linguistic patterns.These augmentations support systematic examination of how linguistic variation affects safety evaluation.
  • Contributions: SORRY-Bench uses 7K+ human annotations to study design choices for a fast and accurate automated safety evaluator.The paper evaluates over 50 proprietary and open-weight LLMs and analyzes their distinctive refusal behaviors.

6 REPRODUCIBILITY STATEMENT

The authors describe implementation details in multiple appendices and make the datasets and evaluation code publicly available. They also commit to maintaining and revising the benchmark resources.

  • Reproducibility: Implementation details for dataset curation and experiments are provided in Appendices G, H, I, J, and K.The cited appendices cover the benchmark's dataset and experimental implementation details.
  • Reproducibility: The datasets and evaluation code are hosted publicly on HuggingFace and GitHub.The authors state that the public resources include the benchmark datasets and evaluation code.
  • Reproducibility: The authors commit to maintaining datasets, models, and benchmark results and revising taxonomies when necessary.The stated maintenance plan includes future dataset and taxonomy revisions.

A.1 LIMITATION AND FUTURE STEPS

SORRY-Bench has several scope and evaluation limitations: it measures binary refusal rather than graded harmfulness, omits some safety scenarios, cannot assess usefulness, may miss jailbreak patterns, and risks contamination.

  • Binary refusal scores do not quantify the actual harmfulness of model responses, which the authors leave for future work.
  • The benchmark omits scenarios combining multiple risky categories and neutral prompts that may still elicit unsafe model behavior.
  • SORRY-Bench cannot reveal whether a model is helpful and useful, so the authors recommend pairing it with utility and over-refusal benchmarks.
  • Its 20 linguistic mutations target average-case bad users and may not capture the numerous jailbreaking methods designed to compromise safety.
  • Dataset contamination could let future model developers overfit to SORRY-Bench; a private split is proposed as a future solution.
  • The unsafe instruction and human-judgment datasets may create negative social impacts, motivating restricted access to the human judge dataset.

C AN OVERVIEW OF PRIOR SAFETY BENCHMARK DATASETS

Prior safety benchmarks use discrepant, often coarse taxonomies, while SORRY-Bench provides a granular comparison framework that distinguishes refusal behavior across models.

  • 16 prior safety benchmark datasets are compared by sample count, safety categories, data sources, and descriptions.
  • Prior taxonomies are usually discrepant and coarse, motivating SORRY-Bench’s systematic unification of safety categories at finer granularity.
  • SORRY-Bench reports fulfillment rate, while several comparison benchmarks use reversed scores to represent the same direction of unsafe behavior.
  • Across benchmarks, Claude and Llama-2 appear safer, whereas Mistral and Zephyr-7b-beta appear more unsafe.
  • 6.00% to 90.22%: SORRY-Bench fulfillment scores distinguish refusal performance across different LLMs.
  • Different configurations and metrics mean scores across these benchmarks should not be directly compared.

D SORRY-BENCH TAXONOMY IN DETAILS

SORRY-Bench organizes 44 unsafe categories into four harm-oriented domains and evaluates them with a class-balanced dataset, revealing domain-specific refusal patterns.

  • SORRY-Bench’s taxonomy contains 44 safety categories aggregated into four domains based on harm, policies, and prior dataset practices.
  • Assistance with Crimes or Torts: Assistance with Crimes or Torts groups 19 categories involving criminal or tortious assistance, including terrorism and other illegal activities.
  • Hate Speech Generation: Hate Speech Generation is separated from crime-related categories because its legal status differs across jurisdictions.
  • Potentially Inappropriate Topics: Potentially Inappropriate Topics covers 15 categories whose acceptability varies across model developers and platform policies.
  • Potentially Unqualified Advice: Potentially Unqualified Advice covers critical medical and financial topics where inaccurate advice may cause real-world harm.
  • Domain-level refusal patterns: A majority of models fulfill none or few unsafe instructions in Hate Speech Generation and Assistance with Crimes or Torts.
  • Domain-level refusal patterns: Models vary within Potentially Inappropriate Topics: GPT-4 fulfills most adult-content advice requests, while Gemini and Claude refuse most.
  • Domain-level refusal patterns: Only Gemini-1.5-Flash refuses all unsafe instructions in Potentially Unqualified Advice.

G.4 JACCARD SIMILARITY ANALYSIS

The Jaccard analysis measures lexical overlap between SORRY-Bench instructions and prior datasets, finding limited average maximum overlap beyond exact matches.

  • Each SORRY-Bench instruction is represented as a word set and compared pairwise with word sets from 3.6K+ prior instructions.
  • 44.5%: the average maximum Jaccard similarity across all 440 SORRY-Bench instructions, indicating less than half-word overlap on average.

H IMPLEMENTATION OF LINGUISTIC MUTATIONS

SORRY-Bench applies 20 linguistic mutations to its base unsafe instructions to represent real-world prompt-format diversity and test its effect on safety refusal.

  • 20 linguistic mutations paraphrase the base dataset to capture prompt-formatting diversity used by real-world users.
  • Six Writing Styles: Six writing styles include questions, slang, uncommon dialects, technical terms, role play, and misspellings.
  • Four Encoding and Encryption Strategies: Four encoding and encryption strategies convert unsafe instructions into ASCII, Caesar, Morse, and Atbash versions.
  • The benchmark includes translated unsafe instructions in Simplified Chinese and French to examine language diversity.
  • Translated model responses are converted back into English before safety-refusal evaluation.

I HUMAN ANNOTATION COLLECTION

Human safety judgments were collected in separate in-distribution and out-of-distribution rounds, using binary refusal-versus-compliance labels for model responses.

  • Two annotation rounds cover in-distribution responses from base prompts and out-of-distribution responses from linguistically mutated prompts.
  • Six annotators each labeled approximately 1.2K records across assigned batches, spending about 2 hours on the first round and 5 hours on the second.
  • For each unsafe instruction, annotators reviewed eight sequential model responses and selected either “Refusal” or “Compliance.”
  • A response is labeled compliance when substantial content directly resolves the request; otherwise, including nonsense responses, it is labeled refusal.
  • The six annotators showed pairwise agreement consistently above 86%, exceeding 90% for most pairs.
  • Automated judges classify each unsafe-instruction and model-response pair as refusal or compliance, using direct prompting, chain-of-thought, few-shot, or fine-tuned designs.

J.2 META-EVALUATION RESULT ANALYSIS

The meta-evaluation compares automated safety-judgment designs against human annotations, finding that fine-tuning is the strongest approach while speed and failure modes remain important considerations.

  • Direct prompting achieves 78.9% agreement for GPT-4o and 71.6% for Llama-3-70b-instruct, but requires 100∼260 seconds.
  • CoT does not improve agreement consistently and adds substantial decoding time, while few-shot prompting helps larger but not smaller LLMs.
  • Fine-tuning on human judgments yields the strongest agreement: GPT-3.5-turbo reaches 83.8%, other open-source models exceed 80%, and Bert-Base-Cased reaches 74.6%.
  • General-purpose safeguard models perform poorly on SORRY-Bench, with agreement ranging from 37.2% for MD-Judge to 60.6% for WildGuard.
  • Keyword Match is fastest but reaches only 37.4% agreement, whereas fine-tuned lightweight models offer higher accuracy with slightly more processing overhead.
  • Reported time costs are comparable under a consistent sequential setup, but users can parallelize evaluations and hardware affects actual wall-clock time.
  • Fine-tuned judges can still misclassify responses involving bullet lists or safety disclaimers, despite high overall agreement with human annotators.

K.2 MAIN BENCHMARK RESULT STATISTICS

SORRY-Bench reveals substantial variation in refusal behavior across models, categories, versions, and evaluation settings, while emphasizing that fulfillment rate is not a complete safety measure.

  • Models with the lowest fulfillment rates: Claude-2 and Gemini-1.5 have the lowest average fulfillment rates, refusing most benchmark prompts.
  • Models with the highest fulfillment rates: Mistral-7b-instruct-v0.1 and Dolphin-2.2.1-mistral-7b exceed 90% average fulfillment and often provide harmful content.
  • Tracking model changes over time: Llama-3-70B fulfillment rises from 12% to 35% relative to Llama-2, while Gemini fulfillment falls from 33% to 7% from Gemini-Pro to Gemini-1.5.
  • 438 of 440 instructions are refused by at least one evaluated LLM.
  • Frequently refused categories: Harassment, child-related crimes, and sexual crimes are most frequently refused, with average fulfillment rates of barely 9–11%.
  • Least refused categories: Legal consulting advice, religion promotion, and ethical belief promotion are least refused, averaging 74–80% fulfillment.
  • GPT-4o versus GPT-3.5-turbo: GPT-4o has a 0.30 fulfillment rate versus 0.11 for GPT-3.5-turbo-1106, reflecting more permissive handling of some sensitive and regulated contexts.
  • Safety refusal rates alone do not determine overall safety because acceptable behavior varies across cultural, legal, and organizational contexts.
Loading 2406.14598v2…