Source-linked AI summary
No One Model Catches Every Harm: Benchmarking Content Moderation Across Safety Scenarios
Afshin Orojlooyjadid, Hitesh Patel
TL;DR
The paper addresses the unresolved question of which safety model fits which harmful-content scenario. It benchmarks 53 models across 11 datasets and four categories under prompt-only and prompt-response settings, finding that model strengths vary by category and conversational safety remains unsolved. The results provide a structured basis for model selection while identifying multilingual generalization and long-tail coverage as limitations.
Problem
Practitioners lack systematic evidence for choosing specialized moderators, frontier LLMs, or compact alternatives for particular harms.
Method
The study benchmarks 53 models on 11 datasets organized into four challenge categories under prompt-only and prompt-response settings.
Results
Model strengths vary by harm category and setting: frontier general-purpose models lead on jailbreaks, specialized models lead when responses are visible, and conversational safety remains unsolved.
Takeaways & Limitations
Model choice should consider harm type, response visibility, and over-refusal costs rather than parameter count alone.
Takeaways & Limitations
The evaluation is entirely English-language and caps each dataset at 1,000 samples, limiting evidence about multilingual settings and real-world long-tail harms.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are increasingly deployed in real-world applications, yet they remain vulnerable to generating harmful content. From adversarial jailbreaks that bypass safety filters to implicit hate that evades detection, the range of risks these models pose continues to grow. While both specialized content moderators and general-purpose LLMs are being used as safety layers, the question of which model is best suited for which type of harmful content remains unanswered. We present the most comprehensive evaluation of LLM safety capabilities to date, systematically testing \textbf{53} models across \textbf{11} datasets that we organize into four distinct categories. Our evaluation under both prompt-only and prompt-response settings uncovers critical blind spots: large frontier models that lead on one category fall significantly behind smaller, specialized alternatives on others, and real-world conversational safety remains largely unsolved across all model families. These findings challenge the assumption that scale alone ensures safety, and provide the community with a structured framework for informed model selection.
1 Introduction
The paper addresses limited guidance on choosing among specialized moderators, frontier LLMs, and compact alternatives for different harms. It introduces a comprehensive benchmark organized by harm category and evaluation setting.
- LLMs are widely deployed but remain vulnerable to explicit and implicit harmful content, making moderation a production priority.
- Practitioners lack systematic guidance on which moderation model best fits a given type of harm.
- The study presents what it describes as the largest benchmarking study of LLM safety to date.
- The evaluation categorizes datasets into adversarial jailbreak resistance, policy enforcement, over-refusal, and conversational safety across prompt-only and prompt-response settings.
- The benchmark covers 53 models and 11 datasets and includes an optimized safety prompt plus analysis for customer scenarios.
2 Related Work
Prior work spans diverse harm-detection datasets and dedicated moderation models, but comparisons remain fragmented. This leaves practitioners without a landscape-level view of model suitability across harms.
- Existing datasets cover explicit toxicity, adversarial and implicit harms, and over-refusal, alongside dedicated moderation models.
- Each model is typically evaluated against a narrow set of baselines rather than across a common landscape.
- The resulting evidence does not show practitioners which model best handles which type of harm.
3 Methodological Framework
The benchmark evaluates 53 moderation models on 11 datasets spanning four harm categories under prompt-only and prompt-response settings. It includes both specialized moderators and general-purpose models while preserving specialized models’ native input formats.
- 53 models are evaluated, including specialized moderation models and commercial or open-source LLMs using a Llama-Guard-style prompt.
- 11 datasets are selected across easy, medium, and hard-to-detect harmful content, with each dataset capped at 1,000 records.
- The study uses binary Safe and Unsafe labels to accommodate differing dataset label definitions and enable comparison with prior work.
- C1: Adversarial & Jailbreak Resistance: C1 covers adversarial jailbreak resistance, with harmaug, harmbench, and xrtest emphasizing adversarial prompts, red-teaming, and jailbreak attempts.
- C2: Standard Policy Enforcement / C3: Refusal & Over-Refusal: C2 covers standard policy enforcement, while C3 covers refusal and over-refusal on benign or tricky prompts where false positives may occur.
4 Results
Across four safety categories, model performance depends on the harm type and whether the model response is visible: frontier models lead some prompt-only tasks, specialized moderators lead many response-aware tasks, and conversational safety remains difficult.
- Overall results: On Q, large commercial models achieve the highest overall average F1, reaching 75–77% across datasets.
- Overall results: On QA, Llama-Guard-3-8B leads average performance at 78.6%, followed closely by BingoGuard-Llama-8b and Gemma-2-27b-it.
- C1: Adversarial & Jailbreak Resistance: C1 adversarial and jailbreak datasets are difficult in Q, with average F1 scores of 52.7–61.2%, while response visibility makes QA easier for most models.
- C2: Standard Policy Enforcement: Specialized moderators lead C2 policy enforcement, including BingoGuard-Llama-8B at 87.1% and BingoGuard-Phi3-3B at 82.2% on standard safety benchmarks.
- C3: Refusal & Over-Refusal: C3 exposes a utility trade-off: 15 models exceed 95% F1 on Q, but balanced moderation requires avoiding false positives on benign prompts.The xstest and simplesafety datasets serve as a low-bar sanity check for utility alongside aggressive safety detection.
- C4: Conversational & Real-world Safety: C4 conversational safety is the hardest category, averaging approximately 52% F1 in both Q and QA.On QA, BingoGuard models exceed 63%, leading all models by at least 2%.
5 Conclusion
The benchmark finds that no single moderation model performs best across all safety scenarios, making harm type, response visibility, and over-refusal cost important for model selection.
- No single model wins across all scenarios: frontier LLMs lead on jailbreaks, specialized safety models dominate when responses are visible, and conversational safety remains unsolved.
6 Limitations
The study is limited by its English-only evaluation, capped dataset sizes, and binary labeling convention, which may omit multilingual, long-tail, severity, and multi-label phenomena.
- English-only evaluation leaves generalization to multilingual and code-switched harmful content unclear.Harmful content may manifest differently across these settings.
- Capping each dataset at 1,000 samples may underrepresent long-tail harmful-content patterns in real-world deployments.
- Binary safe/unsafe labels enable cross-benchmark comparability but discard severity signals and multi-label nuance.The convention follows prior content-moderation work because alternative schemes are not consistently available across all 11 datasets.
7 Ethics Statement
The study aims to help practitioners select moderation models transparently while evaluating publicly released systems without retraining or redistributing harmful content.
- The goal is a transparent cross-model comparison that helps practitioners select moderation models for specific deployments.
- Models are evaluated in their publicly released form without modification, fine-tuning, or retraining.Closed-source models are accessed through official APIs.
- The study uses licensed datasets and reports aggregate metrics rather than per-prompt harmful-content outputs.
A Extended Related Work
Prior work introduced diverse harm datasets and dedicated moderators, but evaluations remained narrow; this study instead compares existing model families across four challenge categories and both Q and QA settings.
- Evolution of Harm Detection Datasets: The benchmark spans datasets covering explicit toxicity, adversarial and implicit harms, policy enforcement, and over-refusal.
- Evolution of Dedicated Moderation Models: Dedicated moderators evolved from input-output safeguarding toward unified moderation tools and severity-based moderation.
- Prior Benchmarking Efforts: Prior moderation studies typically validated new models against limited baselines and narrow benchmark scopes.
- Study Scope: The study compares 53 models across 11 datasets organized into four challenge categories under Q and QA settings.
- Evaluation Design: QA uses provided dataset responses rather than newly generated responses because their quality would require human validation.
- Dataset Sampling: Datasets larger than 1,000 records are sampled to 1,000 while preserving the original harmful/safe ratio.
- Statistical Analysis: Paired bootstrap confidence intervals compare rank-1 and rank-2 models on shared resamples, with tie tiers identifying statistically indistinguishable leaders.Only 4 of 18 cases show a statistically significant top-1 versus rank-2 gap.
B.4 Reasoning Effort Ablation
The ablation examines reasoning effort, latency–accuracy tradeoffs, and prompt-template design for moderation evaluation; results favor structured prompting and small specialized models, while some comparisons remain limited.
- Reasoning Effort: Reasoning effort from None to High yields only around 1% total average F1 gain, without a significant effect.
- Latency–Accuracy Tradeoff: Small specialized moderators occupy the fastest–most-accurate region, while larger models add latency without commensurate accuracy gains.
- Prompt Template: The proposed prompt adds role assignment, borderline-handling guidance, and strict output formatting to the Llama Guard template.
- Prompt Ablation: The proposed prompt improves tested open-source models by 4–15% over the Llama Guard template and 18–28% over the simple prompt.Improvements are most pronounced on adversarial datasets such as HarmAug and XRTest.
- Scope Boundary: The prompt ablation covers only three open-source models because of API cost constraints.Transfer of observed gains to closed-source frontier models is expected but left for future empirical verification.
C Averaged Result per Dataset
This section presents averaged evaluation metrics across datasets for prompt-only and prompt-response moderation settings, alongside prompt-template ablations for three open-source models.
- Tables 9 and 10 average precision, recall, F1, and accuracy across all datasets for Q and QA modes.These averages support comparisons emphasizing recall or precision rather than F1 alone.
- Tables 7 and 8 compare macro F1 across datasets for three open-source models under prompt-template ablations in Q and QA modes.
D Detailed Result per Dataset
This section provides detailed per-dataset macro F1 results for content moderation models in Q and QA modes, with color sorting intended to support model comparison and selection.
- Tables 11 and 12 report macro F1 scores for each content moderation model across all datasets.Table 11 covers Q mode, while Table 12 covers QA mode.
- The scores are color sorted, with green indicating higher scores and red indicating lower scores.
- Tables 9 and 10 provide corresponding averages of precision, recall, F1, and accuracy across datasets for Q and QA modes.
- Table 11 gives per-dataset macro F1 scores when only the prompt is evaluated.
- Table 12 gives per-dataset macro F1 scores when both the question and response are evaluated.