PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety-Flag: A Unified Benchmark for the Reliability and Calibration of LLM Content Moderators

Yibo Hu

arXiv:2609.19072v1cs.CLcs.CYcs.LG

TL;DR

Content moderators face consequential false alarms and missed harmful content, while existing evaluations do not compare models across benchmarks and reliability dimensions together. Safety-Flag evaluates models on seven benchmarks using fixed, balanced items; it finds differing error profiles, overconfidence among general-purpose LLMs, and variation in confidence-based abstention.

  • Problem

    Existing evaluations typically report accuracy or F1 one benchmark at a time, leaving no common comparison of general-purpose LLMs and dedicated guards across error direction, calibration, and error ranking.

    Source: §1 · p. 1

  • Method

    Safety-Flag maps seven benchmarks to a balanced flag/do-not-flag protocol and evaluates general-purpose LLMs and dedicated guards on the same items, releasing item-level decisions and confidence scores.

    Source: §1 · p. 1

  • Results

    Across models, false-alarm and miss behavior differs, general-purpose LLMs are systematically overconfident, and confidence varies in how well it supports abstention; temperature scaling improves calibration without changing decisions or error ranking.

    Source: §1 · p. 2, §10 · p. 8

  • Takeaways & Limitations

    The released suite and leaderboard support reproducible, deployment-aware comparisons of content moderators.

    Source: §10 · p. 8

    The suite covers about 200 balanced English, single-turn, binary items per benchmark and reports aggregate error without group- or dialect-conditional breakdowns; multi-agent moderation is out of scope.

    Source: §7 · p. 7

Abstract

from arXiv · show

Large language models are increasingly used for content moderation, but most evaluations still report aggregate accuracy on individual benchmarks. We introduce Safety-Flag, which places seven widely used safety benchmarks (BeaverTails, XSTest, Ethics, WildGuard, Aegis, ToxiChat, and ToxiGen) into a single balanced flag / do-not-flag protocol. We release item-level decisions and confidence scores for six general-purpose LLMs and four dedicated guards, together with three reference models, evaluated on the same items. Safety-Flag measures three dimensions of moderator reliability: error direction, probability calibration, and confidence-based error ranking for human review. They often disagree. Aggregate accuracy does not reveal error direction: one model flags $85\%$ of benign content, whereas another misses $54\%$ of harmful content. All six general-purpose models are overconfident; fitting one temperature per model reduces calibration error by $2.8$--$6.0\times$ without changing predicted labels or confidence ordering. Confidence-based abstention lowers selective risk for every model, although the gains depend on how well confidence ranks errors. Dedicated guards produce fewer false alarms and are better calibrated, but several have higher miss rates outside their documented coverage. We release the benchmark, fixed item lists, evaluation code, per-item model outputs, and leaderboard at: https://github.com/yibo-hu-lab/safety-flag-benchmark.

1 Introduction

Aggregate accuracy and F1 omit error direction, confidence reliability, and the effect of routing low-confidence cases to human review. Safety-Flag unifies seven benchmarks and evaluates models on fixed, balanced items; results show distinct error profiles and reliability dimensions.

1 Introduction

  • Safety-Flag maps seven safety benchmarks to a balanced flag/do-not-flag protocol and releases item-level decisions and confidence scores for paired model evaluation.The study compares general-purpose LLMs, dedicated guards, and reference models on the same items.Source: §1 · pp. 1–2
  • Models range from systematic over-flagging to systematic under-flagging, and these profiles remain stable across seeds and leave-one-benchmark-out analyses.The opposite error directions are also illustrated by Llama and OLMo.Source: Figure 1 · p. 1, §1 · p. 2
  • Temperature scaling improves probability calibration without changing decisions or confidence ordering, while human-review benefits depend on how well confidence ranks errors.Across models, stated confidence exceeds actual accuracy, and reviewing the least-confident half lowers the error rate on the items left automated, with varying gains.Source: Figure 1 · p. 1, §1 · p. 2
  • Dedicated guards reduce false alarms and improve calibration, but several have higher miss rates beyond their documented coverage.Across models, strength on one reliability dimension can coexist with weakness on another.Source: §1 · p. 2

2 Related Work

Prior work addresses moderation benchmarks, calibration, and human escalation, but Safety-Flag brings general-purpose LLMs and dedicated guards together on fixed items and measures three reliability dimensions.

2 Related Work

  • Existing safety benchmarks probe different harm domains but are seldom unified for cross-benchmark comparison or assessed beyond accuracy and F1.Purpose-built moderation systems are also commonly evaluated on their own test sets.Source: §2 · p. 2
  • Calibration error and reliability diagrams are established classifier tools, while verbalized confidence and whether models know what they know are active LLM topics.Temperature scaling is described as a canonical post-hoc calibration fix [9].Source: §2 · p. 2
  • Prior calibration work includes audits of dedicated guards and escalation models that use confidence-related features for cost-aware selective classification.Liu et al. [18] study guard calibration and post-hoc fixes; Bachar et al. [2] learn an escalation meta-model.Source: §2 · p. 2
  • Safety-Flag extends calibration and escalation studies by evaluating general-purpose LLMs and dedicated guards on fixed items across error direction, calibration, and confidence-based error ranking.The comparison uses one protocol and includes a public per-item release.Source: §2 · p. 2
  • Selective prediction measures the coverage–risk trade-off when classifiers abstain; here, abstention routes moderation items to human reviewers and is compared with random abstention.The framework is classical [6], and conformal abstention has been applied to LLM hallucination [1].Source: §2 · p. 2

3 The Benchmark

The benchmark standardizes items, prompts, outputs, and reliability measures while retaining each source benchmark’s labels and content distinctions. It uses balanced samples and reports per-benchmark results to support paired comparisons.

3 The Benchmark

  • The suite spans broad harms, exaggerated-safety refusals, moral judgments, adversarial prompts, dialogue context, and implicit group-targeted toxicity.The seven source benchmarks are BeaverTails, XSTest, Ethics, WildGuard, Aegis, ToxiChat, and ToxiGen.Source: §3 · p. 2
  • Each of seven benchmarks contributes 198–200 fixed items with approximately equal harmful and benign counts, mapped to the same flag/do-not-flag decision.The source labels and exact item IDs are retained, and the balancing avoids rewarding a model simply for flagging everything.Source: §3 · pp. 2–3
  • At n≈100, macro-F1 rankings and every model’s error-direction sign are stable; the ECE ranking reaches ρ=0.95 at n≈150.The sample-size analysis uses about 200 items per benchmark for error-direction and calibration estimates with bootstrap confidence intervals.Source: §3 · pp. 2–3
  • A cross-scheme label audit found high agreement with mapped labels (gpt-5.5 κ=0.89; Claude κ=0.87) and between judges (κ=0.92).ToxiGen uses released human toxicity scores: 4–5 map to flag, 1–2 to do not flag, and score-3 items are excluded.Source: §3 · p. 3
  • A fixed two-option prompt records token-logprob confidence and the model’s 1–10 verbalized confidence rescaled to [0, 1].The token-logprob signal is calculated from the two option-token logprobs after a fixed judgment prefix and renormalized over the two options.Source: §3 · p. 3
  • The evaluation reports false-alarm and miss rates, calibration metrics, and confidence-based risk–coverage summarized by AURC, with cell-level 95% bootstrap intervals.The same fixed items support model selection across reliability dimensions and further calibration, abstention, and escalation research without rerunning models.Source: §3 · p. 3

4 Setup

The setup standardizes flag/do-not-flag judgments and confidence collection across fixed benchmark items, while keeping specified models outside the primary ranking.

4 Setup

  • Six open-weight instruction-tuned LLMs are evaluated through a shared pipeline using greedy decoding and a two-option flag/do-not-flag prompt.The setup records token-logprob confidence from the chosen option and the model’s verbalized 1–10 confidence rescaled to [0, 1].Source: §4 · p. 3
  • Qwen2.5-32B is an indicative scale point rather than a primary-ranked model because it uses a single seed and mixed precision; Table 3 ranks the leaderboard by macro-F1.The table caption also identifies false-alarm and harmful-item miss rates and excludes the reference models from the primary ranking.Source: §4 · p. 3, Table 3 · p. 4
  • Parsing fails on only 2 of 8,388 outputs (< 0.03%), and models are scored on fixed lists of 198–200 balanced item IDs.Prompts, reproduction code, and label harmonization are released with the resource.Source: §4 · p. 3
  • Sampled-answer agreement is assessed with five generations per item, alongside token-logprob and verbalized confidence analyses.Source: §4 · p. 3
  • Three reference models are scored on identical items but excluded from the primary ranking; gpt-5.4-mini has no reported logprob ECE or AURC.Its API hides option-token logprobs; each reference model is a single run.Source: §4 · p. 3
  • Temperature scaling fits one scalar per model on held-out pooled items, while selective prediction compares confidence-ranked risk at fixed coverage with random abstention.Recalibrated ECE is evaluated on the other half of the pooled items.Source: §4 · pp. 3–4
  • Table 2 maps each benchmark’s released unsafe class to flag while retaining its original judged unit and approximately balancing flag and do-not-flag items.Source: Table 2 · p. 4

5 Results

The leaderboard and reliability analyses show that aggregate performance obscures distinct error profiles, calibration quality, and the value of confidence-based review. These dimensions differ across models, and temperature scaling improves calibration without changing decisions or confidence ordering.

5.1 The reliability leaderboard

  • Gemma-2-9B leads the ranked models (macro-F1 0.824), while Qwen2.5-32B scores highest overall (F1 0.863) as an indicative scale point; Llama-3.1-8B ranks last (F1 0.443).The two API reference models do not surpass Qwen2.5-32B, and both under-flag; R1-Distill-Llama-8B shifts the instruct base toward a more balanced profile.Source: §5.1 · p. 4

5.2 Error direction differs sharply across models

  • Llama-3.1-8B flags 85% of benign content, while OLMo-2-7B misses 54% of harmful content; their error-direction scores are +0.77 and −0.44, respectively.The contrast is seed-stable and robust to leaving out any one benchmark; lower false-alarm rates tend to coincide with higher miss rates rather than uniformly better accuracy.Source: §5.2 · pp. 4–5, Figure 3 · p. 5

5.3 Calibration varies sharply across models and classes

  • Pooled logprob ECE ranges from 0.127 for OLMo-2-7B to 0.368 for Llama-3.1-8B, but OLMo’s low aggregate ECE coexists with the highest miss rate.Class-conditional ECE follows dominant error type: Llama’s harmful-item ECE is 0.030 and benign-item ECE is 0.730, while OLMo shows the reverse pattern.Source: §5.3 · p. 5

5.4 Temperature scaling improves calibration but preserves decisions

  • Temperature scaling cuts logprob ECE by 2.8–6.0× across random fit/eval splits while leaving predicted decisions and confidence ordering unchanged.Best-fit temperatures range from 2.7 for OLMo to 9.0 for Llama; calibration and error ranking therefore require separate evaluation.Source: §5.4 · p. 5

5.5 Selective risk under abstention

  • At 50% coverage, confidence-based abstention beats random abstention for every model, with gains from 0.074 for OLMo-2-7B to 0.135 for Mistral-7B.The gains track error-ranking quality; pooled and within-benchmark risks are broadly similar, token-logprob confidence gives the lowest AURC for all six models, and temperature scaling cannot change the ranking.Source: §5.5 · p. 5

6 General-purpose LLMs vs. dedicated guards

The comparison evaluates dedicated guards and general-purpose models on the same items using a shared flag/no-flag decision and confidence framework. Guards have lower false-alarm rates and better average calibration, but some miss more harmful items beyond their documented coverage.

6 General-purpose LLMs vs. dedicated guards

  • The comparison evaluates six general-purpose LLMs and four open guards on identical items, mapping each guard’s native output to flag/no-flag and normalizing logprobs for its native decision-token pair.The same definitions are used to compute false-alarm rate, ECE, and AURC.Source: §6 · p. 5
  • Ethics is excluded because its moral-action items fall outside the guards’ content-safety construct, and WildGuard and Aegis results separate benchmark-overlap from non-overlap cells.Source: §6 · p. 6
  • Dedicated guards have lower mean false-alarm rates than the general-purpose mean of 0.329 and average ECE of 0.13 on cells without documented training overlap, versus 0.205 for general-purpose models.Llama Guard 3 and ShieldGemma miss 34.5% and 50.2% of harmful items, respectively; ShieldGemma’s policy covers only four harm types and misses most BeaverTails harms.Source: §6 · p. 6

7 Discussion

Deployment priorities change which moderator is preferred, while calibration and confidence-based review address different needs. Dedicated guards reduce false alarms and calibration error, but their miss rates and error ranking do not improve uniformly.

7 Discussion

  • Temperature scaling lowers calibration error for every model without changing predicted labels; its calibration effect is shown across 20 fit/evaluation splits.Figure 6 annotations report the fold reduction in ECE.Source: Figure 6 · p. 7
  • Human review reduces the remaining error faster when a model's confidence ranks its errors well.Figure 7 compares error rates as fewer items are handled automatically.Source: Figure 7 · p. 7
  • All four dedicated guards have lower false-alarm rates and ECE than the general-purpose LLM mean, but miss rates and AURC do not improve uniformly.Ethics is excluded as out-of-construct from the Figure 8 comparison.Source: Figure 8 · p. 7
  • A policy paraphrase preserves error direction in 11 of 12 cells, while a label-order swap flips it in four cells, all for Gemma-2-9B and OLMo-2-7B.These swaps indicate residual answer-position bias in those two models; absolute-rate comparisons therefore hold prompt and label order fixed.Source: §7 · p. 7
  • The results are bounded by coarse verbalized confidence, an approximately 200-item-per-benchmark English single-turn binary suite, and aggregate errors without group- or dialect-conditional breakdowns.Qwen2.5-32B is a single-seed, mixed-precision scale point with no ranking claim; multi-agent moderation is out of scope.Source: §7 · p. 7
  • Qwen2.5-32B has the lowest cost when errors are equally weighted; OLMo-2-7B is preferred when harmful content is rare (π≤5%), while λ=10 favors lower-miss models.Figure 9 shows Qwen2.5-32B winning across most regimes and Gemma-2-9B favored only when harmful content is both prevalent and costly to miss.Source: §7 · pp. 6–7, Figure 9 · p. 8

8 Ethics and Broader Impact

Safety-Flag reuses released benchmark items and provides derived artifacts for moderation audits and research. Its documentation preserves source-label provenance and notes both intended uses and a possible adversarial use.

8 Ethics and Broader Impact

  • Safety-Flag releases balanced item selections, harmonized labels, standardized prompts, per-item verdicts and confidence, and reproduction code, without releasing new harmful content.Items come from already released benchmarks; raw text must be obtained from source benchmarks under their original licenses, with provenance recorded per item.Source: §8 · p. 8
  • The resource is intended to support pre-deployment moderator audits and research, while its measurements could also help adversaries find moderators that under-flag particular harms.The passage notes that the underlying models and benchmarks are already public.Source: §8 · p. 8
  • Because benchmarks retain their source policies and label provenance, Safety-Flag reports results per benchmark and avoids letting one mapping determine a conclusion.Source: §8 · p. 8

9 Availability

The release includes the benchmark suite, item IDs, harmonized labels, model outputs, code, and leaderboard. It distributes derived data and code under stated licenses while leaving source text with its original benchmarks.

9 Availability

  • The archive provides the seven-benchmark suite, balanced item IDs, harmonized labels, standardized prompt, per-item outputs, recalibration and reproduction code, and leaderboard.The archive is available through the project site and permanent Zenodo concept DOI.Source: §9 · p. 8
  • Code is MIT-licensed and derived data CC BY 4.0; source text is not redistributed and must be obtained under each benchmark's license.Shipping IDs and labels rather than source content keeps the release compatible with the source licenses.Source: §9 · p. 8

10 Conclusion

The conclusion presents Safety-Flag as a common, item-level evaluation whose results distinguish moderator error profiles, overconfidence, and the usefulness of confidence for abstention. The accompanying roadmap and resource-construction section describe appendix coverage and documentation scope.

10 Conclusion

  • Safety-Flag evaluates seven safety benchmarks and releases item-level decisions and confidence for ten primary open-weight moderators plus three reference models; results show varied error profiles, general-purpose overconfidence, and varied abstention ranking.Temperature scaling improves calibration without changing decisions or error ranking; the suite and leaderboard support reproducible, deployment-aware comparisons.Source: §10 · p. 8

Appendix roadmap

  • The appendix roadmap covers resource documentation, robustness to smaller samples and benchmark removal, per-cell error-direction and calibration results, selective prediction, and prompt sensitivity.Source: Appendix roadmap · p. 10

A Resource Construction and Documentation

  • The resource-construction section documents what the release contains, redistribution terms, and whether binary harmonization is faithful to source labels.Source: A Resource Construction and… · p. 10

A.1 Benchmark specification

The appendices document the released benchmark artifacts and audit, then detail stability, per-benchmark reliability, recalibration, guard comparisons, selective prediction, and prompt-sensitivity analyses. Their tables specify the comparisons and protocols behind these analyses.

A.1 Benchmark specification

  • Safety-Flag versions reproducible item-level outputs, redistributes derived artifacts rather than source text, and audits harmonized labels against two cross-vendor judges.The audit re-labels a balanced sample of 25 items per benchmark and reports each judge’s agreement with mapped gold and judge-vs-judge agreement.Source: Table 5 · p. 10, Table 6 · p. 10, Table 7 · p. 10

B Stability Analyses

  • Subsampling finds macro-F1 rankings and error-direction signs stable at n≈100, while leave-one-benchmark-out analyses preserve the reported error-direction profiles.The appendix also checks pooled-ECE ranking under subsampling and macro-F1, error-direction, ECE, and AURC rankings when each benchmark is dropped.Source: B Stability Analyses · p. 10, Table 8 · p. 11, Table 9 · p. 11

C.1 False alarms and misses by benchmark

  • Tables 10–11 report each model’s false-alarm and miss rates, and logprob calibration error, separately for every benchmark.The tables provide per-model-by-benchmark cells on identical items.Source: §C.1 · p. 11, §C.2 · p. 11, Table 10 · p. 12

C.3 Recalibration robustness

  • Table 12 reports the calibration-error reduction from one fitted temperature per model and its stability across fit/evaluation splits.It summarizes 20 random 50/50 splits, including held-out ECE before and after scaling.Source: §C.3 · p. 11, Table 12 · p. 11

C.4 General-purpose LLMs vs. dedicated guards

  • The guard comparison reports mean false-alarm, miss, and calibration results on identical items, excluding Ethics as out-of-construct.The leave-one-benchmark-out caption identifies lower mean false alarms and ECE for guards as the headline comparison.Source: §C.4 · p. 11, Table 13 · p. 11, Table 9 · p. 11

D.1 Comparing confidence signals

  • The appendix compares confidence signals for error ranking and pooled, within-benchmark, and normalized abstention rankings.AURC is lower-is-better; predictive entropy is omitted because its binary-decision ranking matches logprob confidence.Source: §D.1 · p. 11, Table 14 · p. 12, §D.3 · p. 12

D.2 Risk at fixed coverage

  • At 50% coverage, confidence-based abstention yields lower selective risk than random abstention for every model.The tables also report risk at fixed coverage when each flagger answers only its most-confident items.Source: §D.2 · p. 12, Table 15 · p. 12, Table 16 · p. 13

E.1 Policy paraphrasing and label-order swaps

  • Prompt-sensitivity results compare four models on three benchmarks under policy paraphrasing and swapped answer-option order, marking error-direction flips.The analysis asks how sensitive error direction is to wording and option order.Source: §E.1 · p. 13, Table 18 · p. 13, E Prompt Sensitivity · p. 13
Loading 2609.19072v1…