PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy Is Not Capability: Some Safety Benchmarks Measure Which Definition of "Unsafe" You Trained On

Vahid Jebraeeli, Jonathon Romero, Yijin Wang, Adrian Chan, Jianbang Zhang

arXiv:2610.25060v1cs.LGcs.AIcs.CLcs.CR

TL;DR

Guardrail evaluations treat “unsafe” as a property of prompts, though it depends on the labeling policy, and existing evidence often cannot separate those definitions. This paper compares models trained on prompts labeled under two operational rubrics and audits benchmark overlap, score sensitivity, and validation accuracy. The results show that both benchmark scores and model rankings can track policy choices, motivating evaluation reports that state contamination, seed variability, and the grading definition.

  • Problem

    Guardrail scores assume that systems evaluated on the same prompts answer the same question, although “unsafe” can depend on the labeling policy.

    Source: §1 · pp. 1–2

  • Method

    The authors compare guardrail arms trained using two label definitions and audit benchmark overlap, score variation, and validation rankings.

    Source: §1 · p. 2

  • Results

    Label-source sensitivity ranged from 0.99 on HarmBench-copyright to 0.043 on WildGuard-benign, and validation accuracy tracked policy agreement more than detection capability.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors argue that guardrail evaluations should report contamination, a seed floor, and the definition used for grading.

    Source: §1 · p. 2

    The study uses twenty ModernBERT encoders, one proprietary taxonomy, and two rubrics that differ in annotation modality as well as rubric, so the specific ordering may not transfer.

    Source: §9 · p. 8

Abstract

from arXiv · show

A guardrail classifier is reported as if unsafe were a property of text. It is not: it is a property of a policy, and the policy enters the model through its labels. We exploit an accident of an industrial deployment in which the same ~53k prompts were labelled twice, under two operational definitions ("is this a request for harmful content?" and "would a moderator block this?"), and used to train ten guardrail arms sharing architecture, codebase and optimiser family, at two model scales. First, the guardrail's own training corpus already contained the benchmarks: JailbreakBench-benign is 88% and OR-Bench-hard 82% present in the training pools, two over-refusal benchmarks, the kind a guardrail reports to argue it does not over-block. Everything below is decontaminated. Second, safety benchmarks stratify sharply by how much of their score is decided by the label definition rather than by the model: with architecture and optimiser fixed, the range across the ten arms is 0.99 on HarmBench-copyright and 0.043 on WildGuard-benign, a 23x spread. Third, a benchmark score can be moved across its entire range by 0.03% of the parameters. Across these arms HarmBench-copyright spans 0-100 and OR-Bench-hard over-refusal 3.4-56.4%, and the four arms whose last supervision signal carries a given policy occupy one extreme of both (exact rank p=0.005, or 0.04-0.14 at worst), at unchanged detection capability. For a distilled student the teacher's soft labels do this even when every hard label carries the other policy, so teacher provenance is an unrecorded confound in any released checkpoint. Fourth, validation accuracy on the one split that grades all twenty arms tracks agreement with the annotator's policy (rho=+0.74), not detection capability (rho=+0.34). We argue that guardrail evaluations should report contamination, a seed floor and the definition they were graded against as first-class variables.

1 Introduction

Guardrail evaluations treat “unsafe” as a property of prompts, although it reflects a labeling policy. Using prompts labeled under two operational definitions, the paper audits contamination, policy sensitivity, and validation accuracy.

1 Introduction

  • The same prompt can be blockable under one deployment policy and ordinary under another, so “unsafe” reflects a labeling decision rather than a fact about the string.The authors describe their contributions as diagnostic rather than methodological.Source: §1 · pp. 1–2
  • The two pipelines disagreed on one prompt in six, and their systematic differences reflected distinct questions: whether a moderator would block it or whether it requested harmful content.Ten arms were trained across the two label families and evaluated on fifteen benchmarks.Source: §1 · p. 2
  • The training corpus included 88% of JailbreakBench-benign and 82% of OR-Bench-hard; all results below are decontaminated.The introduction also reports that validation accuracy tracked agreement with the annotator’s policy (ρ = +0.74) more than detection (ρ = +0.34).Source: §1 · p. 2
  • Label-source changes moved HarmBench-copyright across 0.99 of its scale but WildGuard-benign across only 0.043, while the last supervision signal could move a benchmark score across its full range.The paper reports that validation accuracy ranks arms more by policy agreement (ρ = +0.74) than detection capability (ρ = +0.34).Source: §1 · p. 2
  • The authors conclude that evaluation reports leave unstated how much a score answers whose policy the model was trained on.They propose reporting contamination, a seed floor, and the definition used for grading.Source: §1 · p. 2

2 Setting

The guardrail predicts coarse safety classes and a proprietary taxonomy, using two differently defined label families. The study compares multiple training routes and model scales while keeping architecture, codebase, and optimiser family shared.

2 Setting

  • The classifier emits one of four coarse classes and a 24-way multi-label vector; binary unsafe is HARMFUL ∪ COMPOUND.The proprietary taxonomy is grouped into Safety, Security, Privacy, and Reliability pillars.Source: §2 · p. 2
  • The teacher has 394.8M parameters and the student 149.0M; both use ModernBERT encoders with three linear heads.The student adds a knowledge-distillation term on four-class and pillar logits.Source: §2 · p. 2
  • The head-calibration route freezes the encoder and refits the heads on calibration labels, updating 0.03% of model parameters.Figure 1 identifies three routes for introducing an operational definition; §5 tests which routes install a policy.Source: §2 · p. 3, Figure 1 · p. 3
  • Policy A treats some copyrighted-text, refusal-prompt, and edgy-but-harmless requests as blockable, whereas Policy B treats them as safe; neither was validated against a human-adjudicated reference set.On comparable labels, exact agreement was 83.6% for four-class labels and 90.1% at the binary level.Source: §2 · p. 3
  • The ten arms introduce Policy A through blends, all-hard labels, a distillation teacher, or head-only calibration, but also differ in pool size and some configuration choices.All twenty models were scored using one harness in a deterministic pass.Source: §2 · p. 3
  • The evaluation covers fifteen public benchmarks and 33,137 prompts, using catch rate for twelve and over-refusal rate for three.Decontamination leaves two usable over-refusal benchmarks.Source: §2 · p. 3

3 First audit: the guardrail’s training corpus contains the benchmarks

The first audit finds substantial benchmark overlap with the guardrail’s training and validation pools, especially on over-refusal benchmarks. The authors recompute results on shared clean subsets and note that decontamination and label validity can remain entangled.

3 First audit: the guardrail’s training corpus contains the benchmarks

  • 88.0% of JailbreakBench-benign and 82.1% of OR-Bench-hard were contaminated, based on exact normalized-text matches that provide a lower bound.The audit checked prompts against training, calibration, and validation pools used across the grid.Source: §3 · p. 4
  • The two most contaminated benchmarks are over-refusal benchmarks, because public benign-but-edgy datasets were harvested to build the BOUNDARY_SAFE class.The paper describes this overlap as dataset convergence rather than carelessness.Source: §3 · p. 4
  • The authors recomputed results on a shared clean subset, dropping JailbreakBench-benign with only 12 clean prompts; OR-Bench-hard retained 236 of 1,319.Across the fourteen usable benchmarks, ten ranges rose or were flat and four fell slightly after decontamination.Source: §3 · p. 4
  • Decontamination and label validity remain entangled for OR-Bench-hard: all 236 surviving prompts drew at least one harmful vote, so the score shift may reflect label composition as well as memorization.The authors say they cannot separate those effects.Source: §3 · p. 4

4 Second audit: how much of a benchmark’s score the label definition decides

Benchmark scores vary markedly in their sensitivity to label source, but interpreting that spread requires comparison with seed variability. The clearest separations are between policy-A-last arms and the other recipes; the middle ordering remains unresolved.

4 Second audit: how much of a benchmark’s score the label definition decides

  • 0.99 of its scale separates the ten-arm range on HarmBench-copyright, versus 0.043 on WildGuard-benign, a 23× spread between benchmarks.The range is averaged over two model scales; the authors caution that 75 clean prompts in two templates support only two significant figures.Source: §4 · p. 4, Figure 2 · p. 5
  • The authors define a seed floor Φ_b because label-source range alone cannot distinguish policy sensitivity from ordinary run-to-run score variation.They state that only Δ_b/Φ_b is interpretable.Source: §4 · p. 4
  • The seed floor is a pessimistic estimate from four nearby recipes, and the authors treat it as an off-scale estimate at 149M because those configurations differ from the evaluated arms.At 395M the recipes differ in one configuration key and training pool; at 149M they also differ in class-loss weight and backbone.Source: §4 · p. 5
  • Across twelve runs of four nearby recipes, HarmBench-copyright never exceeded 73, while policy-A-last arms reached 100 at 149M and 74–100 at 395M.For OR-Bench-hard, the four recipes never fell below 11.2%, while policy-A-last arms scored 1.6–3.3% and 2.5–3.8%, respectively.Source: §4 · p. 5
  • The qualitative split distinguishes benchmarks driven by contested cases from those driven by explicit weapons and violence requests, but the middle ordering does not survive the seed-floor analysis.The authors report HarmBench-copyright at 1.7× its floor and OR-Bench-hard at 1.5×, and claim only the split.Source: §4 · p. 5
  • Scores of 100 and 0 on HarmBench-copyright need not indicate better versus worse detection: in this grid they can reflect which rubric supervised the model last.The reported correlation across the two scales is partial (ρ = 0.71), and the scales are not independent.Source: §4 · p. 5

5 Third audit: a benchmark score is steerable at 0.03% of the parameters

The operational definition can be changed at the final supervision stage, even when that stage updates only 0.03% of parameters. For distilled students, a policy-A teacher or head-only calibration yields extreme benchmark scores while detection capability remains close to data-mixing arms.

5 Third audit: a benchmark score is steerable at 0.03% of the parameters

  • Policy-A data mixing left HarmBench-copyright at 0.0–1.3 across E2–E5, while OR-Bench-hard over-refusal remained 37.7–56.4%.E6 and E9 also fell short because their distilled policy-B teacher's soft labels overrode their hard-label data.Source: §5 · pp. 5–6
  • 100.0 HarmBench-copyright with 4.7% OR-Bench-hard over-refusal (E1) followed head-only calibration; E7 and E10 also reached 100.0, with 6.8% and 3.4% over-refusal.The calibration refit 0.04M of 149.05M parameters; the table reports the decontaminated results and groups the 149M arms.Source: Table 1 · p. 6, §5 · p. 6
  • Policy-A-last arms occupied one extreme of HarmBench-copyright and OR-Bench-hard at both scales, with exact rank p=0.005 at 149M and p=0.004 at 395M.The authors qualify the rank-test evidence: clustered, two-sided tests give p=0.071 and 0.036, respectively.Source: §5 · p. 6
  • E8's policy-A teacher reached 91.1% mean catch rate over six capability-loaded benchmarks, versus 88.3–86.8% for data-mixing arms; E1 reached 81.0%.The paper reports E7 at 89.1%; E1 used the smallest calibration set and had the weakest starting representation.Source: §5 · p. 6
  • Teacher provenance can preserve an old policy after a team relabels and distils, and it is absent from model cards; an early calibration pass also erased GENERIC_SAFE.The paper cautions that the same calibration lever that makes policy changes cheap can also destroy a class.Source: §5 · pp. 6–7

6 The same disagreement, between independently built systems

Evaluations of independently built guardrails can diverge sharply on identical prompts because their operational definitions differ. The paper's comparisons also show that verdict mapping and deployment conditions shape what the reported scores mean.

6 The same disagreement, between independently built systems

  • E8 matched Qwen3Guard-8B's loose-mode capability-loaded score at 91.1 versus 91.8, while all four policy-A-last 149M arms had 4.5–6.3% benign over-refusal.The comparison is not like-for-like in deploy size and latency; E8 was selected on these same benchmarks.Source: §6 · p. 7
  • On 75 clean HarmBench-copyright prompts, Granite Guardian scored 0 and Qwen3Guard-4B scored 99 on identical inputs.The authors attribute the difference to Granite's common-sense harm-risk definition versus Qwen3Guard's explicit Copyright Violation category.Source: §6 · p. 7
  • Mapping Qwen3Guard's Controversial verdict as unsafe raises its 8B model's OR-Bench over-refusal from 29.7% to 48.5%; mapping BOUNDARY_SAFE as unsafe raises the paper's 149M rate from 5.7% to 96.0%.The paper presents strict and loose mappings as policy choices made after training, with no retraining or data change.Source: §6 · p. 7

7 Fourth audit: validation accuracy tracks policy, not capability

The shared validation split ranks the arms more closely with policy-loaded benchmark scores than with capability-loaded scores, despite grading models trained under both policies. The split is also substantially contaminated by policy-A training data.

7 Fourth audit: validation accuracy tracks policy, not capability

  • The multilingual validation split contained 69.3% verbatim overlap with policy-A training data, yet exposure did not produce the observed ordering (rank correlations −0.15 and +0.16).A monolingual validation split with 0.2% overlap existed but was not used.Source: §7 · p. 7
  • Validation accuracy correlated ρ = +0.74 with policy-loaded benchmarks and ρ = +0.34 with capability-loaded benchmarks across twenty models.The contrast held within each scale: +0.64/+0.32 for students and +0.83/+0.37 for teachers; intervals overlap at n=20.Source: §7 · p. 7

8 What we think should change

The authors recommend reporting benchmark overlap, a seed floor, and the operational definition and split behind each evaluation. They argue these variables are needed to interpret guardrail results, rather than simply adding benchmarks.

8 What we think should change

  • Report benchmark overlap, a seed floor, and the operational definition and split used for grading as evaluation variables.The authors say their 54–71-point HarmBench-copyright seed spread changed which results they were willing to report, and recommend at least three seeds across multiple recipes.Source: §8 · p. 8
  • State the rubric's rulings on contested cases and how any contested verdict is mapped; a differently sourced split alone is not sufficient.The two pipelines shared 24 taxonomy pillars but disagreed on one prompt in six, and the differently sourced split still ranked arms by policy.Source: §8 · p. 8

9 Limitations

The authors identify limits in seed coverage, experimental matching, benchmark scope, and comparison conventions that constrain how broadly to interpret their results.

9 Limitations

  • Each arm uses one seed, and a seed floor from neighboring recipes leaves the middle of Figure 2(b) unresolved.The authors instead rely on a rank statistic and two separations in level that clear the seed floor.Source: §9 · p. 8
  • The study covers twenty ModernBERT encoders, one proprietary taxonomy, and two rubrics differing in both annotation modality and rubric.The specific ordering may not transfer, although the authors say the qualitative claim should and reproduce it in §6 without either pipeline.Source: §9 · p. 8
  • The arms differ in pool size (8,489–144,744) and checkpoint-selection rule, while pool size and route are collinear at 395M.The overlap audit is a lower bound.Source: §9 · p. 8
  • Benchmark-harmful prompts count as caught when flagged, so catch rate measures rubric agreement, and binary scoring counts BOUNDARY_SAFE as safe.The latter is worth 90 points of over-refusal; HarmBench-copyright’s 0.99 range comes from 75 prompts built from two lyric templates.Source: §9 · p. 8
  • The SOTA comparison is asymmetric: the study’s arms were developed against these benchmarks, baselines were not, and each system uses its own verdict convention.Baseline benchmark exposure is unknown, and the comparison’s one-sided direction cannot be bounded; pooled deployment precision is 0.968 and recall is 0.496.Source: §9 · p. 8

10 Related work

Prior guardrail work uses fixed taxonomies, while this paper isolates the effect of training on different labels for the same prompts. It also connects policy-bearing supervision to known effects of distillation and small parameter subsets.

10 Related work

  • LlamaGuard [12], ShieldGemma [36], WildGuard [7], and Aegis [5] exemplify fine-tuned guardrails with fixed taxonomies; LlamaGuard’s customcategory parameter is reported as cosmetic [19].The authors did not rerun those independent reports.Source: §10 · p. 8
  • Granite Guardian [18] attributed early social-bias detector errors mainly to distinguishing use from mention of harmful content, encoded through taxonomy-driven synthetic data.This paper isolates a related choice by training on two labellings of the same prompts.Source: §10 · p. 8
  • Prior work documents annotator disagreement and aggregation costs [24], including for toxicity [29] [6]; this paper examines its behavior under training.The paper treats the final supervision signal as a separable component.Source: §10 · p. 8
  • Distillation is known to carry bias and calibration [8] [9] and other behavioral traits; the authors add the teacher’s policy as overriding student hard labels.Small parameter subsets also steer behavior in adapters and LoRA, while head-only calibration retrains the last layer on frozen features [10] [11] [14].Source: §10 · p. 9

A Evaluation suite, aggregate membership, and arm provenance

The evaluation combines fifteen public benchmarks into policy-loaded, capability-loaded, and over-refusal groups, while auditing arm provenance, benchmark exposure, and related work.

A Evaluation suite, aggregate membership, and arm provenance

  • The fifteen-benchmark suite contains 33,137 prompts; its policy-loaded, capability-loaded, and over-refusal aggregates use distinct scoring criteria and exclusions.JailbreakBench-benign is excluded from over-refusal aggregation because only 12 clean prompts remain.Source: A Evaluation suite, aggregate… · p. 12
  • Validation accuracy correlates more with policy-loaded benchmarks (ρ = +0.74) than capability-loaded benchmarks (ρ = +0.34), but the ordering is not resolved within route blocks.The policy-loaded set includes HarmBench-copyright, DoNotAnswer, and SaladBench-defense; the capability-loaded set includes six explicit-harm benchmarks.Source: A Evaluation suite, aggregate… · p. 13, Figure 3 · p. 13
  • On HarmBench-copyright, the least-trained head-only arm still sits at the policy-A extreme, scoring 100.0, alongside 4.7% on OR-Bench-hard.E1 stopped after two of five epochs and was never rerun.Source: A Evaluation suite, aggregate… · p. 14
  • In the exact joint permutation test of whether A occupies extreme ranks on both benchmarks, the qualifying subset is the same on each, so the second benchmark adds no independent evidence.Source: A Evaluation suite, aggregate… · p. 15
  • At 395M, the policy-A-last grouping survives removal of head-calibration arms (p = 0.048), but the analogous 149M grouping does not (p = 0.14).The 395M grouping was revised after the 149M definition failed there and was not pre-specified.Source: A Evaluation suite, aggregate… · pp. 14–15
  • Across arms, effective OR-Bench-hard exposure through traced training lineages is 81.0–82.3%, while prompt-resampling uncertainty is smaller than seed variance in the middle of the benchmark range.The matched test finds all 28 benchmark-by-scale extreme-arm gaps significant, with worst-case exact McNemar p = 0.008.Source: A Evaluation suite, aggregate… · p. 17

B Prior art on the contested middle class, and on contamination

  • Qwen3Guard varies the Safe-to-Unsafe class ratio within a fixed taxonomy, whereas this paper varies label definition across fifteen decontaminated benchmarks and twenty models.Its distillation is framed as annotation-noise reduction; this paper reports policy transfer through teacher soft labels despite contrary hard labels.Source: B Prior art on the contested middle… · pp. 17–18

C Seed floors: all 37 triplets

Across 37 triplets, seed-floor spreads are much smaller than those of the four selected v10x recipes on key benchmarks, but the floor and within-split ordering depend on recipe choice. The section also qualifies the comparison by noting that the arms lack seed siblings and the measured prompt sets differ.

C Seed floors: all 37 triplets

  • 58.5 points on HarmBench-copyright and 38.6 on OR-Bench-hard are the v10x median spreads, versus 3.0 and 1.6 for the other 33 triplets.The selected family’s spreads are 19.5× and 24.2× the corresponding non-v10x medians.Source: C Seed floors: all 37 triplets · p. 18
  • The widest of the 37 triplets on HarmBench-copyright, OR-Bench-hard, SaladBench-attack, and WildGuard-benign is in each case a v10x recipe.These are the four benchmarks used in §§4–5.Source: C Seed floors: all 37 triplets · p. 18
  • The ordering by ∆b/Φb is not identified: SaladBench-attack and WildGuard-benign fall below 1 under the selected floor, but none do under the median of all 37.The alternate orderings correlate with the printed ordering at Spearman −0.03, −0.08, and +0.26; the authors claim the top/bottom split, not the order within it.Source: C Seed floors: all 37 triplets · p. 19
  • The level comparison is scoped to the twelve v10x runs and four recipes; across all 111 runs, HarmBench-copyright reaches 100 and OR-Bench-hard falls to 0.9%.Those 111 runs differ in label supervision, the effect measured by ∆b, rather than only in seed.Source: C Seed floors: all 37 triplets · p. 19
  • Ten of the 37 triplets are not strictly seed-only, while the 27 strict triplets produce similar benchmark spreads and non-v10x medians.The four selected recipes are not independent draws on HarmBench-copyright: all four spreads increase with seed index, an event estimated at probability 7.7 × 10^-4 if seeds are exchangeable.Source: C Seed floors: all 37 triplets · p. 19
  • The arms have no seed siblings, so the floor is a proxy from neighbouring recipes; closing that gap requires 40 further training runs.The closest on-path triplets have lower spreads than the quoted floor, and the relevant warm-start recipe’s HarmBench-copyright spread varies by data version.Source: C Seed floors: all 37 triplets · p. 19
  • The ratio is indicative rather than exact because Φb uses full benchmarks while ∆b uses decontaminated subsets.Recomputing ∆b on full benchmarks changes OR-Bench-hard’s ratio from 1.49 to 1.09; both reported below-1 entries survive the swap.Source: C Seed floors: all 37 triplets · p. 19

D The over-refusal label audit

The label audit uses a one-time model jury to examine prompts in the BOUNDARY_SAFE class and stratify apparent over-refusal. It finds substantial disagreement and shows that over-refusal comparisons vary across the jury-defined strata, with the smallest stratum too small for a firm estimate.

D The over-refusal label audit

  • The one-time jury independently labels 20,456 harvested prompts using four 26–35B instruction-tuned models and a two-factor prompt; it has no human adjudication or inter-rater statistic beyond vote counts.The authors describe the jury as their instrument, not ground truth.Source: D The over-refusal label audit · p. 19
  • At a uniform 3-of-4 vote threshold, 50.9% of the pool is BOUNDARY_SAFE, 32.9% GENERIC_SAFE, 4.5% HARMFUL, and 11.6% has no three-vote majority.By plurality, the partition is 60.3 / 33.5 / 6.3%, and AMBIGUOUS never wins a row.Source: D The over-refusal label audit · p. 19
  • The jury flags 6.9% of the pool as HARMFUL, including 17.9% of OR-Bench-hard prompts and 26.4% of FalseReject prompts.The clean 236, the 211 prompts flagged by at least two jurors, the 112 flagged by at least three, and the 25 left unclear have distinct over-refusal rates across systems.Source: D The over-refusal label audit · p. 20
  • Of 236 clean prompts, 211 are flagged harmful by at least two jurors, 112 by at least three, and 92 are exact 2–2 ties.Joining on normalized text recovers all 1,319 OR-Bench-hard prompts, so the audit’s figures are not a coverage artifact.Source: D The over-refusal label audit · p. 20
  • On the 25 prompts the jury leaves unclear, OR-Bench-hard over-refusal is 4.0% for both evaluated in-house arms and ranges from 4.0% to 84.0% across the listed systems.The authors characterize this as the only stratum resembling a real over-refusal test, but caution that n=25 supports a direction rather than an estimate.Source: D The over-refusal label audit · p. 20
Loading 2610.25060v1…