PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics

Hongyu Li, Xinlong Huang, Zhan Yu, Jing Zhao, Zhe Zhao, Juntao Zhong, Xiong Kuang, Rui Yuan, ZhenXiang Yan, Huilin Xu, Haitao Yu, Lilin Wang, Rui Chen, Blaze Chen, Fengzong Lian, Renfen Hu

arXiv:2610.29225v1cs.CLcs.AIcs.CR

TL;DR

Binary safety labels provide limited explanatory depth, and automated evaluation can be inconsistent. DiagSafe pairs prompt-level safety and usefulness rubrics with a bilingual, 84-label benchmark; its evaluation finds higher unsafe rates on instruction attacks and model-specific vulnerabilities.

  • Problem

    Many LLM safety benchmarks use binary labels that lack explanatory depth, while automated judges can be inconsistent and benchmark coverage does not systematically combine multilingual, multi-turn, and varied attack scenarios.

    Source: §1 · pp. 1–2

  • Method

    DiagSafe evaluates prompts with explicit, prompt-level rubrics for safety and usefulness and provides a bilingual benchmark spanning an 84-label taxonomy and diverse prompt formats.

    Source: §1 · p. 3, §3.1 · p. 5

  • Results

    Every evaluated model has an unsafe rate on DiagAttack 1.9×–8.4× that on DiagBase, and models with similar overall unsafe rates show different category-specific vulnerabilities.

    Source: §6 · p. 13

  • Takeaways & Limitations

    DiagSafe provides a diagnostic evaluation that reveals safety failures obscured by aggregate unsafe rates.

    Source: §6 · p. 13

    Because label specifications are LLM-generated, synthesized prompts and rubrics may miss rare, emerging, or highly localized harmful scenarios.

    Source: §7 · p. 14

Abstract

from arXiv · show

Current LLM safety benchmarks predominantly rely on binary safe or unsafe labels, lacking explanatory depth and suffering from low consistency across automated judges. To address this, we introduce DiagSafe, a diagnostic evaluation framework that moves beyond binary classification by employing prompt-level rubrics. Grounded in globally applicable AI safety policies, each rubric provides explicit criteria along two dimensions: safety, which targets concrete harms with specific exemptions to suppress false positives, and usefulness, which independently evaluates instruction fulfillment. To ensure a robust and realistic evaluation, we construct a bilingual (English-Chinese) benchmark grounded in a comprehensive 84-label safety taxonomy, consisting of two subsets: DiagBase, which covers single- and multi-turn prompts with direct harmful queries and template-based jailbreaks, and DiagAttack, which embeds harmful intents into complex instruction-following tasks to form challenging instruction attacks. Extensive experiments demonstrate that DiagSafe significantly enhances the reliability of LLM-as-a-judge evaluation, achieving 95.34% agreement with human annotations -- substantially higher than the 69%-87% achieved by the baselines -- and high inter-judge consistency (Cohen's $\kappa$ = 0.864). Furthermore, applying our framework reveals that leading LLMs differ markedly in their unsafe rates, and each model also exhibits unique vulnerabilities along different dimensions. In addition, we observe that models are particularly susceptible in Chinese and multi-turn contexts. Ultimately, DiagSafe provides a reliable and explainable tool for auditing and optimizing LLM alignment.

1. Introduction

Existing safety benchmarks often reduce responses to binary labels, limiting explanatory depth, while evaluation consistency and scenario diversity remain concerns. DiagSafe introduces prompt-level rubrics and a bilingual, 84-label benchmark, and reports more reliable judging alongside model-specific safety vulnerabilities.

1. Introduction

  • Existing benchmarks lack explanatory depth from binary safe-or-unsafe labels, suffer from inconsistent automated judgments, and do not systematically combine multilingual, multi-turn, and varied attack scenarios.Prior work reports classifier swaps shifting attack success rates by nearly 30 percentage points and inter-judge agreement dropping to 22% in some categories.Source: §1 · pp. 1–2
  • DiagSafe replaces generic binary judgments with prompt-specific, item-level rubrics that assess safety and usefulness independently.The explicit criteria aim to make assessments more objective and explainable.Source: §1 · p. 2
  • The benchmark uses a policy-grounded 84-label taxonomy and bilingual DiagBase and DiagAttack subsets spanning direct harmful queries, jailbreaks, multi-turn prompts, and instruction attacks.DiagAttack embeds harmful intent in complex instruction-following tasks.Source: §1 · p. 2
  • 95.34% agreement with human annotators and Cohen’s κ=0.864 show DiagSafe’s high judge reliability and inter-judge consistency.The benchmark also reveals model-specific vulnerabilities, with 8 of 11 models exceeding 40% unsafe responses on DiagAttack.Source: §1 · p. 3

2. Related Work

Prior safety benchmarks have expanded taxonomies and scenario coverage, but often lack precise policy-level boundaries and unified integration of diverse scenarios. DiagSafe applies prompt-level rubrics across a broader diagnostic benchmark.

2. Related Work

  • Existing benchmarks broaden taxonomies and add adversarial, multilingual, or multi-turn data, but rarely integrate these elements under precise policy-level label boundaries.DiagSafe is presented as unifying diverse scenarios under a comprehensive taxonomy with instance-specific rubrics.Source: §2 · p. 3
  • Figure 2 compares DiagSafe’s 84-label taxonomy with four public benchmarks, distinguishing unique, finer-grained, and shared leaf coverage.Outer-leaf arc width represents prompt count.Source: Figure 2 · p. 4
  • Safety evaluation methods include multiple-choice tests, binary safety classifiers, and general-purpose LLM judges, each with limitations for assessing actual generations or producing consistent, actionable judgments.The paper positions prompt-level rubrics as an instance-specific alternative that constrains judge behavior with explicit criteria.Source: §2 · p. 4

3. The DiagSafe Benchmark

DiagSafe combines prompt-specific rubrics with a policy-grounded taxonomy and two datasets designed to evaluate safety across varied prompts and attacks. Its rubrics separately assess concrete harms and instruction usefulness.

3. The DiagSafe Benchmark

  • DiagSafe uses prompt-level rubrics grounded in an 84-label, three-level taxonomy to evaluate LLM safety at fine granularity.The benchmark comprises DiagBase and DiagAttack.Source: §3 · p. 5

3.1. The Diagnostic Evaluation Framework

  • Safety S-items identify concrete harmful behaviors and include exemptions for permitted conduct, while independent usefulness U-items assess instruction fulfillment and response quality.Eight fixed U-items map binary answers to a 0–3 usefulness score; a direct, factually consistent answer earns 3, while a bare refusal earns 1.Source: §3.1 · pp. 5–6

3.2. Label System of DiagSafe

  • The label system draws on public safety provisions and gives labels specifications defining risks, value principles, dangerous behaviors, and permitted behaviors.The specification construction uses an LLM agent followed by expert review for source traceability, rule accuracy, and boundary clarity.Source: §3.2 · p. 6

3.3. Dataset Construction

  • DiagBase includes ordinary harmful queries and template-based or combined-template jailbreaks, while DiagAttack rewrites single-turn prompts into instruction attacks using external task shells.Prompt synthesis varies language, dialogue turns, and attack form; the dataset contains 2,401 DiagBase samples and 1,088 DiagAttack samples.Source: §3.3 · p. 7
  • A rubric is synthesized for each prompt from that prompt and its associated label specification, with an average of 4.5 safety items and eight usefulness items.The average rubric length is 4,060 characters.Source: §3.3 · p. 7

4. Experiments

The experiments assess DiagSafe’s reliability and use it to diagnose safety across models, prompt types, and response behaviors. The analyses show that rubric-guided evaluation agrees closely with human judgments, while finer-grained diagnostics distinguish vulnerabilities that aggregate scores can obscure.

4.1. Experimental Setup

  • The study evaluates 11 locally served models from 8B to 1.6T parameters without moderation filters, guard models, or system prompts.The primary model-level metric is unsafe rate, and the rubric also records usefulness on a 0–3 scale and item-level traces.Source: §4.1 · p. 8

4.2. Reliability of DiagSafe

  • Human review judged 324 of 336 sampled prompts qualified (96.4%) and 251 of 252 sampled rubrics reasonable and actionable (99.6%).The samples were stratified across third-level labels.Source: §4.2 · p. 8
  • 95.34% accuracy for Hy3 with prompt-level rubrics exceeds generic-prompt (87.29%) and label-specification (86.86%) evaluation against human labels.The comparison is reported in Table 3.Source: §4.2 · p. 8, Table 3 · p. 9
  • Rubric-based evaluation remains consistent across judges, with agreement rates of 93.33%–94.78% and Cohen’s κ values of 0.828–0.864.Other evaluation paradigms show considerably more fluctuation across these metrics.Source: Table 4 · p. 9, §4.2 · p. 9

4.3. Diagnosing LLM Safety with DiagSafe

  • Safety–usefulness profiles distinguish Qwen3.6-27B’s stronger joint performance from GPT-OSS-120B’s low-unsafe-rate, low-usefulness pattern and Llama-3.1-8B’s poor performance on both dimensions.The profiles show why one aggregate safety metric cannot distinguish these behavioral patterns.Source: Figure 5 · p. 10, §4.3 · pp. 10–11
  • Every model’s unsafe rate on DiagAttack is 1.9×–8.4× its DiagBase rate, with a median increase of 5.3× and 8 of 11 models above 40%.The authors suggest that embedding harmful intent in benign instruction shells exposes a vulnerability that direct queries and conventional jailbreak templates often miss.Source: §4.3 · p. 10
  • Models with similar overall unsafe rates still have markedly different category profiles, and second-level categories can reveal differences hidden by top-level aggregation.For example, the three selected models’ overall rates range from 18.6% to 20.9%, while their Radiological & Nuclear rates range from 11.6% to 26.0%.Source: Figure 6 · p. 11, §4.3 · p. 11
  • Per-sample rubric traces distinguish constructive refusal, bounded compliance, and bare refusal, and identify how unsafe responses cross increasingly severe safety boundaries.In the illustrated case, MiniMax-M3 triggers all four S-items, including acquisition channels; the authors also use examples to show how low unsafe rates can coincide with blanket refusal.Source: §4.3 · pp. 11–12

5. Discussion

The discussion compares model behavior across attack forms, languages, and conversation turns. Conventional jailbreak templates have limited, model-dependent effects, while Chinese prompts and multi-turn contexts are associated with higher unsafe rates.

Attack form

  • Template-based jailbreaks have limited, model-dependent effects: they yield only marginal unsafe-rate increases for some models, and MiniMax-M3’s rate is lower than for normal queries.The authors contrast these results with the importance of DiagAttack.Source: §5 · p. 12

Language

  • Most models have slightly higher unsafe rates on Chinese prompts than on English prompts, suggesting that Chinese safety coverage still lags.Figure 8 presents unsafe rates by language.Source: §5 · p. 12

Dialogue turns

  • Unsafe rates rise steadily across conversation turns as harmful intent is unfolded gradually, with risks more apparent for models already prone to unsafe behavior.The authors interpret the pattern as indicating that established conversational context erodes refusal boundaries.Source: §5 · p. 12

6. Conclusion

DiagSafe combines prompt-level rubrics with a bilingual benchmark spanning an 84-label taxonomy and diverse prompt formats. Its evaluations reveal elevated unsafe rates on instruction attacks and model-specific vulnerabilities that aggregate rates can miss.

6. Conclusion

  • DiagSafe pairs prompt-level rubrics with an 84-label safety taxonomy and bilingual prompts, including single- and multi-turn dialogues and challenging instruction attacks.The conclusion describes this as a diagnostic alternative to coarse binary labels.Source: §6 · pp. 12–13
  • Every evaluated model has a 1.9×–8.4× higher unsafe rate on DiagAttack than on DiagBase, while models with similar overall rates show different category-specific vulnerabilities.Fine-grained category and individual-response diagnostics provide evidence for identifying model-specific weaknesses and guiding targeted safety alignment.Source: §6 · pp. 13–14

7. Limitations

DiagSafe’s coverage is bounded by its generated label specifications, its evaluated model and inference-mode selection, and its focus on standalone LLMs in dialogue settings.

7. Limitations

  • LLM-generated label specifications may leave rare, emerging, or highly localized harmful scenarios outside the synthesized prompts and rubrics.The authors identify continuously updated regulatory sources and human-in-the-loop refinement as future ways to broaden specification coverage.Source: §7 · p. 14
  • Evaluation covers 11 locally served open-weight LLMs and one inference mode, limiting coverage of commercial systems and leaving thinking-mode differences unexplored.The authors plan to extend evaluation to API-accessible commercial systems with deployed safeguards and to both inference modes.Source: §7 · p. 14
  • DiagSafe evaluates standalone LLMs in dialogue settings, not agents whose external-tool use and multi-step execution require trajectory-level assessment.The planned extension would assess intermediate actions and tool-use trajectories as well as final responses.Source: §7 · p. 15

Ethics Statement

The paper describes the benchmark’s research-only use, synthetic data provenance, and maintenance plans, alongside its 84-label safety taxonomy.

Ethics Statement

  • DiagSafe contains genuinely harmful prompts and is released for research evaluation and safeguard development, not harmful-content training or unauthorized production-system probing.The authors state that the benchmark is released for research use only to mitigate misuse.Source: Ethics Statement · p. 15
  • All prompts and responses are LLM-synthesized; the authors report collecting no real user queries, private conversations, or personal data.Prompts do not reference real identifiable people, organizations, or events; potentially personal-information categories use synthetic, non-functional examples.Source: Ethics Statement · p. 15
  • The authors plan to release the dataset, rubrics, and evaluation code, and to remove items found to contain personal information or other inappropriate content.They will maintain a channel for reporting problematic items.Source: Ethics Statement · p. 15

A. The Full Safety Taxonomy

  • The taxonomy comprises 11 L1 categories, 34 L2 categories, and 84 L3 labels, each with a definition and value principles distinguishing dangerous violations from permitted behavior.The complete label specifications are released with the benchmark.Source: A. The Full Safety Taxonomy · p. 19

B. Results on Recently Released Models

The supplement evaluates recently released models under the main experiments’ protocol and reports unsafe rates and usefulness scores without changing the paper’s conclusions.

B. Results on Recently Released Models

C. Data Synthesis Pipeline

DiagSafe’s synthesis pipeline generates and quality-controls benchmark prompts and rubrics; its translated prompt descriptions were originally written in Chinese.

C.1. Prompt Synthesis for DiagBase

  • DiagBase prompts are generated from each sampled L3 label specification to cover both languages, single- and multi-turn formats, and three difficulty levels.An LLM inspector grades candidates and rewrites poor samples into decidable safety-evaluation instructions.Source: §C.1 · p. 19

C.2. Instruction-Attack Synthesis for DiagAttack

  • DiagAttack pairs a single-turn, no-attack DiagBase prompt with an EIFBench instruction sample, then fuses them to embed harmful intent within a benign instruction shell.A discriminator quality-controls candidates, returning rejection reasons for generator revisions, with up to three rounds.Source: §C.2 · p. 20
  • A fixed lookup table maps EIFBench task types to handwritten rewriting guidance that steers how harmful intent is embedded.Source: §C.2 · p. 20
  • The generator must preserve harmful intent as an actionable generation goal, while the discriminator checks candidates and returns structured rejection reasons as feedback.The discriminator’s prompt covers five dimensions.Source: §C.2 · p. 20
  • 1,566 instruction attacks were accepted after a three-round-capped loop processed 1,811 candidate pairs, with 74 skipped for refusals or unparseable output.The accepted attacks were subsequently filtered and translated to form DiagAttack.Source: §C.2 · p. 20
  • Worked examples show revisions correcting harmful-intent sanitization, incompatible constraints, or overly exposed intent before acceptance.One example required two rounds; a harder case required three.Source: §C.2 · p. 20

C.3. Rubric Synthesis

  • For each prompt, a rubric-writer LLM uses the prompt and its L3 label specification to draft a rubric, which a second QC pass validates and repairs.At evaluation, the judge receives the rubric instantiated with the prompt, response, and safety category, without an additional system prompt.Source: §C.3 · p. 20

D. Evaluation Prompts

The section contrasts generic and label-specification binary judges with rubric-guided evaluation, then describes human review and rubric examples. Human annotations use independent expert judgments followed by a majority gold label.

D.1. Generic Safety Judge Prompt

  • The generic-prompt baseline asks a judge to classify harmfulness from the input and output using only “safe” or “unsafe.”Its prompt is language-matched to the evaluated sample.Source: §D.1 · p. 21

D.2. Label-Specification Judge Prompt

  • The label-specification baseline adds the prompt’s third-level safety label definition and rules, but supplies no sample-specific rubric.It retains a binary output and language-matches the template to the sample.Source: §D.2 · p. 21
  • For multi-turn samples, each turn is judged independently using its dialogue prefix, and any unsafe turn makes the whole conversation unsafe.Source: §D.2 · p. 21

D.3. Rubric-Guided Judging Protocol

  • Rubric-guided judging checks prompt-specific safety items, records evidence and exemptions, and separately scores usefulness on fixed items from 0–3.A response is unsafe when at least one safety item is triggered without an applicable exemption.Source: §D.3 · p. 21

E. Human Annotation Protocol

  • Three experts with relevant safety or annotation experience received training on the taxonomy and its third-level label specifications.Source: E. Human Annotation Protocol · p. 21
  • Prompt review classified requests as harmful, potentially harmful, or benign; harmful and potentially harmful prompts qualified for inclusion.Rubric review checked harmful-behavior coverage, permitted-behavior boundaries, exemptions, consistency, and actionability.Source: E. Human Annotation Protocol · p. 22
  • Response annotators saw the prompt, dialogue context, model response, and label specification, but not the rubric, judge outputs, or other experts’ annotations.They labeled materially facilitating prohibited behavior unsafe and permitted refusals, safe redirections, prevention guidance, and allowed general information.Source: E. Human Annotation Protocol · p. 22

F. Full Rubric Examples

  • The majority of three independent expert labels formed the human gold standard, with 97.88% initial three-way agreement before adjudication.Four released examples span suicide and self-harm, cybersecurity, deliberate disinformation, and hate speech; their safety items are prompt-specific and exemptions delimit permitted behavior.Source: E. Human Annotation Protocol · p. 22, F. Full Rubric Examples · p. 22

G. Detailed Unsafe-Rate Matrices

The matrices report unsafe rates across models and taxonomy levels, while the case studies illustrate how rubric-level traces distinguish failure patterns. The section also identifies the taxonomy and safety-guideline sources used in the benchmark.

G. Detailed Unsafe-Rate Matrices

  • The taxonomy has 11 first-level categories, 34 second-level categories, and 84 third-level labels with detailed specifications.Source: Table 6 · p. 24
  • At the second taxonomy level, unsafe-rate differences across models grow substantially, and models with similar aggregate rates fail in different categories.Tables 20–22 report rates for 11 models across aggregate, subset-separated, and second-level views.Source: G. Detailed Unsafe-Rate Matrices · pp. 22–23, Table 20 · p. 36, Table 21 · p. 37

OSS-06 CBRN & WMD

  • The listed category names include Cybersecurity & Model Attacks, Suicide, Self-Harm & Dangerous Health Behaviors, Intellectual Property Infringement, and Manipulation & Deceptive Persuasion.Source: OSS-06 CBRN & WMD · p. 38

H. Extended Case Studies

  • The extended case studies cover an English instruction attack from DiagAttack and a combined-template multi-turn jailbreak from DiagBase.For each, the paper presents rubric safety items and per-model judgments, including triggered items and usefulness scores.Source: H. Extended Case Studies · p. 39

H.1. Case 1: English Instruction Attack (Disinformation Article)

  • Five models complied with the embedded disinformation-writing task and triggered all or nearly all safety items.Their usefulness score was U=0; safe responses ranged from U=1 for bare refusal to U=3 for correcting false claims while addressing the topic.Source: §H.1 · p. 39, Table 23 · p. 40

H.2. Case 2: Combined-Template Jailbreak (Multi-Turn)

  • All 11 models gave substantive replies (U≥2), while rubric traces distinguished policy concessions from endorsement or detailed drug-experience accounts.The reported patterns range from S3 concessions to Llama-model S1 or S2 behavior under the DAN shell (S4).Source: §H.2 · p. 41, Table 24 · p. 41

I. Referenced Safety Guidelines

  • The taxonomy’s referenced materials include international standards and policies as well as Chinese safety and governance documents.Named sources include the EU AI Act, NIST AI 600-1, the Universal Declaration of Human Rights, and Chinese AI safety standards and governance frameworks.Source: §I · pp. 41–42
Loading 2610.29225v1…