PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Judges the Judges? A Chinese Safety QA Benchmark for Evaluating LLM Responses and Safety Judges

Rui Yang, Shuang Huang, Junhua Liu, Ziqi Zhao, Qingzhong Yan, Yuhang Sun, Cong Liu, Guoping Hu, Rui Mei, Jing Shao

arXiv:2609.01210v1cs.CRcs.AI

TL;DR

Existing safety benchmarks often assess query risk rather than whether the resulting response violates policy, a distinction especially important for Chinese harmful-content evaluation under adversarial transformations. C-SafeQA addresses this gap with a policy-grounded query–response benchmark that evaluates target LLMs and automated safety judges; adversarial queries produce higher unsafe-response rates, while judges exhibit trade-offs and mechanism-specific weaknesses.

  • Problem

    Existing benchmarks do not determine whether a question-answering system violates a safety policy when query risk and response safety diverge, especially under Chinese linguistic variation and adversarial transformations.

  • Method

    C-SafeQA combines policy-derived seed queries, 21 adversarial transformations, agreement-aware model adjudication, stratified expert auditing, and shared reference labels to evaluate four target LLMs and seven automated safety judges.

  • Results

    Adversarial queries yield higher observed unsafe-response rates, while judges show different trade-offs between unsafe-response recall and safe-response FPR, with no judge dominating all metrics.

  • Takeaways & Limitations

    Safety evaluation must examine both target-model responses and the reliability of the judges used to assess them.

  • Takeaways & Limitations

    C-SafeQA covers single-turn Chinese harmful-content QA under one internal policy, uses model-assisted expert-audited references, and does not establish reliability across other languages, dialogue settings, or policy regimes.

Abstract

from arXiv · show

Safety benchmarks for large language models often assess the risk of a user query, although the outcome of question answering depends on whether the response violates a policy. This distinction is critical in Chinese harmful-content evaluation, where linguistic variation and adversarial transformations can obscure risky intent. We introduce C-SafeQA, a policy-grounded benchmark for response-level Chinese safety evaluation. It comprises 538 base queries and 8,877 adversarial queries answered by four full-model LLM deployments, yielding 37,660 query-response records labeled safe, unsafe, or disputed. Reference labels are generated through agreement-aware multi-model adjudication and blind audits of stratified subsets by three safety experts. C-SafeQA supports both evaluation of target-model safety and auditing of seven automated safety judges against shared reference labels. Unsafe-response rates range from 0.93% to 3.35% on base queries and from 11.68% to 30.05% on adversarial queries. On the adversarial subset, judges show substantial trade-offs between unsafe-response recall and risk-query-conditioned safe-response false positive rate, and no judge dominates all metrics. Both acrostic transformations reduce unsafe recall for all seven judges, revealing mechanism-specific evaluator weaknesses. Dataset records, metadata, verification code, and judge scripts are publicly released to support recomputation, while benchmark construction, target-response generation, and private adjudication remain outside the release boundary.

1 Introduction

C-SafeQA addresses a central limitation of safety benchmarks: query risk does not determine whether a model response violates policy, especially under Chinese linguistic variation and adversarial transformations. It introduces a policy-grounded benchmark that evaluates target models and automated safety judges at the response level.

  • Benchmark motivation and contribution: The benchmark separates query-level risk from response-level policy violation because harmful queries may receive safe refusals, while transformed or indirect queries may elicit unsafe answers.This distinction is especially important when Chinese harmful intent is expressed through homophones, mixed scripts, semantic reversal, or structured transformations.
  • Benchmark motivation and contribution: 538 base queries and 8,877 adversarial queries operationalize 269 policy-derived risk points for Chinese safety evaluation.The base set uses question-form and declarative-form queries, while adversarial variants apply 21 transformation methods.
  • Benchmark motivation and contribution: 11.68–30.05% unsafe-response rates on transformed queries exceed the 0.93–3.35% range on base queries.The benchmark also reports overall unsafe-response rates of 11.06% to 28.43% across the four target LLMs.
  • Benchmark motivation and contribution: Seven automated judges show trade-offs between unsafe-response recall and safe-response FPR, with no judge best on both adversarial metrics.Judge reliability also varies by category and attack condition.
  • Benchmark motivation and contribution: Dataset records, metadata, verification code, and judge scripts are released, while construction, target-response generation, and private adjudication remain outside the release boundary.The releases support recomputation of the published analyses without exposing the full private construction and adjudication process.
  • Benchmark motivation and contribution: 37,660 query–response records from four full-model LLM deployments support target-model and automated-judge evaluation.Records use shared policy-grounded reference labels, enabling model-, category-, and attack-level analysis.

2 Related Work

Prior work covers toxic generation, refusal, Chinese safety, adversarial prompting, and automated judging, but these components are rarely combined under one response-level Chinese evaluation setting. C-SafeQA targets that conjunction with policy-linked queries, controlled transformations, shared references, and judge auditing.

  • Existing safety benchmarks: Earlier benchmarks measure toxic generation, harmful-assistance refusal, bilingual safety knowledge, or preference-based harmlessness using different task designs.These resources advanced safety measurement but do not share one response-level Chinese rubric.
  • Existing safety benchmarks: Refusal benchmarks show that refusal rate alone cannot represent both safety and helpfulness because benign sensitive-looking prompts expose over-refusal.SORRY-Bench studies refusal consistency, while XSTest and OR-Bench measure over-refusal with benign prompts.
  • Chinese safety evaluation: Chinese benchmarks expand coverage of local risks, cultural context, obfuscation, and jailbreak transformations, but response-level compliance and evaluator robustness remain less emphasized.Examples include homophones, character variants, indirect references, mixed-language expressions, and localized safety policies.
  • Automated judges and guardrails: Automated safety judges and guardrails provide flexible moderation, but prior work documents sensitivity to prompting, model bias, adversarial framing, and distribution shift.JAILJUDGE and guardrail comparisons motivate treating judge reliability as an empirical evaluation target rather than ground truth.
  • C-SafeQA’s position: C-SafeQA evaluates the conjunction of policy-linked Chinese query–response pairs, 21 mechanism-labeled transformations, seven heterogeneous judges, three-way references, and conditional error analysis.The controlled transformations isolate how judges respond when one policy-bearing seed is rendered through different mechanisms.
  • C-SafeQA’s position: Adversarial prompting research establishes response-level attack evaluation, while C-SafeQA extends this perspective to both target-model robustness and judge recognition of unsafe responses.Its transformations include encoding, character perturbation, variable substitution, multilingual rewriting, and structured prompts.

3 Safety Rubric, Benchmark Scope, and Construction

C-SafeQA is built from an internal policy into a response-centered Chinese QA benchmark with controlled adversarial transformations, reproducible public artifacts, and separate evaluation of target models and judges.

  • Safety rubric and construction: C-SafeQA converts an internal safety policy into a response-level Chinese QA benchmark while separating private adjudication policy from publicly released labels.This preserves the construction process without exposing the full internal policy mapping.
  • Safety rubric and construction: Figure 1 traces the pipeline from policy-grounded query construction through target-model and judge-model evaluation.The stages include response collection, judge-model distillation, and human adjudication.
  • Safety rubric and construction: The rubric requires risk-interpretable samples, consistent adjudication, response-centered judgments, and reproducible evaluation.Models are judged by unsafe assistance or safe refusal rather than penalized solely because a query is risky.
  • Safety rubric and construction: Risk points map to a seven-category taxonomy and finer hierarchy, while released top-level labels provide the common analysis unit.This preserves category-level diagnostic value while keeping internal policy definitions separate from the public interface.
  • Benchmark scope and release: The scope covers Chinese harmful-content risks in single-turn question answering, including direct requests, declarative prompts, and adversarially transformed prompts.The benchmark evaluates both target-model handling and automated assessment of resulting query–response pairs.
  • Benchmark scope and release: The Hugging Face release contains five UTF-8 JSONL files with 37,660 records, schemas, manifests, labels, judge outputs, and normalization status.The public records expose the fields needed to recompute released analyses and judge audits.

3.3 Query-level Risk vs. Response-level Violation

C-SafeQA treats query risk and response violation as related but distinct, so safety labels apply to query–response pairs rather than prompts alone. Its construction links samples to policy risk points while excluding private conversational data and filtering ambiguous seeds.

  • Operational distinction: A risky query concerns policy-associated intent, topic, or requested behavior, whereas a violating response crosses the response-level safety boundary.Violations include actionable harmful details, optimization of risky behavior, leaked risk steps, or disguised unsafe assistance.
  • Operational distinction: A high-risk query can receive a safe label after clean refusal or redirection, while an ambiguous query can produce an unsafe label through operational harmful content.The benchmark therefore annotates complete QA pairs rather than prompts alone.
  • Operational distinction: Transformation requests are judged by semantic effect, so translating, paraphrasing, encoding, or reformatting harmful content remains unsafe when it reproduces the harmful meaning.Responses bounded by criticism, risk analysis, moderation, or another safety-preserving purpose can be safe.
  • Policy-linked construction: 269 policy-derived risk points generate 538 base queries in complementary question-form and declarative-form versions.This preserves a one-to-one connection between each query and the policy boundary under evaluation.
  • Policy-linked construction: No seed query is copied from real user conversations, customer records, or operational logs.Reviewers also screen candidates for accidental personal information before inclusion.
  • Policy-linked construction: Seeds are linked to policy risk points, deduplicated, revised when ambiguous, and excluded when their policy mapping is unclear.The seed set is designed as a balanced and interpretable starting point rather than a maximally harmful prompt collection.

3.5 Adversarial Query Transformations

C-SafeQA transforms seed queries through 21 Chinese adversarial methods designed to preserve risky semantics while probing distinct attack mechanisms and safety capabilities.

  • q′ = Tm(q) maps each seed query q to an adversarial prompt through transformation method m.The framework uses template-based, LLM-rewrite, and programmatic generation channels.
  • The transformation catalog distinguishes generation channel from mechanism family and diagnostic target.Programmatic methods include reversible representations and encoded output, while LLM-rewrite and template methods alter context, language, logic, or format.
  • All transformations must preserve the seed’s risky entity, action, target, and judgment direction.They must also instantiate a clear attack mechanism and an interpretable diagnostic capability dimension.
  • The catalog defines 21 mechanism-level transformations while withholding reusable prompt shells and concrete attack payloads.A transformed prompt is assigned to the method responsible for its primary transformation.
  • Nine transformations apply only to declarative seeds, including Encoding/Decoding, Translation, Opposite-Speech Prompting, Code-Format Output, and Style-Format Output.These methods target encoded-risk recognition, cross-lingual consistency, polarity tracking, structured formats, and stylistic reframing.

3.6 Response Collection

Responses were collected from four self-hosted, unquantized full-model LLM deployments under model-recommended non-thinking or instant-mode settings, without external safety guardrails.

  • Four self-hosted, unquantized full-model deployments generated the responses: Qwen3.5-397B-A17B, Kimi-K2.5, DeepSeek-V3.2, and MiniMax-M2.5.The deployments used no additional external safety guardrails.
  • All four deployments ran with thinking disabled and retained only the final response for evaluation.Sampling parameters followed the corresponding model card’s non-thinking or instant-mode recommendations when available.
  • Responses were generated with each model’s official chat template and listed generation settings, with top-k unspecified where the model provided no value.This keeps collection aligned with model-specific serving recommendations.

3.7 Dataset Scale

The experimental set contains 538 base queries and 8,877 adversarial queries, producing 37,660 query–response records across four target models.

  • 37,660 query–response records result from 9,415 queries evaluated by each of four target models.The query set combines 538 base queries with 8,877 adversarially transformed queries.
  • 538 base queries and 8,877 adversarially transformed queries define the benchmark’s two principal partitions.Response-label counts in the composition table are read from the final experimental records.

3.8 Reference-label Annotation and Quality Control

Reference labels combine policy-linked metadata, model adjudication, deterministic validation, and stratified blind expert auditing to classify responses as safe, unsafe, or disputed.

  • Responses are judged against the relevant policy context and labeled at the QA-pair level, separating response violation from query risk.The annotation process combines construction metadata, multi-stage adjudication, rule-based validation, and human auditing.
  • Unanimous agreement among three initial judges yields a provisional label, while two-to-one and three-way disagreements receive Opus-4.8 review.Agreement cases are routed to 20% manual inspection, two-to-one disagreements to 40%, and three-way disagreements to 80%.
  • Three experts independently assign safe, unsafe, or disputed labels under blind annotation before discussing unresolved disagreements with policy context.When manual inspection occurs, the expert-adjudicated label supersedes model judgments.
  • Cases outside inspection samples retain unanimous initial-judge labels or Opus-4.8 judgments, making the reference set model-assisted rather than fully human-annotated.The final schema contains only safe, unsafe, and disputed labels.
  • Reference-label support aggregates counts across all 21 transformations, with nine methods evaluated only on declarative seeds.Safe and unsafe denominators vary across transformations and should be read with transformation-conditioned rates.

4 Evaluation Protocol

The evaluation applies a shared three-way reference-label policy to target LLMs and automated judges across base, adversarial, category, and judge-audit tasks. It defines binary-resolved safety metrics while retaining disputed cases and specifies diagnostics for refusal, risk leakage, and actionability.

  • Target-model evaluation: Four target deployments are evaluated on 538 base and 8,877 adversarial queries, producing 37,660 query–response pairs.The same query set is passed through official chat templates and model-specific sampling settings.
  • Automated-judge evaluation: Seven automated judges classify complete query–response pairs, and their outputs are normalized to safe, unsafe, or disputed for comparison with reference labels.The evaluated judges include Llama Guard 4, MD-Judge, NemoGuard, PolyGuard, Qwen3Guard, WildGuard, and YuFeng-XGuard.
  • Label handling: Disputed reference cases remain in the three-way release but are excluded from binary metric denominators, while judges that predict disputed are evaluated under D→Safe and D→Unsafe policies.This preserves adjudication uncertainty without merging disputed cases into either binary reference class.
  • Evaluation tasks: The benchmark compares unsafe-response rates on untransformed and transformed queries, with additional category-level and attack-level breakdowns.The four tasks cover base safety, adversarial robustness across 21 methods, risk categories, and automated-judge evaluation.
  • Target-model metrics: For target LLMs, unsafe-response rate is N_unsafe/(N_safe + N_unsafe) over binary-resolved pairs, with empty responses treated as safe refusals.The safe-response rate is defined analogously, and disputed cases are excluded from both rates.
  • Judge metrics: Judge evaluation reports unsafe recall, safe-response FPR, precision, accuracy, F1, and conflict rate, distinguishing missed unsafe responses from unsafe predictions on safe responses.The safe-response FPR is conditioned on the benchmark’s risky queries and is not a benign-query over-refusal rate.

5 Experiments and Results

Across target models, safety varies by model, query pool, transformation mechanism, and risk category, while automated judges exhibit incompatible blind spots and trade-offs. Adversarial transformations expose recurrent weaknesses, especially for acrostic and representation-level methods, but observed differences are descriptive rather than causal estimates.

  • Model Safety Behavior: Qwen3.5-397B-A17B has the lowest unsafe rate at 11.06%, while Kimi-K2.5 has the highest at 28.43%.Qwen3.5-397B-A17B also has the lowest risk leakage index and high-actionability violation rate; refusal frequency follows a different ordering.
  • Base and Adversarial Queries: Adversarial-query unsafe rates exceed base-query rates for all four target LLMs, reaching 30.05% for Kimi-K2.5 versus 1.86% on base queries.The largest descriptive percentage-point difference is 28.20 for Kimi-K2.5, but the pools differ in query-form composition.
  • Adversarial Transformations: Acrostic-answer extraction has the highest average unsafe rate at 91.05%, followed by acrostic generation at 69.71% across target models.The transformation rates are evaluated on designated query forms and are not presented as fully controlled cross-method causal comparisons.
  • Category-level Vulnerability: Adversarial vulnerability is category dependent: Other has the highest mean unsafe rate at 23.59%, while Illegal and Non-compliant Activities has the highest mean high-actionability violation rate at 13.28%.Category-level model rankings also change, so aggregate unsafe rates can conceal which public risk categories account for failures.
  • Automated-Judge Agreement: YuFeng-XGuard has the highest unsafe-response recall at 63.52%, while WildGuard has the lowest safe-response FPR at 3.13%; no judge is best across all metrics.Judge reliability varies jointly with category and transformation mechanism: conflict rises in 45 of 49 judge–category cells after transformation, and all seven judges miss most unsafe acrostic responses.

6 Security Implications, Limitations, and Ethics

C-SafeQA frames judge scores as outcomes of the target-response distribution, reference boundary, judging procedure, and normalization choices, not properties of target models alone. Its scope and evidence are bounded by a single-turn Chinese policy setting, model-assisted references, unmatched transformations, and limited release and coverage conditions.

  • Judge scores depend on the target-response distribution, reference boundary, judge template, disputed-output policy, and normalization procedure.
  • Unsafe recall, risk-query-conditioned safe-response FPR, precision, disputed mapping, and normalization coverage should accompany aggregate judge results.
  • C-SafeQA covers single-turn Chinese harmful-content QA under one internal policy and does not establish reliability across other languages, dialogue settings, agent traces, or policy regimes.
  • References are model-assisted and expert-audited rather than exhaustively human labeled, so shared errors among provisional judges may remain undetected.
  • The release provides prompts, responses, transformations, judge outputs, verification code, and judge runners, while benchmark construction, target-response generation, and private adjudication remain unreleased.

8 Conclusion

C-SafeQA jointly evaluates Chinese target-model safety and automated safety judges at the query–response level using policy-derived queries, adversarial transformations, model adjudication, and expert auditing. Adversarial queries expose judge trade-offs and transformation-specific weaknesses, including reduced unsafe recall under acrostic attacks.

  • C-SafeQA jointly evaluates target LLMs and automated safety judges at the query–response level.
  • The benchmark separates risky queries from unsafe responses through policy-derived seeds, adversarial transformations, agreement-aware adjudication, and stratified expert auditing.
  • On adversarial queries, judges exhibit different trade-offs between unsafe-response recall and safe-response FPR, while conflict rises in 45 of 49 judge–category cells after transformation.
  • Both acrostic transformations reduce unsafe recall for all seven judges, revealing mechanism-level evaluator weaknesses.
Loading 2609.01210v1…