Source-linked AI summary
A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
Leo Schwinn, Moritz Ladenburger, Tim Beyer, Mehrnaz Mofakhami, Gauthier Gidel, Stephan Günnemann
TL;DR
LLM judges are widely used to measure safety, but existing validation does not capture distribution shifts in adversarial evaluations. Using a human audit of 6,442 samples, this paper finds near-random judge performance and inflated attack success rates, then proposes benchmarks and practices for more reliable evaluation.
Problem
Existing validation protocols do not adequately test LLM judges under attack, victim-model, and semantic distribution shifts relevant to adversarial safety evaluation.
Method
The paper audits judge reliability against 6,442 human-verified samples across attacks, victim models, and semantic behavior categories.
Results
LLM judges perform on average no better than a random coin flip under distribution shifts inherent to adversarial robustness evaluations.
Takeaways & Limitations
The paper introduces ReliableBench and JudgeStressTest and recommends multiple judge-positive samples, precision-corrected ASR, and filtering for consistently judgeable behaviors.
Takeaways & Limitations
The study does not cover defense-induced shifts and remains limited to the tested attack–model combinations.
Abstract
from arXiv · showhide
Automated \enquote{LLM-as-a-Judge} frameworks have become the de facto standard for scalable evaluation across natural language processing. For instance, in safety evaluation, these judges are relied upon to evaluate harmfulness in order to benchmark the robustness of safety against adversarial attacks. However, we show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming: diverse victim models exhibit distinct generation styles, attacks distort output patterns, and semantic ambiguity varies significantly across jailbreak scenarios. Through a comprehensive audit using 6642 human-verified labels, we reveal that the unpredictable interaction of these shifts often causes judge performance to degrade to near random chance. This stands in stark contrast to the high human agreement reported in prior work. Crucially, we find that many attacks inflate their success rates by exploiting judge insufficiencies rather than eliciting genuinely harmful content. To enable more reliable evaluation, we propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures. Data available at: https://github.com/SchwinnL/LLMJudgeReliability.
1. Introduction
LLM judges are widely used to scale safety evaluation, but standard validation overlooks distribution shifts introduced by attacks, victim models, and semantic behaviors. A human audit finds that these shifts can reduce judge reliability to near-random performance and inflate reported attack success rates, motivating more reliable evaluation practices.
- Motivation: Standard validation compares judges with human ratings on static held-out data, but adversarial evaluations introduce distribution shifts that this process does not capture.These shifts arise from attacks, victim-model differences, and semantic complexity.
- Main finding: The paper reports that LLM judges in adversarial safety evaluations perform on average only slightly better than a random coin-flip.This conclusion contrasts with the high agreement typically reported under prior validation settings.
- Audit: 6,442 human-verified samples support an audit showing that adversarial-evaluation shifts degrade judge performance to near-random chance.The audit examines attacks, victim models, and semantic behavior categories.
- Evaluation validity: Judge failures can inflate reported Attack Success Rates because attack gains may exploit false positives rather than elicit genuinely harmful content.The paper describes corrected ASR as scaling results by judge precision, the probability that a judge-positive is a true positive.
- More reliable evaluations: ReliableBench focuses on more consistently judgeable behaviors, while JudgeStressTest exposes hard judge-failure cases for developing robust evaluators.The paper also recommends multiple judge-positive samples per behavior and correcting ASR for judge precision.
2. Related work
Prior work documents adversarial manipulation, stylistic bias, miscalibration, and semantic failures in LLM judges. However, it has not systematically measured their joint effects under adversarial safety distribution shifts using human-labeled ground truth.
- Existing approaches: Existing LLM-as-a-Judge systems include general-quality judges, safeguard models, ensembles, and standardized safety benchmarks.Examples include GPT-4, Llama Guard, AEGIS, WildGuard, and HarmBench.
- Adversarial vulnerabilities: Known vulnerabilities include prompt injection, high-perplexity or obfuscated inputs, and optimization attacks that can cause harmful outputs to be misclassified as safe.One cited evaluation found optimization-based attacks misclassified 100% of harmful outputs as safe.
- Distribution sensitivity: Judges can prioritize stylistic features over factual correctness or safety, with an apologetic prefix shifting preferences by up to 98%.Other work reports reliance on malicious linguistic patterns rather than authentic harmful knowledge.
- Calibration and semantics: Judge calibration worsens under distribution shifts such as jailbreak attacks or responses generated by different models.Reported failures include overconfident predictions and difficulty assessing the semantic validity of harmful outputs.
- Open gaps: Prior work leaves the joint effects of attack, model, and data shifts, human-grounded semantic challenges, and judge-hacking through optimization or sampling insufficiently examined.These gaps motivate a systematic audit in adversarial safety evaluations.
3. Method
The study evaluates judge reliability across attacks, victim models, behaviors, and semantic categories using human labels and multiple automated judges. Its design includes attack strategies with different degrees of judge interaction and a labeled dataset assembled through automatic filtering and human annotation.
- Data and queries: The study uses 100 randomly subsampled HarmBench queries to distribute labeling resources across attack–model combinations.HarmBench was selected for community comparability and fine-grained semantic and functional annotations.
- Victim models: Four open-weight victim models span different architectures and scales, enabling analysis of model-family and model-size effects on judge errors.The models are Gemma-3-1B, Llama-3.1-8B, Gemma-27-B, and Qwen-3-32B.
- Automated judges: Human labels are compared with ratings from AegisGuard, the Llama-2-13B HarmBench classifier, JailJudge, and LlamaGuard-3.The evaluation uses a diverse set of judges from prior work.
- BoN: BoN generates N independent responses, selects the response judged most harmful, and adds random character-level query perturbations.This sampling procedure can increase the chance of finding a benign response that triggers a judge false positive.
- Attacks and judge interaction: The attack set spans direct prompting, GCG, GCG-REINFORCE, BoN, and PAIR, selected to vary their interaction with judge feedback.GCG-REINFORCE uses judge feedback directly, whereas BoN can implicitly exploit judge weaknesses through extensive sampling.
- Human labeling: Human labeling uses a five-level harmfulness scale, with levels 3–5 treated as harmful and levels 1–2 as benign.The protocol assesses model intent and compliance rather than technical accuracy or feasibility.
- Dataset construction: 6,642 samples comprise the final dataset after automatic judge filtering and human annotation, including 2,370 self-labeled samples and 4,272 Labelbox samples.Labelers passed a quiz, and manual review achieved 95% agreement for harmful-versus-benign classification.
4. Results
Across adversarial evaluation settings, judge reliability varies sharply with attack, victim model, and behavior, often approaching random chance. These failures distort attack-success measurements, while consensus-based and judgeability-focused tools provide partial diagnostics and curated benchmarks.
- Judge Failures: Attack, model, and data shifts substantially degrade judge reliability in adversarial evaluations.Shifts arise from distorted attack outputs, victim-model linguistic variation, and behavior-dependent semantic difficulty.
- Judge Failures: 2746 balanced samples show that judge accuracy depends strongly on the specific attack–victim-model combination.Direct Prompting accuracy ranged from 45% for Llama-3.1-8B with HarmBench to 58% for Gemma-27-B with the same judge.
- Judge Failures: AUROC values for JailJudge ranged from 0.48 on GCG-R to 0.64 on GCG, and threshold adjustment did not resolve near-random performance.The ROC curves largely hovered around the diagonal across attack distributions.
- Judge Failures: Individual judge scores lacked consistent correlation with human ratings, and averaging multiple judges did not produce a clear correlation.This indicates that ensembles did not reliably mitigate systematic judge failures in the examined setting.
- Influence of Attack Optimization: Attack optimization showed no clear increase in human-rated harmfulness, judge scores, or judge accuracy over the optimization trajectory.The results suggest that current optimization strategies may target judge-specific noise or reward signals unrelated to the adversarial objective.
- Influence on Attack Success: Correcting attack success rates for judge precision can substantially reduce perceived attack effectiveness.For BoN, a single judge-positive sample provides low confidence of a true jailbreak, and multiple judge-positive samples improve reliability.
- Measuring Inter-judge Agreement: Judge Concordance measures panel consensus but does not reliably indicate correctness against human ground truth.Near-unanimous agreement can coexist with systematic failure, so consensus is better treated as a diagnostic than as a correctness proxy.
- Measuring Inter-judge Agreement: ReliableBench targets consistently judgeable behaviors, while JudgeStressTest isolates systemic failure cases across judges.ReliableBench’s judgeability generalizes across models and attacks; JudgeStressTest contains 971 samples.
5. Conclusion
The study finds that LLM judges can perform near random chance under adversarial-evaluation distribution shifts, distorting attack-success estimates. It introduces ReliableBench and JudgeStressTest to support more reliable future evaluation.
- LLM judges perform, on average, no better than a random coin flip under distribution shifts inherent to adversarial robustness evaluations.
- Judge unreliability can distort reported attack success rates, including gains attributed to Best-of-N.
- ReliableBench provides a more reliable safety-evaluation dataset, while JudgeStressTest supplies challenging cases with ground-truth human labels.
Impact Statement
The study’s findings may improve future judge reliability while also propagating dataset biases and prioritizing adversarial robustness over other safety dimensions. These limitations affect how broadly the results and their implications should be interpreted.
- Reliance on HarmBench may propagate biases toward single-turn textual safety and specific harm categories.
- The study’s exclusive focus on adversarial robustness might prioritize that metric over other important safety dimensions.
- The research highlights the need for more robust evaluation standards before deploying autonomous systems in high-stakes environments.
A. Extended Results
This appendix section provides additional results and visualizations that are not included in the main text.
- The appendix contains additional results and visualizations omitted from the main text.
A.1. Judge Accuracy Heatmaps
Additional heatmaps show substantial variability in judge accuracy across attacks and victim models for Gemma-3-1B and Qwen-3-32B. Darker heatmap colors indicate lower accuracy.
- Judge accuracy varies significantly across attacks and victim models for Gemma-3-1B and Qwen-3-32B.
- Darker colors in the heatmaps indicate lower judge accuracy.
A.2. Semantic Categories
The paper uses abbreviations for HarmBench’s semantic harm categories, including chemical biological, cybercrime intrusion, and harassment bullying.
- CB denotes chemical biological, CI denotes cybercrime intrusion, and HB denotes harassment bullying.
- HF denotes harmful, ILG denotes illegal, and MD denotes misinformation disinformation.
A.3. ROC Curves
ROC analyses show that judge classification often approaches random performance under adversarial-evaluation distribution shifts, while larger victim models are generally easier to judge.
- Performance often hovers near the diagonal across judges and models, indicating poor classification under adversarial-evaluation distribution shifts.
- Larger models such as Qwen-3-32B tend to be easier to judge than smaller models such as Llama-3.1-8B.
A.4. Correlation with Human Ratings
The paper examines judge scores, true-positive verification, behavior-level agreement, and ReliableBench construction. These analyses show that judge-positive results may require larger samples, agreement does not guarantee correctness, and a selected behavior subset is more reliably judgeable.
- Correlation with Human Ratings: For some attacks, particularly sampling-based attacks such as BoN, a single judge-positive indication provides low confidence of actual success.Larger sample sizes are needed for reliable verification.
- Attack Analysis: Filtering samples labeled positive by StrongReject likely increased non-refusals, while harmfulness remained relatively high and did not increase during optimization.These results provide evidence that current attack optimizers do not cause more harmful content during optimization.
- Concordance Scores: High inter-judge concordance does not consistently correlate with high objective accuracy across 100 behaviors.The paper therefore treats judge agreement as an insufficient proxy for ground-truth correctness.
- ReliableBench: ReliableBench remains robustly easier to judge despite variability across evaluated model-attack configurations.The finding is supported by a consistent decrease in mean accuracy as more behaviors are included.