Source-linked AI summary
Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
Justin D. Norman, Michael U. Rivera, D. Alex Hughes
TL;DR
Judge validation commonly relies on exact match, which does not correct for chance and overstates discriminative ability. The paper evaluates 21 judges across benchmarks and protocols, finding cross-benchmark ranking shifts, universal kappa deflation, and a coexistence of reproducibility and position bias. It distills these results into a Minimum Viable Validation Protocol.
Problem
Judge validation relies heavily on exact match, which does not correct for chance and overstates discriminative ability.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The Minimum Viable Validation Protocol recommends chance correction, at least two contrasting benchmarks, joint consistency and bias measurement, and not foregrounding verbosity bias under standard pairwise rubrics.
Source: §6 · p. 8Verbosity-bias findings are limited to a single pairwise rubric and length-differential operationalization, and should not be read as showing that the bias is universally solved.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability. We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments. Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across benchmarks, high test--retest reliability (>0.95) coexists with severe position bias (>0.10) in two production-deployed judges (instantiating a consistency--bias paradox), and verbosity bias is small (<0.011) across our cohort under a single pairwise rubric. We distill these into a Minimum Viable Validation Protocol.
1 Introduction
LLM-as-a-Judge is widely deployed despite recurring reliability problems, while validation still commonly relies on exact match, which does not correct for chance and overstates discriminative ability. This paper evaluates judges across benchmarks and protocols to address the lack of large-scale comparative evidence.
1 Introduction
- LLM judges are widely deployed in production evaluation pipelines despite documented reliability issues with language-model reasoning.The passage describes hundreds of documented LLMOps deployments and recurring issues including inconsistency, scoring bias, weak domain calibration, and absent meta-evaluation standards.Source: §1 · p. 1
- No prior work had conducted a large-scale, cross-benchmark, multi-protocol evaluation of judge reliability across model families and generations.The study evaluates 21 models from nine providers across three benchmarks and three protocols, producing 541,000 judgments over 118 runs.Source: §1 · p. 1
- Across 21 judges, exact-match validation misses chance-corrected agreement, and the evaluation reports five findings spanning deflation, ranking shifts, consistency, and bias.The findings also include a comparison of benchmark discriminability.Source: §1 · p. 1
2 Related Work
Prior work documents LLMaJ’s expanding use and reliability challenges, while this section defines metrics and introduces two diagnostic concepts. It also situates this evaluation alongside proposed approaches to improving reliability.
2 Related Work
- LLM-as-a-Judge expanded from MT-Bench and Chatbot Arena into varied evaluation settings and industrial pipelines.Examples include G-Eval, AlpacaEval, Arena-Hard, and WildBench.Source: §2 · pp. 1–2
- LLMaJ reliability work spans studies of inconsistency, reliability coefficients, rating indeterminacy, and latent reliability dimensions; Bavaresco et al. (2025) surveyed 11 judges across 20 tasks.That survey recommends validation against task-specific annotation before deployment.Source: §2 · p. 2
2.1 LLM Judge Metrics and Definitions
- The section defines exact match as the proportion of verdicts matching human labels, with pairwise comparisons excluding human-label ties in the reported variant.It notes that exact match does not correct for chance and is sensitive to benchmark label distribution.Source: §2.1 · p. 2
- Test–retest reliability measures agreement across independent re-evaluations, while position-flip rate and verbosity bias operationalize distinct bias dimensions.Verbosity bias is operationalized as the Pearson correlation between response-length differential and the judge’s verdict.Source: §2.1 · p. 3
- 21 judges show exact-match scores exceeding Cohen’s κ by 33.8–41.2 percentage points on MT-Bench, naming the gap kappa deflation.The section describes the gap as a consequence of chance correction that had not been systematically measured at scale across modern judges.Source: §2.1 · p. 3
- High test–retest reliability (α > 0.95) can coexist with severe position bias, a failure mode the paper names the consistency-bias paradox.The section explains that stability measures output reproducibility, not correctness; deterministic preference for position A can yield perfect test–retest reliability and maximum position bias.Source: §2.1 · p. 3
2.2 Attempts to Improve Reliability
- Prior approaches include separating intrinsic consistency from human alignment and developing specialized judge architectures to reduce position bias or cost.The cited architecture studies report promise in reducing position bias and cost while approaching general-purpose model performance.Source: §2.2 · p. 3
3 Methodology
The methodology evaluates 21 judges on three benchmarks under agreement, consistency, and bias-audit protocols. It specifies hypotheses and a multi-phase procedure for conducting the evaluation.
3 Methodology
- The evaluation covers 21 general-purpose judges from nine providers, grouped into three capability tiers and spanning 8B to over 100B parameters.Tier 3 includes April 2026 frontier models and open-source systems.Source: §3 · p. 3
- The benchmarks comprise 2,391 MT-Bench comparisons, 350 JudgeBench items, and 2,981 RewardBench pairs with per-item position randomization.JudgeBench items cover mathematics, coding, creative writing, and analysis; RewardBench places the chosen response equally likely in either position.Source: §3 · p. 4
- The three protocols measure agreement with human labels, repeatability under independent re-evaluation and order swaps, and position and verbosity bias.Agreement reports Cohen’s κ, Krippendorff’s α, and tie-excluded exact match; consistency uses 3–5 evaluations per item and reports test–retest reliability, self-consistency, and position-flip rate.Source: §3 · p. 4
- Seven hypotheses cover expected patterns in deflation, position bias, cross-benchmark rank divergence, benchmark spread, RewardBench labels, flip rates, and consistency–bias.The hypotheses were formulated before frontier-model evaluation.Source: §3 · p. 4
- Across seven phases during March and April 2026, the procedure produced 118 evaluation runs and approximately 541,000 judgments.Evaluations used temperature 0; consistency runs disabled response caching, and position-swap evaluations were used in consistency and bias-audit protocols.Source: §3 · p. 4
4 Results
Across the evaluated judges, chance correction substantially reduces agreement scores, and model performance varies with benchmark and protocol. The results also show that repeatability can coexist with position bias, while measured verbosity bias is small under the study’s rubric.
4.1 Kappa Deflation Is Universal
- 33.8–41.3 pp: every judge shows MT-Bench kappa deflation, with exact match exceeding Cohen’s κ across all 21 models.Mean deflation is 38.6 pp on MT-Bench, 23.7 pp on JudgeBench, and 10.4 pp on RewardBench, tracking differences in label distributions.Source: §4.1 · pp. 4–5, Table 2 · p. 5
4.2 Position Bias Heterogeneity
- Position bias ranges from 0.002 for Gemini 2.5 Pro to 0.192 for Qwen 3 8B, with reasoning and frontier models generally less biased than smaller, cost-optimized models.Within-family variation is substantial: Gemini 2.5 Pro and Gemini 2.5 Flash differ by a factor of 70.Source: §4.2 · p. 5
4.3 Cross-Benchmark Rank Instability
- 11 of 21 judges shift by at least four positions across benchmarks, and only Gemini 3.1 Pro and Claude Opus 4.6 remain top-three on all three.MT-Bench compresses the cohort into a 13.5 pp κ band, compared with 60.4 pp on JudgeBench and 28.1 pp on RewardBench.Source: §4.3 · p. 5
4.4 MT-Bench Ceiling Effect
- JudgeBench’s κ spread is 60.4 pp, 4.5× MT-Bench’s 13.5 pp spread, while MT-Bench’s full-cohort spread is 13.5 pp.Within MT-Bench, the spread narrows to 6.5 pp among the top ten judges.Source: §4.4 · pp. 5–6
4.5 Position-Randomized Evaluation of RewardBench
- All 20 evaluated judges produce RewardBench κ values from 0.616 to 0.898 under per-item position-randomized evaluation.The top seven judges fall within a 2.7 pp band.Source: §4.5 · p. 6
4.6 Consistency Falls on Hard Benchmarks
- Seven of 16 judges have at least a 1.5× increase in position flip rate from MT-Bench to JudgeBench, while Claude Opus 4.6 and Gemini 3.1 Pro improve by 0.58×.Mean test–retest reliability also drops from 0.943 on MT-Bench to 0.911 on JudgeBench.Source: §4.6 · p. 6
4.7 The Consistency–Bias Paradox
- Two judges combine test–retest reliability above 0.95 with position bias above 0.10: Qwen 3 8B and Gemini 2.5 Flash.Qwen 3 8B has test–retest 0.992, position bias 0.192, and JudgeBench κ 0.289; its reproducibility does not establish decision-process correctness.Source: §4.7 · p. 6
4.8 Pairwise Verbosity Bias
- All 21 judges have MT-Bench verbosity bias below 0.011 under the study’s pairwise rubric and fixed length-differential operationalization.The authors caution that this result does not establish low verbosity bias under arbitrary rubric or task variation.Source: §4.8 · p. 6
5 Discussion
The discussion explains why raw agreement and single-benchmark rankings can mislead, then proposes a validation protocol combining chance correction, bias checks, replication, and cross-benchmark evaluation. It also presents the released evaluation data as a resource for further analysis and future comparisons.
5.1 Why Exact Match Misleads
- On balanced ternary MT-Bench labels, exact-match scores of 0.80–0.85 correspond to Cohen’s κ near 0.48, so the authors recommend reporting a chance-corrected metric as the headline reliability number.The cohort-mean ∆κ contracts from 38.6 pp on MT-Bench to 23.7 pp on JudgeBench and 10.2 pp on RewardBench as label distributions change.Source: §5.1 · p. 7
5.2 Limits of Single-Benchmark Validation
- Benchmark ranking instability reflects both differences in discriminative power and benchmarks’ distinct latent constructs, so the authors recommend evaluating on at least two benchmarks spanning preference and correctness.Only Claude Opus 4.6 and Gemini 3.1 Pro remain top-three across all three benchmarks.Source: §5.2 · p. 7
5.3 The Consistency–Bias Paradox and a Minimum Viable Validation Protocol
- Test–retest reliability measures output stability, not decision-process correctness; MVVP reports chance-corrected agreement, measures position bias with paired AB+BA evaluations, replicates across at least three independent runs, and evaluates at least two benchmarks.Runs use temperature 0 with response caching disabled; benchmarks span preference-style and correctness-style label distributions.Source: §5.3 · pp. 7–8
- The protocol treats high stability with high bias as a failure mode, requiring position-bias checks even when test–retest reliability is high.This audit condition is specified for test–retest values above 0.95 and a position-bias threshold below 0.10.Source: §5.3 · p. 8
5.4 Dataset as Community Resource
- The released dataset contains approximately 541,000 judgments across 118 runs, 21 judges, three benchmarks, and three protocols, with raw verdicts, reasoning, responses, latency, and computed metrics.The authors identify meta-analysis, calibration research, and a frozen baseline for future judge evaluations as supported use cases.Source: §5.4 · p. 8
6 Conclusion
Across the evaluated judges, chance-corrected agreement, benchmark rankings, consistency, and bias reveal distinct validation signals. The paper distills these findings into a Minimum Viable Validation Protocol and identifies further research directions.
6 Conclusion
- 33.8–41.3 pp on MT-Bench separate exact match from Cohen’s κ across the evaluated judges.Kappa deflation is reported as universal across the cohort.Source: §6 · p. 8
- Rankings shift up to 14 positions across benchmarks, while two deployed judges combine >0.95 test–retest reliability with >0.10 position bias.Source: §6 · p. 8
- Under a single pairwise rubric, verbosity bias was small (<0.011) across the cohort; the cited finding is cohort-wide and does not establish that every judge individually met the threshold.Source: §6 · p. 8
- The Minimum Viable Validation Protocol recommends chance-corrected agreement, at least two contrasting benchmarks, joint consistency-and-bias measurement, and not foregrounding verbosity bias under standard pairwise rubrics.Source: §6 · p. 8
- Open directions include multilingual and multimodal judging, longitudinal endpoint stability, confidence calibration, and variance-component analysis for sample-size decisions.Source: §6 · p. 8
Limitations
The study’s conclusions are bounded by its benchmark coverage, evaluation window, single rubric, available calibration data, cost estimates, and suppression of reasoning channels.
Limitations
- The evaluation covers three English-language, text-only benchmarks, so its findings should not be extrapolated to multilingual or multimodal judging without further measurement.Source: Limitations · p. 9
- Runs occurred within five weeks in March–April 2026, leaving longer-term stability of hosted-model endpoints unmeasured.Reported values are a snapshot of provider behavior during that window.Source: Limitations · p. 9
- A single pairwise template and fixed metric operationalizations limit what the study establishes about rubric-dependent bias profiles.The verbosity-bias result below 0.011 is not a universal claim that verbosity bias is solved.Source: Limitations · p. 9
- Calibration analysis is deferred because most cohort providers do not expose the token-level logprobs required for Expected Calibration Error and Brier Score.Source: Limitations · p. 9
- Reported per-token costs are list prices and approximate upper bounds, excluding volume tiers, batch discounts, and negotiated enterprise pricing.Source: Limitations · p. 9
- For specified models, reasoning channels were suppressed for comparability and to prevent verdict-token truncation; results do not characterize reasoning-enabled configurations.Source: Limitations · p. 9
A.1 Cross-Benchmark Rank Instability
Figure 2 traces each judge’s rank across MT-Bench, JudgeBench, and RewardBench, making cross-benchmark rank instability visible.
A.1 Cross-Benchmark Rank Instability
- Each trajectory tracks one judge’s rank across MT-Bench, JudgeBench, and RewardBench.Source: §A.1 · p. 12
- A horizontal line indicates a judge holds its position across benchmarks.Source: §A.1 · p. 12
- Compare trajectory slopes: steeper slopes identify judges whose ranks move more as the benchmark changes.Source: §A.1 · p. 12
A.2 Consistency Degrades on Hard Benchmarks
Position flip rates worsen for most judges on JudgeBench relative to MT-Bench, but two frontier judges improve. Cross-benchmark κ rankings also shift substantially for many models.
A.2 Consistency Degrades on Hard Benchmarks
- Most judges’ position flip rates worsen from MT-Bench to JudgeBench, with three showing at least 2.4× degradation.Figure 3 summarizes per-judge changes across the 16-judge consistency cohort.Source: §A.2 · p. 12, Figure 3 · p. 13
- Llama 3.3 70B drops 14 positions from MT-Bench to JudgeBench, while Minimax M2.7 rises 11 positions.Only Claude Opus 4.6 and Gemini 3.1 Pro remain top-three across all three benchmarks.Source: Figure 2 · p. 13
- 0.09→0.17: the cohort median position flip rate rises from MT-Bench to JudgeBench.Claude Opus 4.6 and Gemini 3.1 Pro instead improve by a factor of 0.6×.Source: Figure 3 · p. 13
B Complete Results Tables
The appendix inventories complete agreement, rank, and consistency results, documents the hypothesis predictions and outcomes, and specifies model and benchmark configurations.
B Complete Results Tables
- Tables 4–6 provide per-judge agreement statistics across three benchmarks, cross-benchmark κ ranks, and consistency-protocol measurements.Table 4 includes tie-excluded exact match, Cohen’s κ, Krippendorff’s α, and ∆κ; Table 6 reports test–retest reliability, self-consistency, and position flip rate.Source: B Complete Results Tables · p. 14, Table 4 · p. 15, Table 5 · p. 15
C.1 Formulation context
- The seven predictions were recorded before frontier-model data collection, but were not externally registered and are not presented as formal pre-registration.The stated purpose was to make the calibration of expectations available to readers.Source: §C.1 · p. 17
C.2 Per-hypothesis details
- Table 7 records each a priori hypothesis, its threshold, observed value, outcome, and the body subsection reporting its test.Predictions were made on 2026-04-14, before Phase 6 data collection.Source: §C.2 · p. 17, Table 7 · p. 18
C.3 Calibration assessment
- Three predictions were fully confirmed, one confirmed for a stated subset, two partially confirmed, and one refuted.The refuted prediction expected degenerate RewardBench results; position-randomized evaluation instead yielded κ values of 0.616–0.898.Source: §C.3 · p. 17
D Model Configuration
- The appendix reports per-judge API settings and per-benchmark configuration parameters, with temperature set to 0 for all judges.Table 9 specifies item counts and replicate-run settings, and notes that RewardBench is included only in agreement and bias-audit protocols.Source: D Model Configuration · p. 19, Table 8 · p. 20, Table 9 · p. 20
E RewardBench Loader Configuration
The RewardBench loader’s fixed response ordering would make chance agreement degenerate, so the evaluation randomizes item positions to obtain an agreement signal. The resulting κ values are substantial to near-perfect.
E RewardBench Loader Configuration
- Under the standard loader, chosen responses occupy A and rejected responses B, making every human label A and Cohen’s κ degenerate at 0.000.Per-item position randomization with a fixed seed restores a balanced ordering distribution.Source: E RewardBench Loader Configuration · p. 21
- Per-item position randomization with seed 42 follows the official run_generative.py contract and restores a valid agreement signal.The study applies this configuration because the standard ordering fixes the human label to A for every item.Source: E RewardBench Loader Configuration · p. 21
- 0.616–0.898: Cohen’s κ across the 21 RewardBench judges after the corrected loader is applied.The paper describes this distribution as within substantial-to-near-perfect agreement bands.Source: E RewardBench Loader Configuration · p. 21
F Evaluation Phase Details
The evaluation comprised 118 runs and approximately 541,000 judgments across seven phases. The appendix documents the artifact’s scope, measurement settings, and potential risks.
F Evaluation Phase Details
- 118 runs comprised 63 agreement runs, 21 bias-audit runs, and 34 consistency runs.The campaign spanned seven phases over five weeks in March and April 2026 and produced approximately 541,000 judgments.Source: F Evaluation Phase Details · p. 22, Table 10 · p. 23
G Use of AI Assistance
- AI assistance was limited to editing and manuscript-formatting support, not research claims, experiment code, research design, data collection, or analysis.The authors state that they produced and verified the experimental results and metric computations.Source: G Use of AI Assistance · p. 24
H Responsibility Checklist Notes
- The authors flag temporal scope as a risk because hosted-model endpoints may drift after the March–April 2026 evaluation period.They also caution that reporting κ alone without paired position-swap and consistency checks could give a false sense of reliability validation.Source: H Responsibility Checklist Notes · p. 25
- The released artifact covers English, text-only benchmark data and inherits human-label annotations from the source benchmarks.The documented evaluation spans 21 judges, three benchmarks, three protocols, and approximately 541,000 judgments; multilingual coverage is out of scope.Source: H Responsibility Checklist Notes · p. 25
- Agreement and bias-audit values are reported as single statistics, while consistency metrics summarize 3–5 replicate runs per item.All single-run reporting used temperature 0, and cohort summary rows are unweighted means across judges.Source: H Responsibility Checklist Notes · p. 25
- All judges ran at temperature 0; consistency and bias-audit used position-swap evaluations, while agreement and bias-audit used response caching.The appendix identifies the metric formulations and states that computations are reproducible from the released framework source.Source: H Responsibility Checklist Notes · p. 25