When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validity

1 INTRODUCTION

• Ground truth for open-ended judgments is difficult and expensive to obtain, so many judge benchmarks use LLM judges themselves and their scores guide research.The introduction notes their use as a guiding signal for accepted conference submissions.Source: §1 · p. 1
• Across four judges and two settings, about 55% of judgment variance is unexplained; after ELO transformation, a linear model explains 100% of observed variance.The authors interpret this as ELO hiding latent uncertainty in multi-factor analysis.Source: Figure 1 · p. 2
• The paper proposes automated diagnostics for benchmark confounds and calls for tighter objectives, factor audits, uncertainty reporting, and scope constrained to demonstrated validity.These principles are intended to foreground validity rather than the appearance of stable rankings.Source: §1 · pp. 2–3
• The Arena-Hard Auto analysis finds severe rubric incoherence and factor collapse, while ELO-style aggregation masks uncertainty in seemingly stable rankings.The reported examples include over 90% unexplained variance for DeepSeek-R1-32B and factor correlations over 0.93 across criteria.Source: §1 · p. 3

2 METHODS

• The paper focuses on uncertainty introduced by benchmark design—rubric, questions, metric, and comparison models—rather than primarily on statistical or model uncertainty.Even rubrics without independently scored criteria can be implicitly multi-factored because preferred characteristics may comprise distinct dimensions.Source: §2 · p. 3
• Schematic adherence measures how much a judge’s overall verdict can be explained by its factor-wise rubric scores, including through linear or polynomial factor analysis.Low R^2 indicates that verdicts deviate from the stated rubric; the method also examines integration patterns and context stability.Source: §2 · pp. 3–4
• Psychometric validity assesses whether the full benchmark setting produces internally consistent signal that is meaningfully distinct across rubric factors.The setting includes the comparison models, questions, scoring system, and judge; distinctness across factors is termed discriminant validity.Source: §2 · p. 4
• The psychometric index aggregates normalized, thresholded internal-consistency and discriminant-validity components, with an explicit penalty for judge failure rates.The passage defines the failure rate as the fraction of missing or unscorable judgments for a factor.Source: §2 · p. 4

3 EXPERIMENTAL SETUP

• Arena-Hard Auto supplies 500 challenging Chatbot Arena queries and is intended to approximate Arena preferences; the study evaluates four primary judge models.The judges are GPT-4o-mini, GPT-3.5-Turbo, QwQ-32B, and DeepSeek-R1-32B.Source: §3 · p. 4
• The rubric elicits factor-wise and overall scores for Correctness, Completeness, Safety, Conciseness, and Style, with judges instructed not to mix criteria.Judges rate model pairs on each criterion, reflect on criterion weights, and issue an overall verdict; uncertain cases default to ties.Source: §3 · p. 4
• The study computes all six Likert-scaled scores in one forward pass and primarily analyzes raw scores to preserve information and calibrate uncertainty.Computing factors one at a time preserves rank ordering but can significantly change raw numerical scores; ELO conversions follow the original benchmark approach.Source: §3 · pp. 4–5
• A setting is a tuple of judge, rubric, question set, model set, model hyperparameters, and metric; robustness procedures include imputation, Bonferroni correction, and 1,000 bootstrap iterations.The study also reports extraction-failure rates, assigning a default no-preference judgment when extraction fails.Source: §3 · p. 5

4 RESULTS

• Factor-wise rank correlations often exceed 0.93, indicating dimensionality collapse; cross-loadings and weak separation further show that intended constructs are not meaningfully distinct.The pattern appears before ELO and is exaggerated by the transformation.Source: §4 · p. 5, Figure 3 · p. 7
• Completeness, correctness, and style show α ≳0.90 across judges but only moderate discriminant signal, suggesting judges often confuse these otherwise internally consistent factors.The validity summary combines internal consistency and discriminant components, with a penalty for unscorable judgments.Source: §4 · p. 5, Figure 2 · p. 6
• Safety has lower internal consistency for some judges and elevated unscorable responses, while GPT-3.5’s safety failures particularly depress its overall validity score.Conciseness has high validity in part because it is ignored in the overall judgment.Source: §4 · pp. 5–6, Figure 2 · p. 6
• Unexplained judgment variance ranges from 26.2% for GPT-4o-mini to 87.4–90.5% for DeepSeek-R1-32B without reasoning, with reasoning-enabled DeepSeek-R1-32B at 70.8%.Closed-source judges show higher adherence than open-source reasoning judges, and explicit reasoning yields only modest improvements.Source: §4 · pp. 5–6
• Evaluating factors separately rather than jointly changes absolute scores dramatically but preserves model-ranking structure, so the observed failures are not artifacts of simultaneous scoring.Source: §4 · p. 7

4.1 LIMITATIONS OF RANKING SYSTEMS

• ELO-style aggregation has limitations including non-transitivity, order dependence, hyperparameter sensitivity, unrealistic outcome assumptions, and cold-start data needs.The cited discussion notes that reliable estimates can require 100+ attempts.Source: §4.1 · p. 8

4.2 RANKING FAILURES IN ARENA-HARD AUTO

• R² ≈0.998 ranking stability follows ELO/Bradley–Terry aggregation, which masks judgment complexity by converting multidimensional judgments into weighted win/loss outcomes and logistic probabilities.The transformation renders multifaceted judgments essentially unipolar and filters out systematic bias patterns.Source: §4 · p. 6, Figure 4 · p. 7, §4.2 · p. 8

• LLM-as-judge is widely used for instruction following, safety, and general assistance, but deeper analyses report modest human correlations and persistent construct-validity concerns across tasks and criteria.Source: §5 · p. 8
• Psychometric tools such as Cronbach’s α, HTMT, and factor structure can probe reliability and validity, but formal guarantees and benchmark-oriented diagnostics remain limited.Source: §5 · p. 8
• Prior work documents judge bias and brittleness, including preference leakage, concerns about average-preference optimization, reference-quality effects, and shared biases in uncertainty methods.Source: §5 · p. 8
• ELO/Bradley–Terry aggregation is common in community leaderboards, while theory and empirical work point to volatility, interaction dependence, hyperparameter sensitivity, and erased multidimensional uncertainty.Source: §5 · p. 8
• The paper bridges these strands with diagnostics that test whether overall judgments follow stated schemas and quantify residual uncertainty from weak factor structure.The authors position these diagnostics as preconditions for interpreting LLM-judged rankings.Source: §5 · p. 9

• The analysis focuses on Arena-Hard Auto, so generalization to other benchmarks and domains requires replication; judge behavior may drift, and results reflect one schema and template choices.The psychometric index is deliberately simple and alternative formulations may offer complementary insights.Source: §6 · p. 9
• Future work includes task-conditioned validity checks, principled rubric pruning, uncertainty-calibrated judge ensembling, and protocols that optimize adherence and discriminant validity.Source: §6 · p. 9

• Changing comparison models or benchmark questions can trigger psychometric validity failures that make an overall comparison non-actionable.Source: §7 · p. 9
• The Arena-Hard Auto case study reveals severe schema incoherence, factor collapse, and uncertainty suppression after aggregation.Source: §7 · p. 9
• The authors recommend reliability-aware design with tight objectives, factor-structure audits, and transparent uncertainty reporting.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

• The anonymized source-code repository contains everything needed to reproduce the experiments in principle.Data that could break anonymity requirements, including raw responses, model judgments, and extended tables and figures, is to be made public after the review period.Source: REPRODUCIBILITY STATEMENT · p. 9

LLM USE STATEMENT

• Foundation models were used in a limited way for code and LaTeX generation, visualization rendering, writing polish, and related-work retrieval.The acknowledgment says this use followed ICLR policy.Source: LLM USE STATEMENT · p. 9

A SCHEMATIC ADHERENCE COMPLETE RESULTS

• Open-source reasoning models show dramatically higher unexplained variance than closed-source models, while four judges’ factor-loading eigenvalues indicate a collapse of significance.The factor-loading comparison covers four LLM judges in Setting 2.Source: Table 1 · p. 13, Figure 5 · p. 13

B.1 SCHEMATIC ADHERENCE

• Schematic adherence measures how much overall judgments are explained by rubric-factor scores, with nonlinear models and question-wise clusters allowing for interactions and context-dependent patterns.The analysis also examines factor-weight disparity and entropy, and unexplained variance as a signal of coherence failures.Source: §B.1 · pp. 13–14

B.2 PSYCHOMETRIC VALIDITY

• Psychometric validity combines internal consistency and discriminant validity, while down-weighting factors with missing or unscorable judgments.The measures include Cronbach’s alpha, cross-loading ratio, and HTMT; the composite uses a harmonic mean to penalize disagreement between CLR and HTMT signals.Source: §B.2 · pp. 14–16

B.3 RELIABILITY HEATMAPS, FAILURE RATES, AND DISTRIBUTIONS

• Most factors reach Cronbach’s α ≥0.9 across judges, but safety is weaker for some; HTMT values near or above 0.85 indicate factor overlap.Failure-rate plots identify GPT-3.5 on safety as having the highest missing or unscorable judgment rate, while violin plots summarize component and composite-score dispersion.Source: Figure 6 · p. 16, Figure 7 · p. 17, Figure 8 · p. 17

C EXPERIMENTAL EVALUATION SETTINGS

• The empirical evaluation uses eleven settings, each defined by judge-template configuration, a comparison baseline, and respondent models.Settings include simultaneous assessment across criteria and isolated evaluation of individual criteria.Source: C EXPERIMENTAL EVALUATION SETTINGS · p. 18, §C.1 · p. 18, §C.2 · p. 18

C.3 CONFIGURATION DETAILS

• The settings vary configuration details and model sets, with three model-set versions and three baseline configurations used across evaluations.The supplied passages identify the baseline models as GPT-4-0314, gpt-4o-mini-2024-07-18, and gpt-4-turbo-2024-04-09.Source: §C.3 · p. 18, §C.4 · p. 19, §C.5 · p. 19

C.6 JUDGE TEMPLATE

• The judge template elicits factor-wise judgments on correctness, completeness, safety, conciseness, and style before an overall verdict, with instructions to weigh criteria and default to ties when uncertain.The template also tells judges not to mix criteria.Source: §C.6 · p. 20

C.7 ABLATION ON JUDGE TEMPLATE STRUCTURE

• Evaluating factors jointly versus in isolation changes absolute scores dramatically but preserves model rankings.The authors note that joint scoring is more efficient and identify the scoring difference as a direction for future work.Source: §C.7 · p. 20, Figure 10 · p. 21

C.8 ABLATION ON DEVIATION RATE

• Deviation rates are highly sensitive to setting for open-weights judges QwQ-32B and DeepSeek-R1-32B.Table 5 reports score deviation rates by judge and setting.Source: §C.8 · p. 21, Table 5 · p. 21
• Deviation rates are fairly stable across criteria.The reported pattern varies more by setting than by criterion.Source: §C.8 · p. 21
• Table 5 presents score deviation rates by judge and setting, in percent.Source: Table 5 · p. 21