PaperlayerView exact revision on arXiv

Source-linked AI summary

All Verdicts are Not Equal: Rethinking LLM Judge Reliability

Vineet Kumar, Darshita Rathore, Anindya Moitra

arXiv:2610.12083v1cs.CLcs.AI

TL;DR

LLM-as-a-Judge is widely used, but its reliability under repeated queries and candidate-order swaps remains poorly understood. The paper audits six judges and formalizes trustworthy verdicts as reproducible, order-invariant, and correct. It finds reliability is not captured by accuracy alone, and holistic rubric scoring shifts trustworthiness more than single-format prompting interventions.

  • Problem

    LLM-judge verdicts are often treated as stable stand-ins for ground truth, although repeated queries and candidate-order swaps can change them.

    Source: §1 · p. 1

  • Method

    The paper audits six frontier judges across benchmarks, prompt formats, presentation orders, temperatures, and repeated trials, and defines T as joint reproducibility, order-invariance, and correctness.

    Source: §1 · p. 1, §6 · p. 8

  • Results

    Reliability is not captured by accuracy alone: trustworthy verdicts must be reproducible, order-invariant, and correct, while one configuration had C = 100% but T = 2%.

    Source: §6 · p. 8

  • Takeaways & Limitations

    T is observable from repeated elicitations and requires no ground-truth labels beyond those already provided by existing benchmarks.

    Source: §6 · p. 8

    The audit covers four English-only benchmarks, closed-weight models without internal-state access, and pairwise preference data.

    Source: Limitations · p. 8

Abstract

from arXiv · show

LLM-as-a-Judge is the standard paradigm for NLP evaluation, yet its systemic reliability remains poorly understood despite being widely treated as a deterministic ground truth. We present a comprehensive reliability audit, stresstesting six frontier models across four benchmarks, five prompt formats, two presentation orders, three sampling temperatures, and ten repetitions per condition. Our empirical analysis reveals severe vulnerabilities: verdicts change across identical replications at temperature zero, position-order swaps flip the majority of verdicts on challenging tasks, and the most deterministic judge achieves perfect consistency by trivially repeating incorrect verdicts, agreeing with ground truth only 51% of the time. To formalize these multi-faceted failure modes, we introduce the trustworthy verdict rate (T ), a unified metric capturing the joint probability that an evaluation is reproducible, order-invariant, and accurate. UsingT , we derive a theoretical upper bound on accuracy imposed by position bias and show that reliability is item-specific rather than modellevel. Finally, we demonstrate that shifting from pairwise win-rate to holistic rubric scoring improves trustworthiness more than any single-format prompting intervention, offering an actionable framework for robust NLP evaluation.

1 Introduction

LLM-as-a-Judge is widely used as a stand-in for ground truth, but repeated-query and order-invariance assumptions fail in practice. This paper audits those failures and formalizes reliability beyond accuracy alone.

1 Introduction

  • LLM-judge verdicts are logged, aggregated, and optimized as stand-ins for ground truth across evaluation settings.The paradigm supports reward signals, model comparison, benchmark scoring, and downstream model selection.Source: §1 · p. 1
  • 7–17% of verdicts change across identical replications, and up to 98% flip when candidate order is swapped; one perfectly consistent judge agrees with ground truth only 51% of the time.These results show why single-shot determinism does not establish reliability.Source: §1 · p. 1
  • The audit stress-tests six frontier judges across four benchmarks, five prompt formats, two presentation orders, three temperatures, and ten repetitions per condition.Its cross-cutting design varies one axis at a time to measure its contribution to verdict instability.Source: §1 · p. 1
  • The trustworthy verdict rate jointly measures whether a verdict is reproducible, order-invariant, and correct, and the framework derives an accuracy ceiling imposed by position bias.It also decomposes reliability loss into noise and position-bias components.Source: §1 · p. 1
  • Paradigm choice among direct pairwise, FLASK rubric decomposition, and G-Eval holistic scoring can shift trustworthiness more than any single-format intervention.The paper also finds that accuracy alone, single-order protocols, and single-shot consistency heuristics miss important reliability failures.Source: §1 · p. 2

2 Related Works

Prior work documents LLM judges’ sensitivity to order, adversarial phrasing, ambiguity, and repeated judgments, while much evaluation research reports agreement or accuracy. This paper connects reliability auditing with uncertainty estimation and structured evaluation.

2 Related Works

  • Existing evaluation work reports agreement or accuracy; this paper instead asks whether verdicts remain reliable under repeated sampling, position swaps, and format changes.The cited literature includes human-preference evaluation, RLHF simulation, objective ground-truth benchmarks, and adversarial robustness.Source: §2 · p. 2
  • Prior studies report order and presentation sensitivity, adversarial-phrase effects, and overconfident or inconsistent judgments.This paper formalizes position sensitivity as φ, derives the accuracy ceiling 1 − φ/2, and combines it with stochastic consistency C in T.Source: §2 · p. 2
  • Related work studies abstention and conformal prediction for uncertainty, while format studies report mixed effects from score averaging, chain-of-thought, and checklist rubrics.This paper reports that G-Eval-style holistic multi-criterion scoring improves T more reliably than decomposed rubric voting.Source: §2 · p. 2

3 Formalization

The formalization separates stochastic instability, position bias, and format sensitivity, then defines trustworthy verdicts through their joint reproducibility, order robustness, and correctness. It also derives a position-bias accuracy ceiling and treats trustworthiness as cell-specific and format-dependent.

3.1 Setup

  • The setup studies a judge’s verdict on preference pairs along stochastic instability, position bias, and format sensitivity.The first two axes enter T; format is treated as a design factor and quantities are computed separately for each prompt template.Source: §3.1 · p. 2

3.2 Per-Pair Indicators and Population Rates

  • Position decisiveness is represented by 1 − φ, while accuracy is partitioned between decisive and undecisive pairs, whose baseline correctness is q.These population rates are directly observable from repeated elicitations.Source: §3.2 · p. 3

3.3 Assumptions

  • Under forced-binary, equally weighted preferences, a pair that flips under candidate swap has contradictory verdicts, exactly one matching ground truth; thus q = 1/2.The framework also assumes stochastic reproducibility is uninformative about position decisiveness and that consistent, decisive-pair correctness equals decisive accuracy.Source: §3.3 · p. 3

3.4 A Tight Upper Bound on Observable Accuracy

  • A ≤ 1 − φ/2 is the accuracy ceiling for the forced-binary setup, and the bound is tight when the judge is perfect on every decisive pair.The generalized ceiling is capability-free at fixed position-flip rate; the capability gap is error orthogonal to position bias.Source: §3.4 · p. 3

3.5 The Trustworthy Rate

  • T measures verdicts that are simultaneously reproducible, order-invariant, and correct, factoring into observable noise-survival and capability-adjusted accuracy terms.Judges with identical overall accuracy can differ substantially in T when consistency or position-flip rates differ.Source: §3.5 · p. 3

3.6 Format as a Design Factor

  • The framework notes that its mutual-independence assumption may fail, but reports that T remains a conservative lower bound when that assumption is violated.The text points to Appendix G for the empirical check.Source: §3.6 · p. 3
  • Format sensitivity is summarized by the within-cell dispersion of T across formats while holding judge and dataset fixed.Small dispersion indicates format robustness; large dispersion means trustworthy verdicts depend on prompt choice.Source: §3.6 · p. 4

4 Experimental Design

The audit spans subjective and objective preference benchmarks, six judges from four providers, and multiple prompt formats. It varies presentation order, repetition, and sampling temperature across the evaluation design.

4.1 Datasets

  • Four benchmarks cover standard subjective preferences with MT-Bench and AlpacaFarm, objective ground-truth evaluation with JudgeBench, and adversarial robustness with LLMBar.JudgeBench includes labels derived from MMLU-Pro, Live-CodeBench, and LiveBench across 17 domains.Source: §4.1 · p. 4

4.2 Judges

  • Six judges span four providers: GPT-5.2, GPT-4o, Gemini-2.5-Pro, Gemini-2.0-Flash, Claude-Sonnet-4.6, and Llama-3.3-70B.The set includes frontier closed-weight models, one strong open-weight model, two reasoning-specialized models, and four standard models.Source: §4.2 · p. 4

4.3 Prompt Formats

  • Prompt formats vary direct verdicts, verdict-first JSON/XML, and rationale-first CoT JSON/XML while sharing the same system prompt and user-message template.Each pair appears in AB and BA order; each pair–judge–format–order combination is repeated ten times, with temperatures 0, 0.5, and 1.0, though reasoning models are excluded at nonzero temperatures.Source: §4.3 · p. 4

5 Results & Empirical Findings

Across six judges, five prompt formats, and four benchmarks, the results test how position bias, consistency, temperature, and evaluation paradigm shape trustworthy verdicts. They show that accuracy alone can conceal reliability failures, and that rubric-based scoring can change trustworthiness substantially.

5.1 The Position-Bias Ceiling is Tight

  • Gemini-2.0-Flash’s 98.0% flip rate on JudgeBench caps accuracy at 51.0%, exactly its observed A; reducing position bias, not capability, is needed at the ceiling.Its T is 2.0% despite C = 100%.Source: §5.1 · pp. 4–5, Figure 1 · p. 5

5.2 T Reveals Reliability Differences Hidden by Accuracy

  • On JudgeBench, changing Gemini-2.5-Pro from Direct to Verdict-first JSON raises T from 28.9% to 77.5%, while accuracy rises from 63.5% to 88.5%.The format change also reduces φ from 56.0% to 6.9%.Source: §5.2 · p. 5

5.3 Consistency Alone Does Not Imply Trustworthiness

  • Llama-3.3-70B’s C is at least 98% in every cell, yet its JudgeBench T ranges from 25.4% to 42.5% as φ ranges from 22% to 45%.Because C multiplies the bias-adjusted accuracy term, high repeatability cannot compensate for substantial position bias.Source: §5.3 · p. 6

5.4 What Drives Variation in T?

  • Judge × dataset interaction is the largest source of T variation (η2 = 28.1%), while Gemini-2.5-Pro’s format dispersion on JudgeBench reaches 48.6 points.Rankings reorder across benchmarks, and a single judge-format result may not represent underlying reliability.Source: §5.4 · p. 6
  • Across Ts = 0 to 1.0, mean C drops by 12.1 points while A −φ/2 changes by only +0.3 points across 48 cells, so T’s decline tracks reduced consistency.This supports treating stochastic consistency and bias-adjusted accuracy as distinct reliability factors.Source: Figure 3 · p. 6, §5.4 · p. 7

5.5 Failure Modes and Practical Remedies

  • The four regions of (C, φ) call for different remedies: repeated sampling for noisy-but-fair judges, and order averaging plus ensembling for chaotic judges.Confidently biased judges can appear reliable through high C even as high φ drives T toward zero.Source: §5.5 · p. 7, Figure 4 · p. 7
  • Sonnet-4.6 has the highest mean T, but no judge dominates uniformly because reliability varies with dataset and format pairing.Figure 5’s broad ranges and overlapping whiskers reflect judge-by-format and judge-by-dataset interactions.Source: §5.5 · p. 7, Figure 5 · p. 7

5.6 Rubric-Based Evaluation: FLASK and G-Eval

  • On JudgeBench, G-Eval raises Gemini-2.5-Pro’s T from 28.9% to 77.9% (+49 pp), while FLASK leaves T below Direct for five of six judges.FLASK reduces position flips for some judges but also lowers stochastic consistency.Source: §5.6 · p. 7

6 Conclusion and Recommendations

The conclusion presents a broad audit and a multidimensional account of judge reliability, then recommends reporting and selecting judges using reliability measures alongside accuracy.

6 Conclusion and Recommendations

  • The audit covers six frontier judges, four benchmarks, five prompt formats, two orders, three temperatures, and ten repetitions per condition.It also compares direct, FLASK-decomposed, and G-Eval rubric paradigms on two objective benchmarks.Source: §6 · pp. 7–8
  • At Ts=0, Table 2 compares direct pairwise, FLASK rubric decomposition, and G-Eval scoring using T.The caption identifies the best T per dataset in bold.Source: Table 2 · p. 8
  • The trustworthy verdict rate treats a verdict as trustworthy only when it is reproducible, order-invariant, and correct.The paper states that this measure and its accuracy ceiling can be obtained from repeated elicitations using existing benchmark labels.Source: §6 · p. 8
  • One judge–format–dataset cell had C = 100% but T = 2%, showing that perfect repeated-call consistency can coincide with almost no trustworthy signal.The paper describes this as an empirically tight accuracy ceiling and says reliability is item-specific rather than judge-level.Source: §6 · p. 8
  • Benchmark releases should report (C, φ, A, T) per cell, compare judges using T, and avoid single-order protocols that leave φ unobservable.The paper recommends these practices without requiring new infrastructure.Source: §6 · p. 8

Limitations

The study identifies limits in its benchmark and model scope, available model signals, and evaluation paradigm.

Limitations

  • The audit covers four English-only benchmarks, so its reliability findings may not generalize to low-resource languages.The authors also note minor sampling variance in estimates for low-frequency phenomena such as per-domain flip rates.Source: Limitations · p. 8
  • Closed-weight models without accessible log-probabilities or internal states prevent analysis of finer-grained token-confidence signals.Source: Limitations · p. 8
  • The audit studies pairwise preference data, leaving transfer of its findings to list-wise ranking and multi-model grading for future work.The authors note that pairwise evaluation is currently the most prevalent paradigm.Source: Limitations · p. 8

A Temperature Sweep Analysis

The temperature sweep reports consistency, position sensitivity, accuracy, and trustworthy rate across temperature settings for four tunable judges.

A Temperature Sweep Analysis

  • The sweep includes GPT-4o, Gemini-2.0-Flash, Claude-Sonnet-4.6, and Llama-3.3-70B; GPT-5.2 and Gemini-2.5-Pro are excluded because their public APIs expose no temperature parameter.Source: A Temperature Sweep Analysis · p. 11
  • Verdict-first and CoT results aggregate their JSON and XML variants, reducing the five-format grid to a three-format view.Source: A Temperature Sweep Analysis · p. 11, Table 3 · p. 11
  • Table 3 reports C, φ, A, and T across Ts ∈ {0, 0.5, 1.0}.T is reported as C · (A − φ/2), with all values in percentages.Source: Table 3 · p. 11

B Format Dispersion Analysis

The appendix defines format dispersion as a measure of how prompt-format choices affect trustworthy rates, reports substantial variation across judges and benchmarks, and recommends reporting format ranges.

B Format Dispersion Analysis

  • Format dispersion, ∆Tfmt, measures how much a judge’s trustworthy rate changes across prompt formats while the judge and dataset remain fixed.A low value indicates format robustness; a high value indicates format dependence.Source: B Format Dispersion Analysis · p. 12
  • Table 4 reports ∆Tfmt alongside the worst-case trustworthy rate Tmin, with shading indicating dispersion magnitude.The table reports values in percentage points.Source: Table 4 · p. 12, B Format Dispersion Analysis · p. 12

Findings

  • GPT-5.2 and Llama-3.3-70B have consistently low dispersion, while Gemini-2.5-Pro ranges from T = 28.9% under Direct to 77.5% under VF-JSON on JudgeBench.That JudgeBench range yields a format-attributed dispersion of 48.6 points.Source: Findings · p. 12
  • Average dispersion on JudgeBench is 22.7%, versus approximately 10 points on MT-Bench, AlpacaFarm, and LLMBar.Five of the six largest dispersion values occur on JudgeBench; the authors present task difficulty as one possible explanation, not an established cause.Source: Findings · p. 12
  • A single judge–format–dataset result can misrepresent judge behavior; the authors recommend reporting (Tmin, Tmax) or (Tmin, ∆Tfmt) across a fixed format set.Gemini-2.5-Pro ranks among the strongest on JudgeBench under VF-JSON but among the weakest under Direct prompting.Source: Findings · p. 12

C Variance Decomposition Details

C.1 ANOVA on T

  • Across 120 (judge, format, dataset) cells, the judge × dataset interaction is the dominant source of variance in T.Each cell at Ts = 0 is treated as an observation in a three-factor ANOVA with judge, format, and dataset as factors and their pairwise interactions.Source: Table 5 · p. 13, §C.1 · p. 13

C.2 Four-Facet G-Study on Verdict Direction

The four-facet G-study separates pair, judge, format, and order effects, with repetitions nested within cells. Its variance decomposition finds substantial pair-by-judge and pair-by-order structure, making reliability item-specific.

C.2 Four-Facet G-Study on Verdict Direction

  • The study estimates a fully crossed pair × judge × format × order model, nesting repetitions within cells to separate variance components.The variance components are estimated by Restricted Maximum Likelihood for verdict direction at Ts = 0.Source: §C.2 · p. 13
  • 10–35% of within-pair disagreement comes from pair × judge interactions, while pair × order contributes 4–6% and the order main effect reaches 6.2% on JudgeBench.On JudgeBench, pair × judge interaction exceeds the pair main effect, showing that reliability varies by item rather than only by judge.Source: Table 6 · p. 13, §C.2 · p. 13

D.1 User Message Template (All Formats)

  • All formats reuse a common user-message template with question and response placeholders; BA order swaps the response contents.The format-specific instruction is appended to the shared template.Source: D Prompt Templates · p. 14, §D.1 · p. 14

D.2 Pairwise Evaluation (5 Formats)

  • The five pairwise formats vary verdict-only, verdict-first, and rationale-first outputs across direct, JSON, and XML formats.The rationale-first variants are labeled CoT in the prompt design.Source: §D.2 · p. 14

D.3 GEval Rubric-Based Evaluation

  • GEval uses one API call with a structured rubric and JSON scores plus a verdict; LLMBar prioritizes instruction adherence, while JudgeBench prioritizes factual accuracy and final-answer correctness.The scoring criteria and their ordering differ across the benchmark-specific rubrics.Source: §D.3 · pp. 14–15

D.4 FLASK Multi-Criterion Evaluation

  • FLASK makes four separate API calls, one per criterion, and aggregates the judgments by majority vote.The criterion definitions are evaluated independently in separate calls.Source: §D.4 · p. 15

D.5 Prompt Design Summary

  • Table 7 summarizes the prompt design matrix and defines cost as API calls per pair, judge, and order evaluation.Source: Table 7 · p. 16

E API Cost

F Dataset Details

The authors compare the closed-form trustworthy verdict rate with an assumption-free joint probability across four benchmarks and judges. The resulting gap is typically under 5 percentage points, supporting the closed form as a conservative approximation even when strict independence does not hold.

F Dataset Details

  • The four benchmark sets use pairs fixed at the experiment's start and shared across judges, formats, orders, temperatures, and repetitions.Table 9 summarizes the benchmark datasets.Source: F Dataset Details · p. 17, Table 9 · p. 17
  • MT-Bench uses crowd-sourced preferences, while AlpacaFarm uses 100 instruction-following pairs with balanced labels that make it sensitive to position bias.Source: F Dataset Details · p. 17
  • JudgeBench contains 100 objectively labeled pairs with a verifiably correct response; LLMBar samples 100 pairs across five robustness splits, mostly with adversarial distractors.Source: F Dataset Details · p. 17

G Assumptions underlying T

  • The assumption-free Tjoint probability is compared with Tclosed = C · (A − φ/2) to test the effect of relaxing mutual independence.The stated independence assumption concerns N, P, and R.Source: G Assumptions underlying T · p. 17
  • The gap between Tclosed and Tjoint is typically under 5 pp. across benchmarks and judges, making the closed-form expression a conservative approximation without strict independence.Table 10 reports the comparison across benchmarks and judges.Source: G Assumptions underlying T · p. 17, Table 10 · p. 18

Related papers

Loading 2610.12083v1…