PaperlayerView exact revision on arXiv

Source-linked AI summary

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

Gemma Zhang, Prachi Badarayani, Asmi Kumar, Sadid Hasan, Sulaiman Vesal

arXiv:2609.12002v1cs.LGcs.AI

TL;DR

LLM judges can exhibit systematic biases, while much prior work has focused on pairwise comparisons rather than absolute scoring. This paper studies capability-dependent judgment errors and proposes weighted multi-judge calibration from online error-rate estimates. Across benchmarks, more capable examinees receive more lenient judgments, and label-free weighted voting tracks an oracle under simulated temporal drift.

  • Problem

    Prior work has largely examined pairwise LLM-judge biases, leaving absolute scoring—where false positives and false negatives can distort evaluation—less studied.

    Source: §1 · p. 1

  • Method

    The paper proposes calibrated weighted majority voting that weights judges using online false-positive and false-negative estimates derived from inter-judge agreement without ground-truth labels.

    Source: §1 · p. 2, §5 · p. 7

  • Results

    Across four benchmarks, more capable examinee models receive more lenient judgments from all judges, while disagreement-based WMV matches an oracle under simulated temporal drift and outperforms individual judges and unweighted voting.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Capability-dependent leniency can remain even when models are accurate judges, while calibrated cross-judge voting offers a scalable, label-free mitigation.

    Source: §7 · p. 10

    The temporal-drift experiment uses real judge verdicts but simulated temporal ordering; validation on benchmarks with natural temporal structure remains future work.

    Source: §7 · p. 10

Abstract

from arXiv · show

LLMs are increasingly used as automated judges for model training and evaluation, yet individual judges exhibit systematic biases that undermine reliability. Much of prior work has studied biases in pairwise LLM-as-a-judge settings; in this paper, we focus on absolute scoring tasks, which mirror more realistic use cases. Across four benchmarks and six models (36 judge-examinee pairs), we show that a model's task accuracy strongly predicts its judging accuracy (Pearson $r \geq 0.90$ on most models) and inversely predicts its directional bias ($r \leq -0.83$), but that accuracy alone does not ensure fair evaluation: more capable examinee models consistently receive more lenient judgments from all judges ($r \geq 0.83$). To address this, we propose calibrated weighted majority voting (WMV), an ensemble evaluation method that aggregates multiple LLM judges weighted by online estimates of their false-positive and false-negative rates. We introduce a disagreement-based estimator that derives these error rates purely from inter-judge agreement patterns, requiring no ground-truth labels or task metadata. In a simulated experiment with shifting task distributions, our label-free WMV tracks an oracle with perfect error-rate knowledge to within 0.5 percentage points on average, outperforming both individual judges and unweighted majority voting. These results demonstrate that principled multi-judge calibration can simultaneously improve accuracy and correct for systematic leniency without requiring labeled data, offering a scalable path to reliable automated evaluation as model capabilities increase.

1 Introduction

This paper shifts attention from pairwise preferences to absolute judgments, where errors accepting incorrect answers or rejecting correct ones can distort evaluation. It finds capability-dependent leniency and proposes label-free calibrated voting to mitigate it.

1 Introduction

  • Prior work identifies position, verbosity, style, agreeableness, and self-preference biases that can make judgments reflect features unrelated to answer quality.Source: §1 · p. 1
  • Absolute scoring judges one response independently, unlike pairwise scoring, and its key errors are accepting incorrect answers or rejecting correct ones.Such errors can distort model rankings, training signals, and deployment decisions, while remaining invisible without ground-truth labels.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Calibrated weighted majority voting weights judges using online estimates of false-positive and false-negative rates derived from inter-judge agreement without ground-truth labels.Source: §1 · p. 2
  • Stronger examinee models receive systematically more false-positive leniency from all judges, and calibrated weighted majority voting is evaluated as a mitigation.The label-free disagreement-based estimator is reported to match an oracle under temporal drift, outperforming individual judges and unweighted voting.Source: §1 · p. 2

2 Related Work

Prior work documents multiple biases in LLM judging and proposes methods to measure or mitigate them, largely in pairwise settings. This paper distinguishes its focus on absolute scoring and label-free online mitigation.

2 Related Work

  • Prior studies report position, verbosity, style, formatting, self-preference, and agreeableness biases, including preferences linked to perplexity.Source: §2 · p. 2
  • Proposed mitigations include bias-detection frameworks, scores separating self-preference from quality, style perturbations, and reasoning-based correction.Related pairwise work reports that chain-of-thought can reduce harmful self-preference on verifiable tasks.Source: §2 · p. 2
  • This work differs by studying asymmetric error rates in absolute scoring and evaluating online unsupervised mitigation without ground-truth labels.Source: §2 · p. 3

3 Methodology

The methodology evaluates judges on objective benchmarks with known answers, measures directional error rates, and tests six models across judge-examinee combinations.

3.1 Datasets

  • Four objective benchmarks—QuALITY, GSM8K, MBPP, and AIME—cover reading comprehension, math reasoning, and code generation with unambiguous answers.The evaluation uses GSM8K and MBPP test splits, AIME 2025 train data, and QuALITY development data.Source: §3.1 · p. 3

3.2 Metrics

  • The metrics include false-positive leniency, false-negative strictness, and directional bias, defined as leniency minus strictness, alongside judge accuracy.The methodology also compares FPR and FNR for a judge’s own-model responses versus another model’s responses.Source: §3.2 · p. 3

3.3 Models

  • Six models from OpenAI, Meta, and DeepSeek are evaluated across all judge-examinee combinations using each examinee’s full response as judge input.Models are accessed through Azure APIs with near-deterministic decoding where supported; reasoning models use API-enforced t=1.Source: §3.3 · p. 3

4 Experiments and Results

The experiments examine how judge and examinee accuracy relate to judging accuracy and directional bias, and whether more capable examinees receive more lenient judgments. They also motivate practitioner guidance for choosing judges and mitigating bias.

4 Experiments and Results

  • The experiments ask whether stronger models are more accurate and less biased judges, whether judging accuracy implies less bias, and whether accurate examinees receive more leniency.Figure 2 presents QuALITY directional bias for judge–examinee pairs, with self-evaluation on the diagonal and cross-model judging off-diagonal; the paper notes other datasets generally follow the observed trends.Source: §4 · pp. 3–4

4.1 More accurate models are better judges

  • More accurate models generally judge others more accurately and show lower directional bias, although models almost universally exhibit positive directional bias, indicating agreeableness.DeepSeek is an overall stricter judge, and the accuracy–bias relationship is especially strong when its data points are excluded.Source: §4.1 · pp. 4–5

4.2 More accurate judges are generally less biased

  • More accurate judges are generally less biased, but GPT-5 examinees receive high directional bias from most judges, while GPT-5 is comparatively less lenient when judging o4-mini.The authors suggest GPT-5’s stronger task performance may contribute to the leniency it receives.Source: §4.2 · p. 5

4.3 More accurate models are more leniently judged

  • More accurate examinees receive more lenient judgments from LLM judges, including across model families; GPT-5 is the exception to the usual pattern of self-evaluations tracking cross-judge regression lines.The authors attribute GPT-5’s visible self-preference to a distinct bias; their appendix reports bootstrap confidence intervals and a harmful-subset analysis.Source: §4.3 · p. 5, Figure 3 · p. 6

4.4 Summary

  • Practitioners are advised to use the task’s most accurate model as judge, or the judge with the highest accuracy on a golden set to minimize leniency bias.The authors note that increasing capabilities make impartial judging more challenging.Source: §4.4 · pp. 5–6

5 Calibrated Ensemble Evaluation as Mitigation

Cross-judge voting can reduce leniency, but static majority voting can fail when judge reliability varies by task. Calibrated WMV weights votes using online error-rate estimates, and disagreement-based calibration performs nearly as well as an oracle under task drift without labels.

Static majority voting

  • Across three tasks, cross-judge majority voting improved accuracy by 5–14 percentage points and reduced leniency by 15–22 percentage points, but it underperformed self-judging on GSM8K for o4-mini.For o4-mini on GSM8K, majority voting reached 94.4% accuracy versus 97.2% for the self-judge.Source: §5 · p. 6, Table 2 · p. 7

Temporal drift and online calibration

  • Because no judge is uniformly best across tasks and the task mix can shift, static judge selection may be inadequate for changing evaluation workloads.The 12-week simulation shifts linearly from 85% GSM8K and 15% AIME to the reverse, with 40 items sampled each week.Source: §5 · p. 7

Online weighted majority vote

  • WMV weights correct votes by estimated specificity and incorrect votes by estimated sensitivity, down-weighting judges prone to false positives or false negatives.The decision rule aggregates estimated FPR and FNR for each judge.Source: §5 · p. 7

Online calibration estimators

  • The disagreement-based estimator uses other judges’ majority votes as proxy ground truth, deriving proxy false-positive and false-negative counts without ground-truth or task labels.By contrast, the Bayesian estimator uses a small labeled subset, while the stratified estimator requires task labels and the current task mix.Source: §5 · p. 7

Temporal drift and online calibration

  • FPR estimation error stayed below 0.04 during simulated drift, and disagreement-based WMV nearly matched the oracle while outperforming mean single-judge accuracy and unweighted majority voting.The simulation shifted the task mix over 12 weeks; all WMV variants outperformed the mean single judge and unweighted majority vote.Source: §5 · pp. 7–8, Figure 4 · p. 8

6 Ablation and Confounding Studies

These ablations examine whether elite-model leniency reflects explicit identity recognition, answer length, or question difficulty. The findings point to more complex identity-related factors, a task-specific length effect, and persistent self-preference across difficulty strata.

6.1 Test of elite bias through explicit labeling

  • Both elite and non-elite labels reduced false-positive rates relative to blind judging, suggesting that any label prompted stricter evaluation rather than deference to an elite identity.The experiment evaluated identical Llama-3.3-70B responses under unlabeled, GPT-5-labeled, and correctly labeled conditions.Source: §6.1 · p. 8, Table 3 · p. 8

6.2 Length Confound Analysis

  • Length was not a confound on GSM8K, MBPP, or QuALITY, but GPT-4.1-mini showed d = +1.01 on AIME, where longer wrong answers from stronger examinees drove many false positives.The authors interpret the AIME effect as a symptom of capability-dependent bias: answer length can proxy for examinee strength on hard math tasks.Source: §6.2 · p. 9, Table 4 · p. 9

6.3 Difficulty Stratification

  • Question difficulty was defined by the fraction of examinee models answering correctly, with questions divided into hard, medium, and easy bins to compare self-preference gaps.The bins were [0, 0.33), [0.33, 0.67), and [0.67, 1].Source: §6.3 · p. 9
  • Self-preference bias persisted across all difficulty levels; on GSM8K and QuALITY it increased from hard to easy questions, while AIME peaked at medium difficulty and MBPP remained relatively flat.The reported changes were +16.9 to +31.3 on GSM8K, +10.5 to +19.4 on QuALITY, +36.2 at medium difficulty on AIME, and +12–19 across MBPP.Source: §6.3 · pp. 9–10, Table 5 · p. 10

7 Discussion and Conclusion

The study finds that examinee capability shapes judging bias beyond self-preference, while calibrated multi-judge voting offers a label-free mitigation. It also recommends per-judge error reporting and identifies limits to the study’s generalizability and temporal-drift validation.

  • Across four benchmarks, more capable examinees receive more lenient judgments from all judges; self-preference remains distinct, especially for GPT-5, and the bias persists across difficulty strata, response lengths, and identity-label conditions.More accurate models generally make better judges, but overall accuracy can coexist with asymmetric errors across examinees.Source: §7 · p. 10
  • Calibrated WMV down-weights judges prone to false positives, and its disagreement-based estimator matches an oracle with perfect error-rate knowledge under temporal drift.Cross-judge majority voting also improves accuracy and reduces leniency substantially, while the disagreement-based estimator derives a label-free signal from panel agreement.Source: §7 · p. 10
  • Evaluation reports should include per-judge FPR and FNR; when multiple judges are available, the paper recommends calibrated WMV over single-judge evaluation or unweighted majority voting.The authors expect capability-dependent bias to intensify as model capabilities improve.Source: §7 · p. 10
  • Generalizability is limited by coverage of three model families and one prompt template, while temporal ordering was simulated and the disagreement estimator assumes most panel members are usually correct.The authors call for broader model coverage, validation on benchmarks with natural temporal structure, and robustness tests with small or homogeneous panels.Source: §7 · p. 10

A Additional Correlation Results

Tables 6–9 report per-evaluatee or per-judge Pearson correlations for four relationships among model accuracy, judge accuracy, and directional bias.

B Additional Experiments

The additional experiments examine self-preference and harmful self-preference, including cases where GPT-5 accepts its own wrong answers and another judge favors GPT-5’s errors.

  • Table 10 reports self-preference ΔFPR with 95% bootstrap confidence intervals; the difference is statistically significant in most experiments.Positive Δ indicates self-favoring, while negative Δ indicates self-penalizing.Source: B Additional Experiments · p. 13, Table 10 · p. 14
  • On MBPP, GPT-5 marks its only wrong answer correct 100% of the time, while rejecting 98.0% of Llama’s wrong answers.On QuALITY, GPT-5 forgives 85.0% of its own errors when judging against Maverick, but only 3.1% of Maverick’s errors.Source: B Additional Experiments · p. 13
  • Table 9 reports per-judge Pearson correlations between examinee accuracy and received directional bias, with parenthetical values excluding DeepSeek-V3.2 as judge.Source: Table 9 · p. 14
  • GPT-4.1-mini is more lenient on GPT-5’s wrong answers than on its own, hinting at elite-model preference toward stronger models.Source: B Additional Experiments · p. 14

C Experimental Details

The experimental details specify benchmark data, judging inputs, inference settings, and online WMV estimators, including a simulation and prior-configuration evaluation.

  • Inference used managed cloud APIs; temperature was set to 0 where supported, but GPT-5 and o4-mini require temperature 1 and lack an exposed deterministic-decoding mode.Each generation and scoring call ran once, without multi-trial averaging.Source: C Experimental Details · p. 14
  • Generation caps varied by task and model: AIME used 100,000 tokens for o4-mini and 128,000 for GPT-5, while MBPP used 4,096 for reasoning models and 512 otherwise.GSM8K and QuALITY used API-default maximum output lengths.Source: C Experimental Details · p. 14
  • Judges receive the examinee’s full response and task materials, then output a single “correct” or “incorrect” token.Inputs include reasoning traces for math, code with problem and tests for MBPP, and the article, question, answer choice, and justification for QuALITY.Source: C Experimental Details · pp. 14–15
  • The datasets were used without subsampling; listed splits include GSM8K test (1,319), AIME 2025 train (30), MBPP test/evaluation (500), and QuALITY dev (2,202).The QuALITY dev set excludes items missing required fields; test-split labels are not publicly released.Source: C Experimental Details · p. 15, Table 12 · p. 15
  • QuALITY scoring excluded items whose answer generation failed after retries, leaving 1,638–2,287 items per judge–examinee pair; confidence intervals used 1,000 bootstrap resamples at 95%.Source: C Experimental Details · p. 15
  • The 12-week simulation sampled 40 items per week and required at least five judges per item; Bayesian and stratified estimators use labels, while disagreement-based estimation uses inter-judge disagreements alone.Eight labeled items per week updated the Bayesian and stratified estimators; the Bayesian estimator used per-judge FPR and FNR priors, while stratification maintained separate priors by task bin.Source: C Experimental Details · p. 15
  • All 80 tested prior configurations beat unweighted majority vote (90.8–95.0% vs. 87.0%); WMV clips each judge’s positive- and negative-vote weights at a floor of ε = 0.02.The selected prior was pFPR=0.4, pFNR=0.2, strength=10.Source: C Experimental Details · pp. 15–16

Related papers

This paper cites

Loading 2609.12002v1…