PaperlayerView exact revision on arXiv

Source-linked AI summary

Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data

Florian E. Dorner, Vivian Y. Nastl, Moritz Hardt

arXiv:2410.13341v4cs.LGstat.ML

TL;DR

Expert annotation is costly, and model judges can distort comparisons, motivating debiasing methods for scalable evaluation. The paper studies their limits theoretically and empirically, finding that at the evaluation frontier, debiasing cannot save more than half of the ground-truth labels when the judge is less accurate than the evaluated model.

  • Problem

    Expert annotation is slow and costly, benchmarks struggle to keep pace, and model-judge biases can distort model comparisons and rankings.

    Source: §1 · p. 1, §2.1 · p. 4

  • Method

    The paper analyzes the limits of debiasing model judgments theoretically and supports its results with empirical evaluation.

    Source: §1 · p. 2

  • Results

    When the judge is less accurate than the evaluated model, no debiasing method can save more than half the ground-truth labels.

    Source: §1 · p. 2, §1.1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    At the evaluation frontier, LLM judges may fall short of largely replacing expert labelers, while the required order of magnitude of ground-truth labels remains unchanged.

    Source: §6 · p. 12

    The bound assumes uniform sampling and applies when the judge is less capable than the evaluated model; substantially easier evaluation tasks fall outside that condition.

    Source: §6 · p. 12

Abstract

from arXiv · show

High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem. Scalable evaluation methods that avoid costly annotation have therefore become an important research ambition. Many hope to use strong existing models in lieu of costly labels to provide cheap model evaluations. Unfortunately, this method of using models as judges introduces biases, such as self-preferencing, that can distort model comparisons. An emerging family of debiasing tools promises to fix these issues by using a few high quality labels to debias a large number of model judgments. In this paper, we study how far such debiasing methods, in principle, can go. Our main result shows that when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half. Our result speaks to the severe limitations of the LLM-as-a-judge paradigm at the evaluation frontier where the goal is to assess newly released models that are possibly better than the judge. Through an empirical evaluation, we demonstrate that the sample size savings achievable in practice are even more modest than what our theoretical limit suggests. Along the way, our work provides new observations about debiasing methods for model evaluation, and points out promising avenues for future work.

1 Introduction

The introduction frames model-as-judge evaluation as a response to costly, slow annotation, while noting that judge bias can distort model comparisons. The paper analyzes the limits of debiasing and finds that, at the evaluation frontier, it can save at most half the required ground-truth labels under the stated condition.

1 Introduction

  • Expert annotation is slow and costly, and model-as-judge systems offer a potential way to scale evaluation but can bias comparisons and mislead rankings.Source: §1 · p. 1

1.1 Our contributions

  • When the judge performs worse at its task than the evaluated model, optimal debiasing is no better than using twice the ground-truth data.The main result also holds in a slightly weaker sense when proxy scores are continuous rather than binary.Source: §1 · p. 2, §1.1 · pp. 2–3
  • Figure 1 reports that LLM labels heavily perturb model rankings despite high judge accuracy.Source: Figure 1 · p. 2
  • The best possible sample savings are determined by the Pearson correlation between true and proxy scores, while judge-ground-truth agreement alone does not meaningfully constrain bias or savings.Source: §1.1 · pp. 2–3
  • Sample-size savings above a factor of two are rare in MMLU and MT-Bench experiments, appearing only when state-of-the-art models judge substantially weaker models.Source: §1.1 · p. 3

2 Formal setup and motivation

The formal setup treats model evaluations as binary scores and studies how proxy judgments relate to ground truth. It shows that model-dependent judge bias can distort scores and rankings, while agreement rates alone do not guarantee reliable comparisons.

2 Formal setup and motivation

  • Evaluation is modeled as a binary score for a model output, with expected performance taken over prompts and any randomness in the evaluation protocol.Source: §2 · p. 3
  • The setup covers classification and question-answering accuracy, arena-style comparisons, and safety evaluations, with language models or crowdworkers supplying cheaper proxy scores.Source: §2 · p. 3
  • Because precisely modeling the relationship among inputs, ground truth, and proxy scores is generally infeasible, the analysis treats scoring functions as black boxes and studies their joint distribution.Source: §2 · p. 3
  • For binary scores, the setup specifies their joint distribution using the true score, conditional true-positive rate, and conditional true-negative rate.Source: §2 · pp. 3–4

2.1 Judge bias can strongly perturb model rankings

  • Model-dependent judge bias can misrepresent a model's true score even with large samples, and bias that changes direction across models can perturb their ranking.Source: §2.1 · p. 4
  • Under the stated classifier condition, evaluating strictly better classifiers using the classifier's predictions as labels fully reverses their true ranking.Source: §2.1 · p. 4

2.2 High agreement rate is insufficient for evaluation

  • Agreement can depend on the evaluated model in multiclass classification, and a fixed agreement rate can still permit substantial judge bias in either direction.Source: §2.2 · pp. 4–5
  • Without further assumptions, agreement above 99% for each evaluated model would be required to ensure asymptotically correct rankings when state-of-the-art models differ by low single-digit percentages.Source: §2.2 · p. 5

3 Debiasing LLM judgments

The section develops unbiased debiasing estimators and bounds their sample-efficiency gains, then applies those bounds to frontier models and benchmark experiments. It finds that gains remain limited when the evaluated model outperforms its judge, and are often below the theoretical ceiling in practice.

3.1 Uniformly controlling judge bias ensures correct rankings

  • Uniformly controlling estimator bias across models ensures correct rankings with high probability as estimator variance vanishes.The result requires bias to be sufficiently small for every evaluated model; proxy-only estimates cannot be assumed to meet that condition.Source: §3.1 · p. 5

3.2 Background on Prediction Powered Inference

  • PPI combines many proxy scores with a small paired sample to estimate and subtract judge bias, yielding an unbiased estimate of the true score.When proxy scores greatly outnumber paired samples and correlate strongly with true scores, PPI can have lower variance than the ground-truth sample average.Source: §3.2 · p. 6

3.3 The sample efficiency factor τ

  • The sample-efficiency factor is bounded by score–proxy correlation, and proxy samples offer only constant-factor improvement when correlation stays below one.Increasing proxy samples without also increasing ground-truth samples has limited benefit.Source: §3.3 · pp. 6–7

3.4 PPI is near-optimal for non-adaptive binary evaluation

  • For black-box estimators using n paired ground-truth/proxy samples and N independent proxy samples, PPI is near-optimal in variance.The bound applies to unbiased estimators of the expected score in non-adaptive binary evaluation.Source: §3.4 · p. 7

3.5 Limited gains for evaluating state-of-the-art models

  • When the evaluated model outperforms the judge on similarly difficult tasks, any unbiased estimator has at most a factor-two sample-efficiency gain.This follows from the squared-correlation bound combined with Theorem 5; a substantially easier judging task is an exception.Source: §3.5 · pp. 7–8

3.6 High agreement rate is insufficient for sample efficiency gains

  • High judge agreement alone cannot guarantee sample-efficiency gains; even with zero judge bias, some settings yield no variance advantage over ground-truth estimation.In the proposition’s configuration, estimating judge bias from paired observations is as hard as directly estimating the true score.Source: §3.6 · p. 8

3.7 Experiments

  • On MMLU and MT-Bench, τmax stays below two except when flagship judges evaluate substantially weaker models on MMLU, and is often much smaller.The MMLU exception violates the frontier assumption; the bound is a best-case ceiling, not a guarantee of near-two gains.Source: §3.7 · p. 9, Figure 2 · p. 9, Figure 3 · p. 9

4 Going beyond binary evaluation

Non-binary proxy scores extend the analysis beyond binary evaluation, while the theoretical and empirical results continue to limit sample-efficiency gains at the evaluation frontier. In MMLU experiments, non-binary scores improve efficiency, but the factor-two bound remains relevant when the judge evaluates a stronger model.

Theoretical analysis

  • The analysis focuses on PPI because it is optimal in the special case of binary proxies, and assumes proxy scores are bounded in [0, 1].For non-binary scores, soft agreement is interpreted as the estimated probability of the real score using the recalibrated proxy R(˜s).Source: §4 · pp. 9–10
  • Proposition 9 extends the agreement condition to recalibrated proxies: when binary-proxy agreement is below model accuracy, the recalibrated proxy’s soft accuracy is also below it.For ϵ = 1, the text states this implication directly and uses it to generalize the condition from Theorem 6.Source: §4 · p. 10
  • For a non-constant proxy with ϵ(1 − b) ≤ (1 − SO(R(˜s))), Theorem 10 bounds ρ²(s, ˜s) by 1 − ϵ and correspondingly bounds PPI’s sample efficiency.The passage gives the condition and bound, but the bound on PPI’s sample efficiency is truncated in the supplied text.Source: §4 · p. 10
  • When the judge makes only ϵ times as many errors as the evaluated model, the sample-efficiency bound in the binary case is 2/ϵ.This result is stated as part of the generalization beyond binary proxies.Source: §4 · p. 10

Experiments

  • For MMLU, the non-binary proxy assigns an evaluated answer the judge’s probability for that answer; the experiments report improved sample efficiency over binary scores.The figure presents PPI in the infinite-unlabeled-data limit and uses 90% confidence intervals.Source: §4 · p. 10, Figure 4 · p. 10
  • At the frontier, the sample-efficiency factor remains below two when LLama3.1-405B evaluates stronger Claude 3.5 Sonnet; the same holds for slightly weaker GPT-4 and Gemini 1.5 Pro.Unlike in the binary case, the factor is slightly exceeded for LLaMA3-70B, which is 5% less accurate than LLama3.1-405B, rather than just for much weaker LLama2-7B.Source: §4 · p. 10

5 Related work

Related work applies LLM judges across model evaluation and identifies biases that can distort their scores. The section contrasts correcting named biases with estimating and correcting judge bias from ground-truth labels, and situates PPI’s efficiency limits against earlier and contemporary work.

LLM-as-a-Judge

  • LLM-as-a-Judge is used to evaluate model capabilities and arena benchmarks, and also to assess red-teaming and jailbreaks.High agreement with human judges does not necessarily imply accurate judge scores; this paper gives a theoretical justification and shows the issue can persist after PPI debiasing.Source: §5 · p. 11

Bias in LLM judges

  • Known LLM-judge biases include self-preference, preferences for longer or listed outputs, and choice-order bias; manually enumerating and correcting all biases appears infeasible.Confidence-based abstention may introduce its own bias if abstention correlates with evaluated-model performance.Source: §5 · p. 11

Debiasing methods

  • Earlier PPI-equivalent debiasing with 2018 automated metrics improved data efficiency by around 10%, while the authors show PPI is always optimal in their binary evaluation setting.The earlier method also had optimal worst-case variance.Source: §5 · p. 11
  • For modern LLM metrics, prior PPI applications report ground-truth-label efficiency gains rarely above 50% and always below 100%; the authors’ bounds suggest larger gains are unlikely, especially for state-of-the-art models.Source: §5 · p. 11

6 Discussion

The authors conclude that frontier-model evaluation with LLM judges cannot replace expert labels at a substantially smaller order of magnitude, though doubling effective sample size can still help. They qualify the result by sampling strategy and proxy accuracy, and extend its relevance to biased evaluators beyond LLMs.

6 Discussion

  • At the frontier, doubling effective sample size can help practitioners, but the order of magnitude of required ground-truth labels remains the same with or without LLM judges.Source: §6 · p. 12
  • The results concern uniform sampling; stratified PPI may improve efficiency somewhat, but the theorem still applies per stratum and empirical gains at the frontier rarely exceed a factor of two.The factor-two theorem applies only when the proxy is less capable at predicting the score than the evaluated model; larger gains may be possible when evaluation is substantially easier or specialized-evaluator training data is abundant.Source: §6 · p. 12
  • The results also apply to biased evaluators such as poorly instructed crowdworkers; when the crowdworker majority is less accurate than the evaluated model, their labels are of low value relative to ground truth.Source: §6 · p. 12

A.1 Balanced agreement rate as an alternative

The appendix bounds the gains available from balanced agreement and explores gap estimation, experimental setups, and ranking stability. On MMLU, rankings remain near-perfect even with a 60%-accurate judge making random errors.

A.1 Balanced agreement rate as an alternative

  • Balanced agreement bounds squared correlation; when it is near 0.5, the maximum sample-efficiency factor τmax is close to one.Figure 5 shows τmax between the bounds, and closer to the lower bound than the upper bound.Source: §A.1 · pp. 15–16, Figure 5 · p. 16

A.2 Estimating the gap

  • Jointly optimizing estimators for a model pair can reduce the variance of its estimated performance gap, but pair-specific estimates may break transitivity.Different λ values across comparisons can give one model multiple score estimates.Source: §A.2 · p. 16

B Additional details on experiments

  • Experiments use HELM’s top 10 MMLU models, including an in-house LLama3.1-405b run, and six MT-Bench models, excluding ties and focusing on first answers.For MT-Bench, expert judgments are aggregated by majority vote.Source: B Additional details on experiments · p. 17

B.1 High agreement is not necessary for ranking

  • With a 60%-accurate random-error judge, MMLU model rankings remain near-perfect despite low agreement.For binary classification, the text notes that sufficiently large samples can ensure correct rankings when judge errors are independent of model performance and the error rate is below 50%.Source: §B.1 · p. 17, Figure 6 · p. 17

B.2 Additional results on TruthfulQA

TruthfulQA results show that the best possible sample-efficiency factor stays below two when weaker models judge stronger ones. It also remains below two when the strongest considered judge evaluates the weakest considered model.

B.2 Additional results on TruthfulQA

  • TruthfulQA proxy scores use HELM2 model answers judged by the leaderboard’s top three models, including evaluations of those judges and weaker models.The text describes the answers as responses to question-and-answer prompts.Source: §B.2 · pp. 17–18
  • The evaluated data were accessed on 14 November 2024, and the HELM2 leaderboard appeared not to include recent models such as GPT-4.Source: §B.2 · p. 17
  • On TruthfulQA, τmax stays below two when weaker models judge stronger ones, and even when Palmyra-x judges LLama2-7b.The figure plots τmax by judge and evaluated model, with 90% confidence intervals.Source: §B.2 · p. 18, Figure 7 · p. 18

B.3 Additional results on stratification

Stratified MMLU results show substantial variation in τmax across subtasks, while gains above two remain rare even when stronger models judge GPT-4.

B.3 Additional results on stratification

  • Stratified PPI’s per-stratum guarantee requires assumptions to hold in every stratum; a globally weaker judge may be stronger locally, allowing gains above two there.Source: §B.3 · p. 18
  • Across MMLU subtasks, τmax varies substantially but stays below two when GPT-4 judges stronger Claude or LLama3.1-405b models.Figure 8 reports per-stratum factors for the 57 subtasks.Source: Figure 8 · p. 18, §B.3 · p. 19
  • When Claude or LLama3.1-405b judges GPT-4, gains above two occur in only 6/57 or 1/57 subtasks, respectively.Source: Figure 8 · p. 18, §B.3 · p. 19

C.1 Theorem 5

The proof uses a multivariate Cramér–Rao bound and Fisher information for the sampling experiment. It establishes positive definiteness by decomposing information across paired and proxy-only observations.

C.1 Theorem 5

  • The proof invokes a multivariate Cramér–Rao bound under common-support and finite-information conditions for the likelihood.The cited theorem concerns an experiment with a parameter in a product of open sets.Source: §C.1 · p. 19
  • For independent observations, Fisher information is built from the information in paired ground-truth/proxy samples and proxy-only samples.Source: §C.1 · pp. 19–20
  • The information matrix is decomposed into a positive-definite diagonal component and a positive-semidefinite component, making their sum positive definite.Source: §C.1 · p. 20

C.2 Theorem 6

Theorem 6 bounds the maximum attainable squared correlation at 0.5. Proposition 9 proves a lower bound on recalibrated-proxy soft-accuracy error by splitting into two cases.

C.2 Theorem 6

  • The proof reduces the q optimization to stationary points and the boundary values q = 0 or q = 1.It then optimizes over x and δ, considering feasible candidates where a constraint binds or a derivative is zero.Source: §C.2 · p. 21
  • The maximum attainable ρ² is 0.5.The proof optimizes over q, x, and δ by checking feasible stationary points and constraint boundaries.Source: §C.2 · pp. 21–22

C.3 Proposition 9

  • For recalibrated proxy R(˜s), the proof seeks 1−SO(R(˜s)) ≥ min{1−b, 1−AG}, splitting on whether FN ≤ TN or FN ≥ TN.The two cases compare the soft-accuracy error with 1−AG and 1−b, respectively.Source: §C.3 · p. 23

C.4 Theorem 10

The proof uses Bayes-optimal recalibration to reduce the upper-bound analysis to calibrated scores. It then optimizes the bound using the stipulated soft-accuracy condition.

C.4 Theorem 10

  • Bayes-optimal recalibration cannot reduce squared correlation, so it suffices to prove the upper bound for calibrated scores.The proof uses the properness of the Brier score and the monotonic relationship between squared correlation and MSE.Source: §C.4 · pp. 23–24
  • The proof maximizes the bound by minimizing 1−SO, which the assumption sets to ε(1−b).It substitutes this minimum to obtain the bound.Source: §C.4 · p. 24

C.5 Theorem 11

Theorem 11 identifies endpoint values of q as the maximizers of squared correlation. The accompanying propositions relate a derived quantity to model accuracy, characterize feasible constraints, connect variance reduction to correlation, and give a zero-correlation example.

C.5 Theorem 11

  • For fixed parameters, ρ² is maximized at the minimal or maximal feasible q.At b = 0, the derived value is 2BA−1 for BA > 1/2 and 1−2BA for BA < 1/2.Source: §C.5 · p. 25
  • For a lower bound, the proof inserts the minimizing q without requiring that value to be attainable.The denominator terms then reduce to 1.Source: §C.5 · p. 25

C.6 Proposition 1

  • The derived quantity in Proposition 1 decreases monotonically with E s(mi).The proof rewrites it using the probability that the judged model matches the label and the probability of disagreement when it does not.Source: §C.6 · p. 26

C.7 Proposition 2

  • In Proposition 2, the constraint AG = r is attainable across intermediate values between b and 1 in the first case, and between 1−b and 1 in the second.The first case sets p = 1; the second uses q = 1.Source: §C.7 · p. 26

C.8 Proposition 4

  • The relative variance reduction of ˆθPP versus ˆθGT is determined by squared Pearson correlation ρ, with an upper bound taken as N → ∞.The proof expresses the variance reduction in terms of ρ² for fixed n and N.Source: §C.8 · pp. 26–27

C.9 Proposition 8

  • With q = 1−b and p = b, judge bias and squared correlation are zero, while AG can range from 0.5 to 1.The passage identifies this as a no-judge-bias case.Source: §C.9 · p. 27

Related papers

Loading 2410.13341v4…