PaperlayerView exact revision on arXiv

Source-linked AI summary

When Can You Debias an LLM Judge? Identifiability Limits, a Test, and Designs for Top-k Ranking

Jian Xu, Delu Zeng, John Paisley, Qibin Zhao

arXiv:2607.02104v2cs.LG

TL;DR

LLM judges are inexpensive pairwise evaluators, but presentation preferences can make win-based rankings reflect more than quality. This paper characterizes the limits of bias-aware Bradley–Terry correction and proposes designs that test or restore the missing information. Comparisons do not identify the quality/bias split, and correction is justified only under a condition on quality–covariate correlation.

  • Problem

    LLM judges can favor presentation features, so pairwise win counts may reflect verbosity and formatting as well as output quality.

    Source: §1 · p. 1

  • Method

    The paper analyzes a Bayesian Bradley–Terry model with quality and bias covariates, characterizes when prior-based correction is justified, and proposes trusted-anchor and paired-rendering designs.

    Source: §1 · pp. 1–2, §3 · p. 2

  • Results

    Across 48 real judge-pools, the profile likelihood over the item-level bias coefficient is flat to 0.0000 nats and remains so when comparisons scale 26×.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Correction is a prior-selected decomposition rather than a bias estimate from comparisons, and its benefit depends on whether quality–covariate correlation stays below a configuration-dependent crossing point.

    Source: §1 · pp. 1–2, §6 · p. 7

    Correction is warranted only when the covariate captures a spurious preference; when length tracks human quality, as in SummEval, debiasing hurts top-k recovery.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise. Because such judges prefer verbose or well-formatted answers, the natural fix is to add bias covariates to a Bradley--Terry model and estimate the bias away. We show this cannot work as advertised: the quality/bias split is \emph{not identified} by pairwise comparisons, and the failure is exact -- across $48$ real judge-pools the profile likelihood over the coefficient is flat to $\mathbf{0.0000}$ \textbf{nats}, and scaling the comparisons $26\times$ buys none. A ``debiased'' score is selected by the prior, not recovered from data. Our contribution is accordingly not a better estimator but a characterization of \emph{when prior-based correction is justified}, plus designs that supply the missing information when it is not. The assumption the prior encodes -- quality is a priori uncorrelated with the covariate -- pays only while $\mathrm{corr}(\theta,x)$ stays below a crossing point (configuration-dependent, $0.22$--$0.60$), which is what makes the same model help on LLMBar and hurt on SummEval and Nectar. We give two escapes: a \textbf{trusted-anchor gate} that decides per (judge, covariate, task) (no false enables in $6{,}000$ decisions at $K\ge10$ anchors, a rate our sample bounds at $\le6\%$), and a \textbf{paired rendering design}. Across fifteen real LLM judges bias is heterogeneous and capability-dependent: correction improves \topk{} recall by $0.20$--$0.32$ on five biased-but-competent cheap judges and is a no-op on frontier ones (Spearman $\rho{=}{-}0.84$ between competence and gain over the $14$ competent judges, $p{<}10^{-3}$), concentrating the benefit where at-scale evaluation happens.

1 Introduction

LLM pairwise judges are cheap but noisy and systematically favor presentation features, so ordinary win counts can rank presentation alongside quality. The paper shows that comparisons alone cannot separate quality from item-level bias, then characterizes when prior-based correction is warranted and proposes ways to supply missing information.

1 Introduction

  • LLM judges are noisy and favor longer, more elaborate, better-formatted answers or particular candidate positions, making win counts reflect presentation as well as quality.The practical target is often the top-k items, and comparisons far from the top-k boundary can waste effort.Source: §1 · p. 1
  • The paper treats bias correction as a prior-based assumption to characterize and test, rather than as a bias estimate recovered from comparisons.Its approach also considers identifiable or supervised designs when the assumption is not justified.Source: §1 · pp. 1–2
  • Across 48 real judge-pools, the profile likelihood over the item-level bias coefficient is flat to 0.0000 nats, unchanged when comparisons scale 26×.The data identify apparent quality and position effects, but not the split between quality and item-level bias; the prior selects that decomposition.Source: §1 · p. 2
  • Correction helps only while corr(θ, x) stays below a configuration-dependent crossing point of 0.22–0.60; beyond it, the benefit reverses.This condition helps explain why correction can help on some benchmarks and hurt on others.Source: §1 · p. 2
  • A trusted-anchor gate tests the prior assumption per judge, covariate, and task, while paired rendering restores identifiability by design.The paired rendering design is the only design-based intervention studied that restores likelihood identification.Source: §1 · p. 2
  • Across fifteen real judges, verbosity bias is capability-dependent: prevalent on cheap judges but absent on frontier judges, so it must be tested per judge.The top-k-aware acquisition heuristic is secondary: it beats budget-agnostic and D-optimal allocation, while its edge over Thompson/LUCB is specific to a frozen judge.Source: §1 · p. 2

2 Related Work

Prior work establishes pairwise ranking and adaptive comparison selection, while LLM-judge bias mitigation has often been heuristic. This paper studies explicit bias modeling for top-k identification and connects its identification limit to established measurement-model results.

2 Related Work

  • LLM-judge research documents position, verbosity, and self-preference biases, while common mitigations include order-swapping, calibration prompts, and majority vote.The paper's focus is that modeling item-level bias does not make its coefficient identifiable from comparisons; position bias is identified through within-pair order variation.Source: §2 · p. 2
  • Bayesian active learning such as BAL-PM selects informative preference pairs for reward-model training, whereas this paper targets top-k identification under an explicitly biased judge.The distinction is the target task and explicit bias model, rather than reward-model data efficiency.Source: §2 · p. 2
  • The identification limit parallels fixed-effects and incidental-parameter problems: item-level covariates are collinear with free item effects, and within-item variation is the standard remedy.The paper applies that remedy through paired rendering and frames its contribution as exposing the implication for LLM-judge debiasing.Source: §2 · p. 2
  • Bradley–Terry/Thurstone models recover quality from pairwise outcomes, while active ranking and dueling-bandit methods adaptively choose comparisons for top-k identification.The paper's acquisition rule uses posterior top-k membership under a judge model with nuisance bias parameters.Source: §2 · p. 2

3 Problem Setup and Model

The model treats pairwise judging as a Bayesian Bradley–Terry process with latent item quality, observable bias covariates, and a position term. Its central qualification is that comparisons identify apparent quality and position, while the prior chooses the split between quality and item-level bias.

3 Problem Setup and Model

  • The setup has N items with scalar quality θ, pairwise verdicts, and covariates; the goal under budget B is to return the k highest-quality items.In controlled benchmarks, the covariate is a binary elaboration indicator; the external-data study instead uses standardized word count.Source: §3 · p. 2
  • The bias-aware Bradley–Terry model predicts a verdict from quality differences, covariate differences weighted by c, and a position term κ.A positive coefficient for a length covariate encodes verbosity bias; κ captures constant preference for the first-shown answer.Source: §3 · p. 2
  • Figure 1 depicts an active loop in which a top-k-aware rule selects a pair, the verdict updates the bias-aware posterior, and membership probabilities guide the next query.The quality/bias split in that posterior is selected by the prior, not identified by comparisons.Source: Figure 1 · p. 3
  • The shrinkage prior on bias coefficients does not let comparisons decide the quality/bias split: they identify apparent quality ϕ = θ + cx and position κ, while the prior determines the decomposition.Within-pair order variation identifies κ; a smaller τ assumes more strongly that covariate-aligned apparent quality is real quality.Source: §3 · p. 3
  • On an unbiased synthetic judge, the bias-aware model reaches 0.79 top-k recall versus 0.86 for the naive model, a 7-point drop.Hierarchical or empirical-Bayes tuning of τ does not resolve identifiability; each tuned value still selects among prior-induced decompositions.Source: §3 · p. 3

4 Analysis

Naive aggregation folds presentation bias into item quality, while a bias-aware Bradley–Terry model cannot identify quality and bias separately from comparisons alone. The prior therefore selects their split, and the paper characterizes its assumption and a paired design that restores identifiability.

Naive aggregation recovers the wrong top-k

  • When a nonzero bias covariate is omitted, naive estimates converge to apparent quality θ_i + c⋆x_i and can misorder items across the top-k boundary.Because this shift is part of the population estimand, more comparisons do not remove the error.Source: §4 · p. 3

The quality/bias split is not identified

  • With a free quality parameter per item and a fixed covariate, comparisons identify only apparent quality ϕ_i = θ_i + c x_i and position, not θ and c separately.The likelihood is invariant when θ_i becomes θ_i + δx_i and c becomes c − δ.Source: §4 · pp. 3–4

What the prior does—and the assumption it encodes

  • The Gaussian prior selects a point on the unidentified ray, encoding the assumption that quality is a priori uncorrelated with the covariate.The correction removes real quality to the extent that quality and the covariate co-vary; its benefit falls with corr(θ, x) and changes sign near 0.45 in the stated simulation setting.Source: §4 · p. 4

Restoring identifiability

  • Rendering the same content at two verbosity levels supplies within-content variation: shared quality cancels in the comparison, giving a direct read on c.This breaks the invariance that prevents comparisons among fixed renderings from identifying the split.Source: §4 · p. 4

5 Experiments

Across controlled and external evaluations, bias-aware correction improves top-k recovery when presentation features are spurious, but can hurt when they track quality. Experiments test the prior assumption, paired-rendering identification, and an anchor-based gate for deciding when to correct.

Do real judges exhibit the modeled biases?

  • GPT-4o-mini prefers elaborated and markdown-formatted answers in 100% of the respective equal-content contrast pairs, but penalizes obvious filler.The reported pattern distinguishes apparent substance from raw length.Source: §5 · p. 4

Q1: bias modeling recovers the correct top-k

  • For five biased judges (Llama, Qwen, Gemini, DeepSeek, GPT-4o-mini), bias-aware aggregation improves top-k recall over naive by +0.20 to +0.32, reaching 0.76–1.0, with gains persisting at full data.Table 1 reports fifteen-judge full-data fits with the binary elaboration covariate (λb=0.1); correction slightly lowers Claude-Haiku recall from 0.93 to 0.87.Source: §5 · pp. 4–5, Table 1 · p. 5

Top-k-aware acquisition and where correction helps

  • The top-k-aware acquisition rule beats Thompson and LUCB on the two judges with enough pools for tests, but its edge is specific to the frozen protocol.Across 14 competent judges, correction gain and naive recall have Spearman ρ=−0.84 (p<10^-3), with greater gains on biased, competent cheap judges.Source: §5 · pp. 5, 7

External validity on LLMBar

  • Neighbor agreement rises from 0.58 to 0.86 after length/format correction on LLMBar, while the unbiased Natural subset is unchanged (0.94 →0.93).The correction uses gold-label supervision, so this tests the mechanism rather than the unsupervised top-k pipeline.Source: §5 · p. 6, Figure 2 · p. 6

Restoring identifiability: a paired rendering design

  • In 24/24 pools, paired rendering yields a curved profile likelihood while the unpaired design loses 0.000 nats when c moves by 0.5 and θ is re-optimized.The paired design ties two renderings of the same content to one quality value.Source: §5 · p. 6

No amount of data substitutes for the design

  • Across 48 unpaired pools, profile-likelihood range over c is 0.0000 nats at every size, while paired-design evidence rises from 10.8 to 172.7 nats as pool size grows.The unpaired comparisons remain silent about the quality/bias split even when comparisons scale 26×.Source: §5 · p. 6

Prior-assumption sweep

  • Correction benefit falls monotonically as corr(θ, x) increases and changes sign near 0.45 in the synthetic sweep.The paper relates this condition to correction helping on LLMBar but hurting on SummEval and Nectar.Source: Figure 3 · p. 7

Gating: when to enable correction

  • At K=5, false enables occurred in 2 of 6,000 decisions; at K ≥10, none occurred across five judges, while 0/60 pools bounds the pool-level false-enable rate at ≤6.0%.The spurious-pool enable rate rose from 22% (K=5) to 59% (K=40). For an N=30 pool at strong-judge prices, 40 anchors (9% of its pairs) cost ≈$0.14, about 10× the cheap judge’s ≈$0.014 single-pool evaluation.Source: §5 · p. 7

6 Discussion and Conclusion

Bias varies across the fifteen judges and is not universal or capability-independent; because coefficients are prior-selected, comparisons support relative rather than absolute readings. The paper frames the practical choices as testing the prior assumption with anchors or using paired designs to make the split estimable.

Discussion and conclusion

  • Across fifteen judges, bias is heterogeneous and position-bias direction can flip; prior-selected coefficients support relative comparisons across shared settings, not absolute readings.The paper names trusted-anchor gating and paired designs as responses when the prior assumption does not hold.Source: §6 · p. 7

Limitations

The correction is warranted only when a covariate captures a presentation preference rather than a genuine quality signal. Its evidence is also bounded by the acquisition protocol and the small samples and specific prompts used for some frontier judges.

Limitations

  • Debiasing can hurt when length legitimately tracks quality: top-k recovery falls from 0.52 to 0.43 on SummEval and from 0.58 to 0.52 on Nectar.The method helps on LLMBar, where length is spurious, but the reported pattern reverses on SummEval and Nectar.Source: Limitations · p. 9
  • The acquisition advantage over Thompson sampling and LUCB disappears with stochastic re-queries, though the rule still beats round-robin and D-optimality.The main protocol caches one verdict per ordered pair; the full-matrix bias-correction results are unaffected by this caveat.Source: Limitations · p. 9
  • Frontier-judge evidence is limited: Gemini-2.5-pro and Opus-4.8 have one oracle each, while V4-pro has two.Perfect recall in these samples means no bias was detected, not that bias is absent.Source: Limitations · p. 9
  • Claims that frontier judges are unbiased apply only to the tested models and prompt.The controlled benchmark also assumes the covariate captures a spurious preference; the assumption fails when length legitimately signals quality.Source: Limitations · p. 9

Deferred method details and experiments

These deferred details define the model’s covariates, inference, and ordered-pair handling, then report evidence on correctness and judge heterogeneity. The methods also depend on collecting both presentation orders to separate position from quality.

Deferred method details and experiments

  • Covariates encode presentation features such as elaboration, length, formatting, or self-preference; each coefficient measures preference logits gained per feature unit.The controlled experiments use a binary elaboration indicator, holding statement count fixed.Source: Deferred method details and experiments · p. 9
  • The model fits a convex penalized Bradley–Terry objective, then uses a Laplace posterior and Monte Carlo top-k membership probabilities for ranking and acquisition.The point ranking is the top-k set of estimated qualities; membership probabilities are sampled from the approximate quality posterior.Source: Deferred method details and experiments · pp. 9–10
  • A full N=100 pool remains interactive, but the cubic covariance factorization would become the computational bottleneck at item counts in the high hundreds.The authors do not claim to have validated N=300.Source: Deferred method details and experiments · pp. 9–10
  • Across five judges, the judge-clustered mean bias-aware gain was +.24 (t-test p=.0005); Llama (+.32, p=.010) and Qwen (+.22, p=.016) were significant individually.The per-judge tests use pools as the independent unit and exact paired permutation tests with Holm correction.Source: Table 4 · p. 10
  • Among 14 competent judges, naive recall and bias-aware gain had Spearman ρ=−0.84; gains concentrate on competent, biased judges, while tested frontier judges already rank perfectly.The coefficient is a prior-selected decomposition, not a likelihood-identified judge constant; its cross-judge comparison uses the same pools, prior, and covariate.Source: Figure 5 · p. 10
  • Position is modeled as an ordered-query effect: collecting both orders of each pair lets disagreement between orders identify κ separately from item quality.The acquisition evaluates the order shown, including the corresponding first-position effect.Source: Deferred method details and experiments · pp. 9–10

A Top-k-Aware Active Acquisition

The acquisition rule targets comparisons that can change top-k membership rather than merely refine the global ranking. It concentrates effort near the top-k boundary, and its reported curves show faster top-k recovery with fewer comparisons.

A Top-k-Aware Active Acquisition

  • The top-k-aware rule reaches high recall with the fewest comparisons; global uncertainty performs worst at small budgets, while naive recall plateaus at a wrong top-k.The figure’s error bars are pool-level standard errors; curve separation is descriptive, with pool-level tests reported separately.Source: Figure 6 · p. 11
  • The score weights expected information about a quality contrast by uncertainty in the two items’ top-k membership, prioritizing decision-relevant pairs.The membership factor is a boundary-targeted surrogate, not an exact expected top-k-entropy reduction.Source: A Top-k-Aware Active Acquisition · pp. 10–11
  • At budget B=200, top-k-aware acquisition makes about 3× more comparisons than round-robin at the boundary and starves items far from it.Round-robin spreads effort uniformly; boundary targeting is the stated mechanism behind the budget savings.Source: Figure 7 · p. 11
  • The algorithm repeatedly fits the model, estimates top-k membership, selects an unqueried ordered pair, observes the verdict, and returns the estimated top-k.The displayed algorithm refits each step for clarity; experiments refit every eight acquisitions.Source: A Top-k-Aware Active Acquisition · p. 11

Acquisition: full evaluation

The acquisition rule leads by point estimate across five biased judges, but its advantage over strong bandit baselines depends on the frozen-judge protocol. The practical comparison also depends on correction’s validity and on workload-specific costs.

Acquisition: full evaluation

  • For GPT-4o-mini, corrected evaluation reaches 0.89 recall for ≈$0.014, while GPT-5.1 reaches 0.93 for ≈$0.28 on this pricing snapshot.The ≈20× cost ratio depends on workload, prompt length, output budget, hidden reasoning tokens, and vendor pricing.Source: Figure 8 · p. 12, Acquisition: full evaluation · p. 13
  • The rule has the highest point estimate against every baseline on all five biased judges, with pool-level significance established only on Llama and Qwen.On the other judges, the comparisons are underpowered and the paper claims no significance.Source: Acquisition: full evaluation · p. 12, Table 5 · p. 12
  • GPT-4o-mini recall is non-monotone in budget: it peaks at 0.89 near B=200, then settles at 0.76 with all 870 comparisons.The paper treats this as protocol-specific and suggests adaptive subset selection may act as implicit regularization under a frozen, mildly misspecified oracle.Source: Acquisition: full evaluation · pp. 12–13
  • Under stochastic judging, boundary targeting beats round-robin and D-optimality at some budgets, but does not significantly separate from Thompson or LUCB.The authors scope the edge over posterior-sampling bandits to the repeat-free setting.Source: Acquisition: full evaluation · p. 13
  • On Llama at B=120, combining bias-aware modeling with top-k-aware acquisition reaches 0.82 recall, versus 0.61 for bias correction alone and 0.51 for acquisition alone.At N=100 and k=10, bias-aware recall is 0.80 versus 0.60 for the naive model; the paper validates N≤100, not hundreds of items.Source: Table 6 · p. 13, Acquisition: full evaluation · p. 13
  • The benefit of correction decreases monotonically with corr(θ, x) and reverses at a configuration-dependent crossing between 0.22 and 0.60.The crossing varies mainly with true bias magnitude and pool size, so the paper advises against using one value as a transferable threshold.Source: Acquisition: full evaluation · p. 14, Table 7 · p. 14
  • Amortizing the 40-anchor gate over 100 reused pools makes the gated cheap judge 18× cheaper than a frontier judge at 0.89 versus 0.93 recall.Reuse assumes corr(θ, x) remains stable within the task family; a meaningful family shift may require re-gating.Source: Acquisition: full evaluation · p. 14, Table 8 · p. 14
  • On a separate Python and algorithms QA domain, bias-aware recall is 0.72±.06 versus 0.46±.03 naive recall across 10 Llama pools.Both controlled domains make verbosity spurious by construction.Source: Acquisition: full evaluation · p. 14

Proofs

The proofs show that naive aggregation can recover apparent quality rather than true quality, while the bias-aware model cannot identify the quality/bias split from pairwise comparisons. A paired rendering design supplies within-item variation, and the acquisition score targets uncertainty at the top-k boundary.

Proofs

  • Naive aggregation fits apparent quality θᵢ+c⋆xᵢ, which need not preserve the true ordering or top-k set when the covariate shifts items across the boundary.Because this concerns the estimand rather than a finite sample, more comparisons cannot remove the error.Source: Proofs · p. 16
  • The covariate design column is a linear combination of item-quality columns, leaving a null direction that makes θ and c non-identifiable from comparisons.Only ϕ=θ+cx, up to a constant, and position term κ are identified; the prior selects a point along the flat direction.Source: Proofs · p. 16
  • Paired renderings assign two covariate values to the same base content and quality, so within-base comparisons depend on c and remove the quality/bias invariance.The experiment maps both renderings of each base to one quality column; the unpaired analysis gives them separate columns.Source: Proofs · p. 16
  • The acquisition score multiplies comparison informativeness and contrast variance by a boundary-uncertainty weight for items whose top-k membership remains uncertain.Removing the boundary weight recovers pure D-optimality, which is indifferent to whether a comparison changes top-k membership.Source: Proofs · pp. 16–17

Expanded related work

The paper places its non-identification result within fixed-effects and measurement-model work, while distinguishing exact collinearity from the Neyman–Scott inconsistency problem. Its contribution is the transfer of that known issue to LLM-judge debiasing and its practical consequences.

Expanded related work

  • The paper distinguishes exact design-matrix collinearity and non-identification from Neyman–Scott incidental-parameter inconsistency (Neyman and Scott 1948; Lancaster 2000).Both involve free unit effects, but here those effects absorb the covariate coefficient entirely.Source: Expanded related work · p. 17
  • Conditioning on or designing for within-unit variation is a standard remedy in conditional logit and Rasch-model work, and the paired rendering design uses that logic.The paper also connects this approach to measurement and psychometric work.Source: Expanded related work · p. 17
  • The paper’s contribution is applying this known impossibility to LLM-judge debiasing and showing its consequences for prior-induced correction, validity, and practitioner-facing designs.It reports an exact flat direction on real judges and supplies a gate and a paired design.Source: Expanded related work · p. 17
Loading 2607.02104v2…