Source-linked AI summary
Pair Difficulty Matters: Rethinking Pairwise LLM-as-a-Judge Evaluation and Consistency
Bruno Brocai, Maria Becker
TL;DR
Pairwise reliability proxies may not indicate whether an LLM judge recovers human rankings, because they treat comparisons of different difficulty alike. The paper develops a Bradley–Terry account and tests it in simulation and on human-rated corpora, finding that aggregate proxies are weak predictors of ranking performance. It concludes that evaluation should account for pair difficulty and compare rankings with human rankings where possible.
Problem
It is unclear whether position bias, transitivity, and pairwise agreement reliably indicate whether an LLM judge recovers human-derived rankings.
Source: §1 · p. 1Method
Results
Across CLEAR and ASAP, aggregate bias metrics are weak predictors of ranking performance, and non-transitivity is a weak model-selection signal.
Source: §5 · pp. 3–4Takeaways & Limitations
Evaluate judges by agreement with human rankings and interpret aggregate inconsistency metrics in light of pair difficulty.
Source: §6 · pp. 4–5Both datasets are educational, so results may not generalize to settings such as QA and leaderboards lacking item-level ground truth.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
Large Language Model judges are widely used to rank texts and text-generating systems through pairwise comparison, and their reliability is typically assessed via three proxies: position bias, transitivity, and pairwise agreement (self- or human-labeled). Because these proxies drive judge selection and benchmarking, a substantial literature reporting that judges perform poorly on them risks steering practitioners away from otherwise capable evaluators. We argue this assessment is misleading. Under the Bradley--Terry geometry underlying pairwise aggregation, each proxy is dominated by close-rank-gap pairs, where inconsistency is expected and errors cost little aggregate ranking accuracy; far-gap pairs determine the ranking accuracy but barely move the proxies. We formalize this argument and validate it in a controlled simulation and on two human-rated corpora: the proxies correlate only weakly with ranking accuracy against gold, and their predictive component concentrates in the far-gap regime. Judges should therefore be assessed on rank-gap-conditional metrics, ideally against human rankings. Code at https://github.com/brunobrocai/PairDifficulty
1 Introduction
LLM judges are widely used to evaluate generated text, but their reliability is questioned because they can behave inconsistently across runs, pair positions, and preference cycles. The paper argues that these problems and human disagreement should be understood in light of pair difficulty.
1 Introduction
- LLM judges show run-to-run inconsistency, position bias, and non-transitivity in pairwise preference graphs.Position bias—preferring one comparison slot above chance—is described as the most-discussed failure mode.Source: §1 · p. 1
- Pairwise human–LLM agreement and internal-consistency measures are usually treated as separate problems, but both are affected by rank gap.The authors argue that these proxies can be unreliable indicators of recovery of human-derived rankings.Source: §1 · p. 1
2 Related Work
Prior work documents position bias, non-transitivity, and human–LLM disagreement, while often treating inconsistency as a defect to reduce. This paper instead examines how rank gap shapes what aggregate measures reveal about ranking quality.
2 Related Work
- Position bias affects close-quality pairs far more strongly than distant ones; Xu et al. (2025) show non-transitivity likewise concentrates on close pairs and mitigate it using round-robin tournaments with Bradley–Terry aggregation.Source: §2 · p. 1
- Prior studies find that position bias and non-transitivity concentrate on close pairs, while SAGE reports top models fail on roughly a quarter of difficult cases.Wang et al. (2026) report that 85–90% of transitivity violations are tie-driven; judge capability does not monotonically reduce inconsistency.Source: §2 · p. 1
- Human–LLM disagreement also concentrates on close pairs where human annotators disagree, and lower pairwise agreement can coexist with nearly identical ranking correlations.Zheng et al. (2023) report gap-conditional agreement; Thakur et al. (2025) compare percent agreement with ranking correlation.Source: §2 · p. 2
- This paper decomposes inconsistency by rank gap and argues aggregate measures overweight close-pair behavior that calibrated judges can produce.The authors frame this as an extension of Thakur et al. (2025), who decompose by use case rather than rank gap.Source: §2 · p. 2
3 Theory
The Bradley–Terry model implies that pairwise flips have different significance depending on the strength gap: close-pair flips are expected noise, while far-pair flips reveal calibration errors. Because aggregate metrics weight pairs equally, they can obscure the errors that matter for recovering rankings.
3 Theory
- Under Bradley–Terry, the log-likelihood difference between the two outcomes for a pair is exactly |∆|, making far-pair flips more diagnostic than close-pair flips.Close pairs are near-coinflips; far pairs have lopsided outcome probabilities.Source: §3 · p. 2
- Close-pair self-inconsistency is expected noise that averages across comparisons, whereas far-pair flips expose calibration error and worsen ranking at matched comparison budget.The model assigns near-equal probability to close-pair outcomes but strongly predicts one outcome for far pairs.Source: §3 · p. 2
- Equal-weight metrics such as Krippendorff’s α, raw agreement, and position-flip rates overweight close pairs, where disagreement carries little information about ranking quality.The same |∆|-weighting applies to judge–human disagreement when the human reference behaves according to Bradley–Terry.Source: §3 · p. 2
- The theory predicts weak correlation between pair-level reliability or position-bias scores and ranking accuracy, with predictive signal concentrated in far-pair comparisons.The account also aligns with reported close-pair position bias and human disagreement on close pairs.Source: §3 · p. 2
4 Simulation: Human Agreement
A controlled simulation holds pairwise agreement constant while varying where judges’ errors fall across rank gaps. Ranking correlation with human gold changes substantially with that error distribution.
4 Simulation: Human Agreement
- The simulation generates human comparisons from normally distributed Bradley–Terry strengths and makes each model choose the human winner 60% of the time.Models differ in which rank-gap pairs receive their errors; q ranges from evenly distributed disagreement at 0 to disagreement concentrated on far pairs at 1.Source: §4 · pp. 2–3
- Equal pairwise agreement across simulated judges does not imply equal ranking correlation: gold-ranking agreement falls as errors shift toward larger |∆|.Judges differ only in where across the rank-gap spectrum their errors occur; the simulation sets their pairwise agreement to exactly equal.Source: §4 · p. 3, Figure 1 · p. 3
- Pairwise agreement cannot distinguish judges that trade accuracy on far pairs against accuracy on close pairs, although the far-pair-accurate judge ranks better.The paper contrasts easy-pair performance with tie-breaking close pairs using a misaligned heuristic such as length.Source: §4 · p. 3
5 Experiment: Model Inconsistencies
Experiments on two human-rated text corpora show that aggregate inconsistency measures often do not track ranking accuracy against human gold. Position bias concentrates on close pairs, while non-transitivity largely reflects the same close-pair indecision.
Position Bias
- Six judges were evaluated on CLEAR teacher-rated text and ASAP 2.0 essays, with pairwise judgments repeated after reversing answer order.CLEAR comparisons asked which text was easier for students to understand; ASAP 2.0 essays were rated on a 1–6 scale.Source: §5 · p. 3
- Position bias occurs mostly on close pairs, where flipped verdicts become ties under balanced aggregation and reflect indecision on comparisons that are too close to score.Source: §5 · p. 3
- Across CLEAR and ASAP 2.0, consistency, IPP, and gold-ranking correlation do not reliably track each other, so aggregate bias metrics are weak performance predictors, not evidence that bias is benign.Table 1 reports inconsistency measures alongside Spearman ρ for the LLM-derived Bradley–Terry skill against human gold.Source: §5 · p. 3, Table 1 · p. 4
- Gemma-3-4B is almost entirely indecisive on close pairs, yet its far-pair judgments recover a ranking comparable to much more consistent judges.Its ranking ceiling is naturally capped because it cannot resolve the middle pack.Source: §5 · p. 3
- Mild top-quartile inconsistency in every model points to conceptual misalignment: some items humans rank far apart are treated by the model as close.The authors identify correcting this misalignment as a more promising lever for correlation gains.Source: §5 · p. 4
Non-Transitivity
- Pearson r = .928; non-transitivity is dominated by inequality cases and tracks inconsistency, while Gemma-3-4B’s higher non-transitivity does not mean significantly worse human-gold correlation than GPT-5.4-nano.Within each model family, larger models generally produce fewer non-transitivities, so the metric mainly tracks model size rather than reliably indicating judge quality.Source: §5 · p. 4
Position Bias
- The Table 2 analysis uses 12 judge-corpus points and has limited power to detect moderate effects, despite being well-powered for strong effects.Source: Table 2 · p. 4
6 Conclusion
Human–LLM ranking agreement is the most direct criterion for judge evaluation, while local inconsistency should be interpreted in light of pair difficulty. Benchmark design should expose difficulty and emphasize response spread, with pair-difficulty-aware metrics reported where possible.
6 Conclusion
- Human–LLM ranking agreement is the most direct evaluation criterion; imperfect local consistency should not disqualify a judge.Aggregate metrics that weight pairs equally can reward performance on low-impact comparisons and penalize performance on more informative far-gap comparisons.Source: §6 · pp. 4–5
- Close pairs remain relevant for ranking granularity, and their importance may grow as judges improve or tasks compare closely performing systems.The authors judge the far-pair regime to be where gains are currently largest because judges still struggle on many tasks.Source: §6 · p. 5
- For benchmark items, design for response spread rather than simply making prompts harder, since uniformly strong or weak responses compress rank gaps.The authors connect this to separability reported for benchmarks such as ArenaHard.Source: §6 · p. 5
- Where possible, organize gold-standard items into rankings and report pair-difficulty-aware metrics, but the paper does not recommend a specific metric.Metric choice depends on how human gold pairs were elicited; direct closeness ratings and annotator disagreement as a proxy both require further validation.Source: §6 · p. 5
Limitations
The analysis depends on latent-strength estimates and educational-domain corpora, and it is restricted to pairwise judging against human gold standards. These choices limit how directly the findings extend to other settings and evaluation targets.
Limitations
- The rank-gap analysis requires latent strengths; ASAP 2.0 uses noisier pointwise human ratings as a proxy, making its reported pattern a lower bound.Real ties may be indistinguishable from gaps below the rating resolution, so the paper also reports a reanalysis excluding gold ties.Source: Limitations · p. 5
- Both datasets are educational, so results may not generalize to settings such as QA and leaderboards where item-level ground truth was unavailable.The authors chose these corpora because they provide text-level ground truth needed to measure pair difficulty.Source: Limitations · p. 5
- The theoretical argument addresses binary pairwise judging, not listwise judging, which introduces additional position-bias structure.The authors leave that listwise structure outside the present framework.Source: Limitations · p. 5
- Experiments treat human gold as the target, although human ratings are imperfect and may diverge from the quality construct of interest.For objective gold and well-separated candidates, close pairs may be sparse and the practical importance of the misweighting effect may diminish.Source: Limitations · p. 5
Use of AI Assistants
The authors used AI writing assistance for sentence-level paraphrasing and polishing of author-written text, not to generate research ideas, technical claims, related-work positioning, or citations.
Use of AI Assistants
- AI tools supported sentence-level polishing and analysis-script coding; they did not generate research ideas, technical claims, related-work positioning, or citations.Coding-tool use is documented in the released repository README.Source: Use of AI Assistants · p. 5
A Models
The study evaluates six judges across three model families and two size tiers, using pairwise prompts for readability ease and holistic essay quality. The model choices focus on a variance-limited scale, while the prompt sections adapt human-rating tasks to pairwise judging.
A Models
- The study evaluates six judges across three model families and two size tiers, focusing on 3B–14B models because variance limits the analysis at that scale.Larger models exhibit ceiling effects on the corpora used.Source: A Models · p. 7
- The included OpenAI models lack fully deterministic output, which may inflate inconsistency estimates and prevent exact replication.Open-weight inference was run locally with deterministic greedy decoding and Q8_0 quantization.Source: A Models · p. 7
B.1 CLEAR: readability ease
- The CLEAR pairwise prompt asks which of two texts is easier for students to understand.The prompt casts the judge as a school teacher.Source: §B.1 · p. 7
B.2 ASAP 2.0: holistic essay quality
- The ASAP 2.0 pairwise prompt adapts pointwise criteria to holistic essay comparison, including accurate source-based evidence and overall essay quality.The prompt asks judges to weigh the essay's qualities together rather than focusing on one feature in isolation.Source: §B.2 · pp. 7–8
C Simulation Under Different Settings
Across alternative simulation settings, pairwise agreement alone does not determine ranking recovery: where disagreements occur matters. The results also show that pair difficulty matters less as agreement rises.
C Simulation Under Different Settings
- The baseline simulation assumes a BT-behaving human, normally distributed strengths, .6 LLM–human pairwise agreement, and judges that do not tie.The code also allows other settings, which were run as a robustness check.Source: C Simulation Under Different Settings · p. 8
- Under a deterministic human judge, far-pair disagreements are more damaging to ranking recovery, while close-pair disagreements cost almost no ranking quality.In this setting, every disagreement is an error against gold.Source: C Simulation Under Different Settings · p. 8
- With a uniform strength distribution, fewer close pairs and more far pairs make far-pair performance a larger share of the ranking signal, separating ranking recovery more strongly.Source: C Simulation Under Different Settings · p. 8
- As pairwise agreement rises, simulated judges approach the rank-correlation ceiling, their ranking-correlation differences shrink, and pair difficulty matters less.Source: C Simulation Under Different Settings · p. 8
- With identical overall agreement, simulated judges recover the human-gold ranking differently depending on where disagreements occur.Figure 2 shows this pattern when judges coinflip below a pair-gap threshold; the recovery difference is lower in this variant.Source: Figure 2 · p. 8, C Simulation Under Different Settings · p. 8
D ASAP Without Gold-Tied Pairs
The ASAP 2.0 robustness check excludes gold-tied pairs, whose integer scores have low resolution and may conceal quality differences. The main consistency and ranking-recovery conclusions remain under that exclusion.
D ASAP Without Gold-Tied Pairs
- ASAP 2.0’s integer pointwise scores make gold gaps low-resolution, and a tie may indicate a difference below rating resolution rather than exact equality.The analysis reports results including ties and repeats them with ties excluded as a robustness check.Source: D ASAP Without Gold-Tied Pairs · p. 8
- After removing gold-tied pairs, consistency remains close < far for all six judges, and Table 2’s Spearman(far-consistency, correlation with human gold) remains .897.The revised close bucket is gap = 1; the far bucket remains gap ≥2.Source: D ASAP Without Gold-Tied Pairs · p. 9, Table 4 · p. 9
- After gold-tied pairs are dropped from the BT input, ρ against human gold rises for every judge by +0.01 to +0.06, likely because the removed pairs add noise.Source: D ASAP Without Gold-Tied Pairs · p. 9