PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Judges Matters: Measuring Family-Conditioned Preference in LLM-as-Judge Panels

David Ababio Awuni, Luke E. K. Achenie, Benjamin Tei Partey, Elvis Gyasi Owusu, Nii-Nai Derrick Sowah

arXiv:2609.17857v1cs.CLcs.AIcs.CYcs.LG

TL;DR

LLM-as-judge evaluation assumes that changing the judge will not systematically change winners once ordinary quality differences are accounted for, but measuring family effects without candidate-quality confounding is difficult. The paper uses a fully crossed panel and a candidate-family-fixed estimator to measure same-family preference. All four tested families show positive lifts, and the paper also identifies likelihood association and position instability as distinct evaluation concerns.

  • Problem

    After ordinary quality differences are accounted for, it is unclear whether changing the judge systematically changes which response wins, and family effects are difficult to measure without candidate-quality confounding.

    Source: §1 · p. 1, §4.1 · p. 4

  • Method

    The paper tests a fully crossed open-weight panel and holds candidate family fixed while comparing same-family judges with other judges.

    Source: §1 · pp. 1–2

  • Results

    All four tested families show positive same-family lifts of 3.4–8.4 percentage points, with global FPS 0.067.

    Source: §11 · p. 8

  • Takeaways & Limitations

    Reliable LLM-as-judge evaluation requires provenance-aware measurement alongside explicit judge diagnostics, including separate attention to position effects.

    Source: §11 · p. 9

    Response quality has no perfect gold standard, so unmeasured differences may remain; the float16 replication uses smaller judges on the confirmatory allocation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Who the judge is can affect an LLM-as-judge result, but measuring that effect without confusing it with candidate quality is difficult. We study four open-weight families (Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5) in a fully crossed pairwise design with 9,312 judgments. A common per-family statistic is strongly confounded with candidate quality and correlates with Bradley-Terry ability at r = 0.95. We derive a corrected estimator that holds the candidate family fixed and compares judges. All four families then show a positive same-family lift (3.4-8.4 percentage points), with global FPS 0.067 (95% CI [0.053, 0.084], permutation p = 0.0002). The effect remains under panel-based quality controls, an independent human-consensus anchor, and a float16 judging replication. Judge-side likelihood is closely related to the effect: adding likelihood advantage reduces the controlled coefficient by 61%, which we treat as descriptive attenuation rather than causal mediation. Position is a separate failure mode. Across the panel, 55.4% of AB/BA pairs reverse, and reversal above 50% is incompatible with a simple independent content-noise model. Relative to a family-balanced reference, panel composition changes 18.5% of pairwise outcomes. A complete reproducibility archive has been prepared for public release.

1 Introduction

The paper addresses whether a judge’s family systematically changes pairwise outcomes after candidate-quality differences are accounted for. It proposes a candidate-family-fixed comparison and finds positive same-family lifts across four tested families, alongside separate evidence on likelihood and position effects.

1 Introduction

  • Family-conditioned preference is extra support for a candidate when the judge comes from the same family, measured in a fully crossed open-weight panel without judges scoring their own generations.The study frames this as a fixed evaluation panel, not a training pipeline with possible preference leakage.Source: §1 · p. 1
  • The row-wise per-family contrast mixes candidate quality with judge–candidate interaction and correlates with Bradley–Terry ability at r = 0.95.Holding candidate family fixed removes its main effect by comparing support from same-family and other judges.Source: §1 · p. 1
  • Corrected comparisons yield positive same-family lifts for all four families, ranging from 0.034 to 0.084, with each individually significant.The effect also remains under panel-based controls, an independent human-consensus anchor, and a float16 judging replication.Source: §1 · p. 2
  • Adding judge-side likelihood advantage reduces the controlled same-family coefficient by 61%, but the authors treat this attenuation as descriptive rather than causal.A smaller significant family effect remains.Source: §1 · p. 2
  • 55.4% of AB/BA pairs choose different winners; reversal above 50% is incompatible with a simple independent symmetric content-noise model.Counterbalanced reconciliation removes pure position-following but also loses genuine family-preference signal on reversed pairs.Source: §1 · p. 2
  • The 6.7 percentage-point global lift exceeds 8 of 9 adjacent length-controlled win-rate gaps in the AlpacaEval 2.0 top-10 snapshot, a scale comparison only.The comparison does not claim the panel reproduces or reranks AlpacaEval.Source: §1 · p. 2
  • Relative to a family-balanced reference, panel composition changes 18.5% of pairwise outcomes.This makes the identity and composition of judges relevant to which candidate wins.Source: §1 · p. 2

2 Related Work

Related work identifies judge provenance, candidate quality, similarity, and panel dependence as factors in evaluation. This paper positions its contribution as correcting a per-family measurement error in fully crossed pairwise panels and distinguishing family preference from position instability.

LLM judges as measurement instruments

Self-preference, family preference, and quality confounding

  • Self-preference studies concern a judge’s own generations, while family-level work and quality adjustment address overlap between judge and candidate provenance (Panickssery et al., 2024; Liu et al., 2024; Wataoka et al., 2024; Chen et al., 2025a; Chen et al., 2025b; Spiliopoulou et al., 2025; Pombal et al., 2026).The paper identifies a per-family error in zero-sum pairwise panels that can be corrected without a gold score for every response.Source: §2 · pp. 2–3

Similarity, familiarity, and likelihood

Panels, correlated errors, and pipeline effects

  • Panel studies discuss correlated errors and diversity, while preference leakage concerns training provenance; this paper instead measures evaluation decisions against a family-balanced reference (Verga et al., 2024; Kohli, 2026; Li et al., 2026; Yang et al., 2026a; Soumik, 2026).Its stated scope is correction of a per-family measurement error, positive corrected lifts across four families, likelihood association, and distinction from position instability.Source: §2 · p. 3

3 Experimental Design

The fully crossed panel makes candidate quality separable from judge–candidate interaction, which confounds the intuitive per-family contrast. Holding candidate family fixed yields positive identified lifts across all four families.

3 Experimental Design

  • The primary panel crosses four open-weight families over every unordered candidate-family pair and both response orders, yielding 9,312 trials and 582 observations per judge–candidate cell.The retained panel includes Llama 3.1, Qwen 2.5, Gemma 2, and Yi 1.5; AB/BA disagreements are reconciled to ties.Source: §3 · p. 3

4 Measuring Per-Family Preference Correctly

  • The global Family Preference Score compares diagonal same-family support with off-diagonal support, while per-family attribution is the identification challenge.Balanced design makes each judge row sum to 2.00 by construction.Source: §4 · p. 4

4.1 Why the natural per-family contrast is confounded

  • A row-wise same-family contrast mixes candidate-family strength with judge–candidate interaction, so strong candidates can look biased and weak ones can hide a lift.In these data the naive statistic correlates with Bradley–Terry ability at r = 0.95.Source: §4.1 · p. 4
  • The corrected contrast holds candidate family fixed and compares support from its same-family judge with support from judges in other families.This comparison isolates the judge–candidate interaction rather than candidate strength.Source: §4.1 · p. 4

4.2 Three independent checks give the same answer

  • All four identified family lifts are positive after candidate strength is removed, unlike naive scores that track Bradley–Terry ability (r = 0.95).The naive-to-identified correlation changes to r = −0.24; Figure 1 visualizes the contrast.Source: §4.1 · p. 4, Figure 1 · p. 5
  • Double-centering, Bradley–Terry residualization, and prompt-level permutation independently support the correction and its quality-dependent null.The prompt-level permutation null is not centered at zero for the naive per-family statistic.Source: §4.2 · p. 4, Figure 1 · p. 5

5 Results: The Corrected Effect Is Positive in All Four Families

The corrected results show elevated same-family support for every family, rather than a single unusually high Qwen cell. The full support matrix displays the pattern directly.

5 Results: The Corrected Effect Is Positive in All Four Families

  • Global FPS is 0.067, and the corrected family-score range is 0.034–0.084, with all four identified family scores positive and individually significant.Mean diagonal support is 0.551 versus 0.483 off the diagonal; Figure 2 shows each diagonal above the other judges’ mean for that candidate family.Source: §5 · p. 5, Figure 2 · p. 5
  • Llama support is 0.542 versus 0.509, Qwen 0.620 versus 0.544, Gemma 0.543 versus 0.459, and Yi 0.497 versus 0.421.Each comparison is same-family judge support versus the mean support from the other three judges.Source: §5 · p. 5, Figure 2 · p. 5
  • The same-family log-odds coefficient remains stable as quality, similarity, length, candidate-side orientation, and family controls are added.Table 2 reports prompt-clustered sandwich standard errors of approximately 0.031–0.036.Source: Table 2 · p. 5

6 The Effect Survives Measured Quality Controls

The same-family coefficient persists under panel-derived quality and representation controls and under an independent human-consensus anchor. The human comparison supports the finding but does not rule out all quality confounding.

6.1 Panel-internal controls

  • The same-family coefficient changes little as panel-internal controls are added, from 0.270 in M1 to 0.301 in M6; style similarity is not predictive (p = 0.91).Controls include Bradley–Terry quality, style similarity, response length, candidate-side orientation, and family effects.Source: §6.1 · pp. 5–6

6.2 Independent human quality anchor

  • On the human-matched subset, the coefficient is 0.370 without the human control and 0.382 with it, so the external quality signal does not attenuate it.The subset contains 992 reconciled judge–pair observations from 248 consensus items joined across all four primary judges.Source: §6.2 · p. 6
  • The human anchor does not establish that all quality confounding is absent: agreement is κ = 0.482, with consensus on 266 of 400 items.Its value is independence from the LLM panel; smaller unmeasured differences could remain.Source: §6.2 · p. 6

7 Likelihood Familiarity as an Explanatory Channel

Same-family responses are more predictable to the judge, and judge-side likelihood advantage is strongly associated with support. Adding it attenuates, but does not eliminate, the controlled same-family coefficient.

7 Likelihood Familiarity as an Explanatory Channel

  • Same-family responses have mean judge-side per-token log-probability −0.69 versus −1.38 for cross-family responses, a 0.69 natural-log-unit difference.The paper describes this as roughly a twofold ratio in geometric-mean token probability.Source: §7 · p. 6
  • Adding judge-side likelihood advantage reduces the same-family coefficient from 0.299 to 0.116, a 61% attenuation, while likelihood advantage predicts support (β = 0.144, p = 7 × 10−6).The residual same-family coefficient remains significant (p = 0.006).Source: §7 · p. 6, Table 3 · p. 6, Figure 3 · p. 6
  • The 61% reduction is descriptive, not a causal mediated share, because nonlinear-model rescaling and common causes can alter nested coefficients.Familiarity may also capture quality a judge genuinely values; judge-side log-probability remains a useful audit signal despite unresolved causal role.Source: §7 · pp. 6–7

8 Position Bias Is a Separate Failure Mode

AB/BA reversals expose position instability that simple independent content noise cannot explain above 50%. Counterbalancing removes pure position-following but also discards family-preference signal on reversed pairs.

8 Position Bias Is a Separate Failure Mode

  • 55.4% of AB/BA pairs choose different winners; Gemma reverses on 78.8%, above the 50% ceiling for simple independent symmetric content noise.The ceiling also holds for mixtures of different noise rates across items.Source: §8 · p. 7
  • Under pure position-following, reversal exceeds 50% whenever first-position preference differs from chance; Gemma’s 78.8% corresponds to q ≈ 0.88.The panel’s raw first-position A-rate of 0.778 selects the q > 0.5 branch; Llama and Qwen fall below the 50% bound, so reversal alone cannot distinguish content noise from milder position bias.Source: §8 · p. 7
  • Counterbalancing removes pure position-following, but reversed pairs become ties and lose any genuine family-preference signal.Reversed pairs contribute exactly zero signed preference to the headline estimator.Source: §8 · p. 7
  • The position-consistent subset yields FPS 0.211, but post-outcome selection changes the trials and judge mix, so this is sensitivity evidence, not a corrected estimate or bound.Qwen contributes many more stable pairs than Gemma.Source: §8 · p. 7
  • A Gemma-only panel disagrees with the family-balanced reference on 58.3% of comparisons, while about 88% of its single-order choices follow the first position.Position-following and family-conditioned preference are systematic but distinct problems.Source: §8 · p. 7

9 Robustness

Robustness checks preserve a positive direction for the family-preference effect across holdout, specification, and judging-stack variations, while its magnitude depends on evaluation choices.

9 Robustness

  • Float16 judging gives global FPS 0.066 (95% CI [0.045, 0.089]), nearly matching the main estimate of 0.067.All four per-family estimates remain positive, though Qwen and Llama are not individually significant in this holdout-sized replication.Source: §9 · p. 7
  • The 62-prompt confirmatory holdout gives FPS 0.062 (95% CI [0.039, 0.086], p < 0.001).No leave-one-out analysis removes the effect; removing Qwen as a judge lowers global FPS from 0.067 to 0.037, the largest single-family change.Source: §9 · pp. 7–8
  • Across 72 specifications, every estimate is positive, ranging from 0.039 to 0.231 with median 0.105; the 0.067 headline lies in the conservative quartile.The multiverse supports effect direction, but its magnitude depends on evaluation choices.Source: §9 · pp. 7–8, Table 4 · p. 8
  • A family-label placebo shuffle centers near zero, with observed FPS above 96.2% of draws (p = 0.038), supplementary to the prompt-level permutation test (p = 0.0002).Removing any other judge leaves FPS at 0.071–0.091; removing candidate families gives 0.063–0.102.Source: §9 · p. 8

10 Consequences for Evaluation Practice

Panel composition changes pairwise decisions, and the paper’s audit protocol recommends checks for family interaction, judge balance, order reversal, likelihood, and composition sensitivity. Its conclusion reports positive same-family support across tested families alongside a distinct position-instability problem.

10 Consequences for Evaluation Practice

  • 18.5% of pairwise outcomes change when a judge sharing a candidate’s family is compared with a family-balanced four-judge reference.Single-family panels diverge by 22.1% for Qwen and 58.3% for Gemma; removing one family from a balanced panel changes 2.2–13.4%.Source: §10 · p. 8
  • The 6.7 pp global lift exceeds 8 of 9 adjacent length-controlled win-rate gaps in the AlpacaEval 2.0 top-10 snapshot, whose median adjacent gap is 1.09 pp.FPS and AlpacaEval win-rate are different quantities, so this is only a scale comparison.Source: §10 · p. 8
  • The minimal audit protocol calls for an interaction-identifying per-family statistic, balanced judge and candidate-family representation, and AB/BA reversal reporting.It also recommends logging judge-side likelihoods and measuring outcome changes against a family-balanced reference.Source: §10 · p. 8

11 Conclusion

  • Holding candidate family fixed yields positive same-family lifts of 3.4–8.4 percentage points and global FPS 0.067; controls, human consensus, and float16 judging leave the effect in place.The usual row-wise contrast is strongly confounded with candidate ability; likelihood reduces the controlled coefficient by 61%, treated as descriptive attenuation rather than causal mediation.Source: §11 · p. 8
  • Position instability is distinct: counterbalancing removes pure position-following but suppresses family-preference signal on reversed pairs.The reproducibility archive includes prompts, inference settings, all 9,312 judgments, per-trial judge-side likelihoods, human annotations, and analysis artifacts.Source: §11 · pp. 8–9

Limitations

The evidence is bounded by the tested model families, prompts, and inference stack; the likelihood analysis and robustness estimates also have specific interpretive limits. The paper additionally notes dual-use risk from evaluator provenance.

Limitations

  • The evidence covers four tested open-weight families, English open-ended prompts, and one main inference stack, not closed-weight judges or verifiable domains such as code and mathematics.The authors note that ground truth can change how apparent self- or family-preference should be interpreted in verifiable tasks.Source: Limitations · p. 9
  • The 61% likelihood attenuation compares nested nonlinear models and is not causal mediation; likelihood uses only the first 400 characters of each response.A stronger mechanism test would vary predictability or familiarity while holding response quality fixed.Source: Limitations · p. 9
  • The position-consistent FPS of 0.211 is selected-subset sensitivity evidence, while imperfect quality measurement leaves unmeasured differences possible.The float16 replication uses smaller judges on the confirmatory allocation; a full-scale, full-precision replication would provide tighter family-level intervals and a stronger quantization test.Source: Limitations · p. 9

Ethical Considerations

  • Evaluator provenance could be used to construct a panel favoring one model family; the authors plan to release measurements to expose that risk and support mitigations.Original annotations, judgments, and analysis artifacts are intended for CC BY 4.0 release, while third-party materials retain their original licenses.Source: Ethical Considerations · p. 9

A Falcon 40B Exclusion and Position Diagnostics

The section reports a float16 replication of global FPS and diagnoses position-order instability. It also documents the Falcon exclusion, human-label quality checks, and regression robustness specifications.

A Falcon 40B Exclusion and Position Diagnostics

  • Falcon 40B was excluded from the Primary-4 headline because its 88.5% judge tie rate and 0.063 mean candidate support indicated response-health or quality failure.Full-5 results remain sensitivity rows in the specification curve.Source: A Falcon 40B Exclusion and Position… · p. 10
  • 55.4% of AB/BA pairs reversed, exceeding the 50% content-noise bound; Gemma (78.8%) and Yi (53.4%) exceeded it, while Llama (47.3%) and Qwen (42.1%) did not.The observed first-position preference implies q ≈ 0.88 for Gemma and q ≈ 0.63 for Yi under the pure first-slot-following model.Source: A Falcon 40B Exclusion and Position… · p. 10

B Human Annotation Protocol

  • Two retained annotators reached κ = 0.482 across 266 consensus items; panel labels matched 57.1% versus 42.1% for the majority-class baseline.Three graduate-level annotators labeled 400 blinded items, and one was excluded before FPS analysis under pre-specified criteria.Source: B Human Annotation Protocol · pp. 10–11

C Regression Specification and GEE Robustness

  • The nested fractional-logit ladder controls for quality, style, length, orientation, and fixed effects; a separate quasi-binomial GEE checks robustness with exchangeable correlation.The GEE uses a quality contrast without the saturated candidate/opponent fixed-effect block to avoid collinearity.Source: C Regression Specification and GEE… · p. 11

D Full-Precision Replication Detail

  • Float16 judging yielded global FPS 0.066 (95% CI [0.045, 0.089]) on 3,168 pre-filter trials.The replication used the original 66-prompt allocation; the later response-health filter retained 62 prompts.Source: D Full-Precision Replication Detail · p. 11
Loading 2609.17857v1…