PaperlayerView exact revision on arXiv

Source-linked AI summary

Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels

Guneet Kohli

arXiv:2605.29800v1cs.CL

TL;DR

LLM judge panels are intended to improve evaluation through diverse, independent votes, but evidence about their true informational value is limited. This paper measures panel dependence using effective sample size and a Condorcet null model, finding that nine judges often provide only about two independent votes’ worth of information. The results show that correlated errors, rather than aggregation algorithms, limit panel reliability across the tested conditions.

  • Problem

    LLM judge panels are expected to make evaluations more reliable through diverse, approximately independent votes, but their effective independence and reliability have not been directly quantified.

  • Method

    The paper combines Kish effective sample size with a Condorcet null model to compare observed panel reliability with independent-voting predictions.

  • Results

    Across tasks and conditions, nine judges provide about 2.0–2.5 independent votes’ worth of information, with an 8–22 percentage-point accuracy shortfall and aggregation closing at most 11% of the Condorcet gap.

  • Takeaways & Limitations

    Adding judges cannot substitute for genuinely independent evaluation: the panel matches or underperforms the best individual judge, and larger ensembles offer little benefit.

  • Takeaways & Limitations

    The study covers classification and binary preference tasks, and its bootstrap intervals do not account for uncertainty from selecting a different judge panel.

Abstract

from arXiv · show

LLM-as-a-judge panels aggregate votes from multiple models, with the expectation that diverse models yield more reliable evaluations. We develop a framework to measure the true informational value of such panels and quantify how far their reliability falls short of the independent-voting ideal. Testing a panel of 9 frontier LLMs from 7 model families on three natural language inference datasets (each with 100 human annotations per item), we find that the 9 judges effectively provide only about 2 independent votes' worth of information. Roughly three-quarters of the panel's nominal independence is lost because the models make the same mistakes on the same items. The consequences are stark: the panel's actual accuracy falls 8-22 percentage points short of what independent voting would achieve, and the best single judge matches or outperforms the full panel across all conditions. Neither adding more judges nor using smarter aggregation algorithms helps -- established methods close at most 11% of this gap, even with access to the correct answers. We quantify these findings using the Kish effective sample size (n_eff) and a Condorcet null model, and show the deficit is robust across prompt variants, temperatures, chain-of-thought reasoning, and a pairwise preference task (RewardBench). The bottleneck is correlated judges, not the aggregation algorithm, implying that scaling up panels cannot substitute for genuinely independent evaluation.

1 Introduction

This paper tests whether multi-model LLM judge panels deliver the independent-voting reliability that motivates ensemble evaluation. Across tasks and conditions, correlated errors reduce a nine-judge panel to roughly two independent votes, while aggregation provides little recovery.

  • Motivation: A 9-judge panel spanning 7 model families was evaluated on MNLI, SNLI, and AlphaNLI, each with 100 human annotations per item.The study targets whether cross-model diversity yields independent assessment.
  • Core finding: The panel barely improved on the best judge on MNLI and was worse on SNLI and AlphaNLI.MNLI: 72.0% versus 71.8%; SNLI: 77.7% versus 84.2%; AlphaNLI: 88.7% versus 91.2%.
  • Core finding: neff ≈2.0–2.5 shows that nine judges contain roughly two independent votes’ worth of information.The Condorcet gap is 8–22 percentage points, with permutation p < 10^-4.
  • Robustness: The independence deficit remains stable across three NLI datasets, prompt variants, temperatures, and RewardBench, despite accuracy ranging from 69% to 93%.Adding judges beyond 5 yields negligible benefit, with effective independence asymptoting at roughly 2.3–3.1.
  • Implication: Established aggregation methods close at most 11% of the Condorcet gap, even with oracle access to gold labels.The reported bottleneck is correlated inputs rather than the aggregation algorithm.
  • Implication: Paying for nine opinions while receiving roughly two opinions’ worth of information makes larger ensembles inefficient.The paper argues for diversifying how models reason rather than simply increasing panel size.

2 Related Work

Prior work motivates LLM judge panels through diversity and Condorcet-style independent voting, while documenting systematic biases and correlated errors. This paper positions its contribution as a direct measurement of inter-judge dependence and its effect on reliability.

  • LLM-as-a-judge: LLM-as-a-judge research established scalable evaluation while subsequent studies identified systematic biases and measurement-theory concerns.This work shifts attention from individual biases to structural dependence among judges.
  • LLM judge panels: PoLL found diverse panels of smaller models outperform the average individual judge, whereas this paper compares panels with their best individual judge.The papers address different comparison baselines: average judge versus best judge.
  • LLM judge panels: The Trust-or-Escalate framework uses single-model confidence for human-review escalation, while this work studies cross-model disagreement and its unreliability.The two approaches therefore differ in both signal and objective.
  • Correlated errors and scaling limits: Studies of 350+ LLMs report agreement on wrong answers 60% of the time on some benchmarks, supporting concern about correlated errors.The artificial-hivemind literature likewise reports homogeneous outputs and poor calibration on items with human disagreement.
  • Condorcet Jury Theorem and ensembles: Condorcet theory requires independence for majority voting to improve reliably, so correlated voters can undermine ensemble gains.This theoretical background frames dependence as a central condition rather than a minor implementation detail.
  • Statistically principled aggregation: CARE and crowdsourcing methods model annotator or inter-judge errors to improve aggregation, whereas this paper emphasizes diagnosing the underlying independence deficit.The related work distinguishes better aggregation from measuring whether the votes contain independent information.
  • Human label variation and ChaosNLI: ChaosNLI supplies 100 human labels per item, enabling analysis of disagreement patterns beyond single-label ground truth.The paper extends prior work on LLMs’ inability to capture the distributional properties of human disagreement.

3 Methodology

The study evaluates a nine-judge panel using human-annotated ChaosNLI data and complementary measures of effective independence and independent-voting accuracy. It combines correlation-based diagnostics, item-aware simulations, and statistical tests to quantify panel reliability and the Condorcet gap.

  • Datasets and panel: The evaluation uses ChaosNLI-MNLI, ChaosNLI-SNLI, and ChaosNLI-AlphaNLI, with 100 annotator labels per item and human-majority labels as the gold standard.The panel contains 9 judges from 7 model families; MNLI, SNLI, and AlphaNLI provide same-task and cross-task checks.
  • Datasets and panel: The panel comprises 9 judges from 7 model families evaluated against 100-annotator majority labels, with rare parse failures and ties handled deterministically.Parse failures occur in fewer than 0.1% of judgments, while three-class task ties occur in 0.4–1.1% of items.
  • Effective sample size: Kish effective sample size estimates how many independent votes the panel contains from pairwise phi correlations among judges’ binary error vectors.The method assumes approximately equal pairwise correlations; an eigenvalue estimate provides a robustness check without that exchangeability assumption.
  • Effective sample size: The eigenvalue method estimates effective independence as k/λmax, yielding k under independence and 1 under perfect correlation.Here, λmax is the largest eigenvalue of the judges’ phi correlation matrix.
  • Condorcet null model: The Condorcet null model predicts majority-vote accuracy under conditional independence using judge-specific, human-entropy-bin confusion matrices.Monte Carlo simulations sample independent votes conditional on each item’s gold label and difficulty, preserving class-specific error patterns and shared difficulty.
  • Statistical tests: The Condorcet gap is the weighted difference between independent-prediction and actual majority-vote accuracy, with uncertainty assessed by bootstrap resampling.A stratified permutation test breaks inter-judge correlations while preserving judge error rates and human-entropy structure; per-bin binomial tests are exploratory.

4 Results

Across datasets and robustness conditions, nine LLM judges provide only about two independent votes, with correlated errors producing large Condorcet gaps and severe scaling limits.

  • Effective independence: neff = 2.18 (95% CI: [2.07, 2.31]) for the nine-judge panel, with an independence ratio of 24.2%.The eigenvalue estimate neigen_eff = 2.16 closely matches the Kish estimate.
  • Error dependence: 290 items had all nine judges correct and 51 had all nine wrong, far exceeding the fewer-than-one all-wrong item expected under independence.These extremes are the signature of correlated errors.
  • Condorcet gap: 72.0% majority-vote accuracy fell short of the approximately 94% item-aware Condorcet prediction by 22.0 percentage points.Only 6.8% of the gap was attributable to shared item difficulty; 66–87% remained unexplained across datasets.
  • Scaling: The first five judges supplied 90% of achievable independence, while judges six through nine added only +0.22 effective votes.The Kish curve has a hard asymptote at 1/¯ϕ ≈2.6.
  • Cross-dataset replication: Across three NLI datasets, neff remained approximately 2.2–2.5 despite varying task types, label sets, and base accuracy levels.The best individual judge matched or outperformed the panel in every dataset, with panel differences ranging from +0.2pp to −6.5pp.
  • Robustness: neff stayed between 1.94 and 2.18 across prompt variants, label ordering, temperature, chain-of-thought, and RewardBench.Chain-of-thought increased correlation, while RewardBench produced neff = 1.99 [1.83, 2.20].

5 Analysis and Discussion

The analysis attributes the panel’s weak performance to systemic, cross-family error correlation rather than any single judge or aggregation algorithm.

  • Systemic herding: Removing judges changed neff by only −0.13 to +0.02, showing that herding was systemic rather than driven by one model.Removing DeepSeek-V3 or Mistral Large 3 increased neff, while removing Llama 4 Scout decreased it most.
  • Adding judges can hurt: Removing Gemini 2.5 Pro increased accuracy by 1.3pp, and 6 of 9 judge removals improved accuracy.The three removals that hurt included the two most individually accurate judges.
  • Sources of correlation: Even on easy items, neff was 2.67 rather than 9, indicating that herding was not merely a response to item ambiguity.Across NLI classes, neff ranged from 1.85 for contradiction to 2.40 for neutral.
  • Cross-family correlation: The three highest-correlated judge pairs were cross-family, while same-family correlation exceeded the cross-family mean by only 0.047.This weakens explanations based solely on model-family membership.
  • Aggregation: Established aggregation methods closed at most 11% of the Condorcet gap, even with oracle access to gold labels.The best individual judge outperformed every aggregation method on SNLI, AlphaNLI, and RewardBench.

6 Conclusion

The paper finds that LLM judge panels contain far less independent information than their nominal size suggests, and that standard aggregation cannot remove this deficit. Practical improvement therefore requires genuinely different reasoning, not merely more judges or model brands.

  • neff ≈ 2.0–2.5 remains stable across three NLI datasets, prompt variants, temperatures, and RewardBench, while panels match or underperform the best individual judge.The deficit is structural across the tested conditions rather than tied to one experimental choice.
  • Established stable aggregation methods close at most 11% of the Condorcet gap, while unstable correlation-aware weighting reaches 21% on one dataset but hurts on others.The results identify correlated inputs, rather than the aggregation algorithm, as the bottleneck.
  • A 5-judge panel already captures 90% of achievable independence, making additional judges a substantial inefficiency when nine opinions provide only about two independent votes.The paper recommends computing neff as a panel diagnostic and treating panels cautiously when neff/k < 0.5.
  • Halving mean error correlation from 0.39 to 0.20 would raise neff from 2.2 to 3.5 and close roughly half the Condorcet gap, although achieving this remains open.Potential routes include architectural diversity, specialist fine-tuning, and hybrid human-LLM panels.

Limitations

The study’s evidence is broad across tested classification and preference tasks but remains bounded by task type, benchmark-ground-truth assumptions, model vintage, prompting choices, and panel selection.

  • All four evaluated settings are classification or binary preference tasks, so inter-judge correlation may differ for open-ended generation evaluation or code review.Those settings have fundamentally different output structures.
  • The 100-annotator majority label may represent a plurality preference rather than a correct answer on high-entropy items.Distributional alignment metrics reportedly reproduce the same pattern without reducing labels to binary accuracy.
  • The findings are a snapshot of current frontier models, so future models may exhibit lower correlation even though the neff and Condorcet-gap framework remains applicable.
  • Prompt and decoding choices remain a scope boundary: tested neff stayed between 1.94 and 2.18, but few-shot or persona-based prompts could alter correlation.Chain-of-thought increased correlation, producing neff = 1.94.
  • Bootstrap confidence intervals quantify uncertainty for these judges and items but do not capture uncertainty from selecting a different panel.

Ethics Statement

The study uses public benchmark annotations and commercial LLMs without recruiting human subjects, while documenting its evaluation prompts, reproducibility choices, and analysis figures. The supplied ethics materials primarily describe study provenance and methodology rather than additional human-subject risks.

  • The study uses publicly available ChaosNLI data and commercial LLMs, and recruits no human subjects.ChaosNLI annotations were collected and released publicly by Nie et al. (2020).
  • NLI Classification Prompt: NLI judges receive a canonical prompt asking for exactly one label: entailment, neutral, or contradiction.The canonical vocabulary is used to avoid variance from non-standard label phrasing.
  • Reproducibility: Parse failures are assigned deterministically to one of three labels by hash, preserving reproducibility while avoiding systematic assignment bias.
  • RewardBench Pairwise Preference Prompt: RewardBench judges use the default MTBENCH V2 template, with impartial-evaluator instructions, structured response comparisons, temperature 0.0, and max tokens=4096.The required output is [[A]] or [[B]].
  • Analysis Figures: The analysis reports pairwise error correlations across nine judges and visualizes the Condorcet gap between predicted independent accuracy and actual majority-vote accuracy.Figure 3 uses a hierarchically clustered 9 × 9 matrix, while Figure 4 shows per-bin gaps.

D Distributional Alignment Analysis

The analysis tests whether panel reliability persists when evaluation preserves the full human label distribution rather than reducing labels to binary accuracy. Distributional misalignment increases with item ambiguity, while cross-validation shows the Condorcet-gap estimate is stable.

  • Distributional metrics: The analysis compares each panel label distribution with the 100-annotation human distribution using total variation distance and symmetric KL divergence.Results are stratified by human-entropy tercile.
  • Distributional alignment: TV distance correlates with human entropy (ρs = 0.434, p < 10−46), indicating greater panel–human divergence on more ambiguous items.Mean TV rises from 0.192 on low-entropy items to 0.345 on high-entropy items.
  • Distributional alignment: On high-entropy items, the panel’s distribution diverges substantially from the human distribution, consistent with correlated judges collapsing onto one class.The panel is reasonably calibrated on low-entropy items but less aligned on ambiguous items.
  • Stability checks: Cross-validation estimates a 21.9pp MNLI Condorcet gap versus 22.0pp in-sample, with overfitting ratios near unity across all three datasets.The ratios are 0.997 for MNLI, 0.960 for SNLI, and 1.000 for AlphaNLI.

F Sample Size Convergence

The paper evaluates whether sample size is sufficient for reliable effective-independence estimates. The estimate stabilizes by roughly 200–300 items, with substantially narrower uncertainty by 500 items, supporting the 1,000-item measurement.

  • Convergence procedure: Effective sample size is estimated across entropy-stratified subsamples from N = 100 to N = 1,000, using repeated subsampling below the full sample.For N < 1,000, the analysis uses 100 independent stratified subsamples; N = 1,000 uses a bootstrap confidence interval.
  • Convergence result: The neff estimate stabilizes by N ≈200–300, and its interval narrows substantially by N = 500.The full 1,000-item sample therefore provides a reliable measurement.
  • Scaling curve: The neff scaling curve has an asymptote at 1/¯ϕ ≈2.56 across all judge subsets.The mean scaling curve closely matches the Kish formula with ¯ϕ = 0.391.

I Aggregation Method Details

The aggregation analyses examine whether weighting, judge removal, or shared-error diagnostics can improve panel performance. Results show instability in correlation-based weighting and substantial shared failure patterns, while label-tie handling and tie-related robustness checks constrain interpretation.

  • Leave-one-out analysis: Removing 6 of 9 judges improves accuracy, while removing only Maverick, Qwen3, or Scout hurts it.The three removals that hurt include the two most individually accurate judges.
  • Class-stratified herding: Herding occurs across all gold NLI classes but is strongest for contradiction (¯ϕ = 0.482) and entailment (¯ϕ = 0.466).These class-stratified correlations indicate that shared judge behavior is not confined to one label.
  • Correlation-based weighting: Phi-optimal weighting reaches 72.4% on MNLI and 94.1% on RewardBench but falls to 86.2% on AlphaNLI, below majority voting.The method closes 20.6% of the gap on RewardBench, but its performance varies across datasets.
  • Tie robustness: Gold-label ties affect 28 items, but flipping every tied item changes panel accuracy by at most ±0.7pp and neff by at most 1.3%.The paper attributes tie resolution to an undocumented Counter insertion-order artifact and finds the main results robust to reversing those labels.
  • All-wrong items: Among 51 MNLI items where all 9 judges are wrong, over-prediction of contradiction accounts for 51% of failures.The panel’s wrong answer receives mean human support of 35.3% across these items.
  • Vote ties: Majority-vote ties are rare on three-class tasks: 11 MNLI items (1.1%) and 4 SNLI items (0.4%).They are resolved deterministically for reproducibility, while Monte Carlo simulations use seeded random tie-breaking.
Loading 2605.29800v1…