Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM

1 Introduction

• With one judge, generalizability asymptotes to σ2_s/(σ2_s+σ2_sj) as item count grows, so item collection cannot raise the reliability ceiling.The system-by-judge error term contains no item-count factor.Source: §1 · p. 2
• At Eρ2 = 0.75, two judges at 30 items match 77 items with one judge; no item count reaches 0.80.Whether a ceiling clears a usable threshold depends on the judge universe; the analysis uses six current-generation judges as primary and reports the wider panel as sensitivity.Source: §1 · p. 2
• Under pairwise preference in both presentation orders, the ceiling rises to 0.986, but first-position presentation adds an 8.6-point win advantage.The 8.6-point bias is 1.23× the median improvement in the 53 recovered published win-rate comparisons; 70% fall below the measured pairwise floor.Source: §1 · p. 2
• On matched material, MT-Bench’s native protocol raises the system-variance share from 4.8% to 8.4% and reduces the judges needed for Eρ2 = 0.8 from 18 to 3.The authors identify rubric and scale as a cheaper lever than adding items or judges.Source: §1 · p. 2
• At native item counts, measured noise floors are 0.41–1.24 points on a 0–5 scale, while the median reported improvement is 0.28 points.All 17 recovered MT-Bench improvements fall below MT-Bench’s own floor.Source: §1 · p. 2
• Screening items by between-system variance moved system variance roughly two orders of magnitude in a pilot, yielding a 16.6% share versus Arena-Hard’s 19.2%.The screened items were chosen for that property; screening was the only tested lever that raises the ceiling rather than merely approaching it.Source: §1 · p. 2
• Across five open-weight judges, disagreement was 2.17× greater across model families than within them after standardizing scale usage.The paper also reports that system variance falls to 1.9–4.2% on three of four item sets when restricted to ten current frontier models.Source: §1 · p. 2
• An audit of 628 papers, double-coded by two independent models, found that disclosure needed to assess these issues is largely absent and quantified how reliably each disclosure can be coded.Source: §1 · p. 2

2 Related work

• Prior sample-complexity work covers accuracy benchmarks and leaves judged evaluations open; this paper adds the judge facet, which item collection cannot overcome.Related work on judge reliability documents disagreement but does not derive its sample-size consequence.Source: §2 · p. 3
• Unlike prior LLM-evaluation applications of generalizability theory that treat items as the object, this paper treats systems as the object of measurement.That framing makes system-by-judge interaction the binding term in the error variance.Source: §2 · p. 3

3 The measurement model

• The score model decomposes judgments into system, item, judge, and interaction components, with three replicates used because the system-by-item-by-judge term and residual are not separately identified in a single run.The passage gives the model as Xsijr = µ + νs + νi + νj + νsi + νsj + νij + νsij + νr(sij).Source: §3 · p. 3
• With one judge and item count tending to infinity, item-dependent error terms vanish but system-by-judge error remains, producing a reliability ceiling.The consequence follows from standard D-study algebra; the paper’s contribution is measuring it for judged benchmarks.Source: §3 · p. 4
• The paper reports minimum detectable difference as its primary quantity because its bootstrap interval is roughly ±7% of the point estimate, while the interval for system variance exceeds 100%.Generalizability is reported as an interval.Source: §3 · p. 4

4 Experimental setup

• The design uses 42 systems, nine judges, three replicates, and 0–5 pointwise scoring; the same judges, rubric, and scale span all four item sets.The system panels are close but not identical, so cross-set differences are attributable to items only with that caveat.Source: §4 · p. 4
• The screened set adds independent generation-sample pairs with a true difference of zero for a Type I error check, but its system panel differs from the other sets.Refitting without the null arm changes the reported MDD and ceiling little, limiting its effect on cross-set comparison.Source: §4 · p. 4
• With six current-generation judges, three of four item sets clear Eρ2 = 0.8; under the nine-judge universe, none does.MT-Bench's ceiling is 0.798 with six judges, and a single judge cannot reach 0.8 at any item count; the structural result is unchanged across universes.Source: §4 · p. 4
• The screened item set retains candidates by between-system variance, targeting system signal that item selection for difficulty, diversity, or realism may not provide.A small pilot showed roughly two orders of magnitude change, but its arms differed in item count as well as screening, so the paper treats only the direction and order of magnitude as informative.Source: §4 · pp. 4–5
• Table 1 reports MDD and Eρ2 at 30 items, one judge, and one run, with bootstrap intervals and the judge count needed to reach Eρ2 = 0.8.Its P(≥0.8) column gives the bootstrap probability that the ceiling reaches the conventional threshold.Source: Table 1 · p. 5

5.1 Noise floors

• 0.58–1.24 points on a 0–5 scale is the MDD range at 30 items, one judge, and one run; system variance is 4.1–4.5% on MT-Bench and AlpacaEval 2.Table 1 uses the nine-judge sensitivity universe, not the primary six-judge universe.Source: §5.1 · p. 5

5.2 Items cannot lift the ceiling

• Adding items alone leaves all four single-judge curves below 0.8 under the nine-judge universe: asymptotes are 0.600 (MT-Bench), 0.706 (AlpacaEval 2), 0.746 (screened items), and 0.792 (Arena-Hard), which has a 41% chance of clearing the threshold.At 30 items, reaching Eρ2 = 0.8 requires 25 judges for MT-Bench, 7 for AlpacaEval 2, and 2 each for the screened set and Arena-Hard.Source: §5.2 · p. 5, Figure 1 · p. 6

5.3 Protocol design moves the ceiling

• The measured item cost diverges near the ceiling: on matched MT-Bench, one judge cannot reach Eρ2 = 0.8 at any item count when the ceiling is 0.798.At Eρ2 = 0.75, two judges at 30 items match one judge at 77 items; at 0.78, the comparison is three judges versus 209 items.Source: §5.3 · pp. 6–7
• 8.4% of variance is system difference under MT-Bench’s matched native protocol versus 4.8% under the generic rubric; the ceiling rises from 0.623 to 0.798.The judges needed for Eρ2 = 0.8 at 30 items fall from 18 to 3.Source: §5.3 · p. 6, Table 2 · p. 7

5.4 The pairwise protocol, and where the ceiling goes

• Order-balanced, the single-judge ceiling is 0.986; first-slot systems win 8.6 percentage points more often than when placed second, while 37 of 53 (70%) published improvements fall below the floor at native item counts with one judge.Source: §5.4 · pp. 7–8
• 37 of 53 published win-rate improvements fall below the order-balanced pairwise floor at one judge, and 27 of 53 fall below it at two judges.The measured floors are 19.1 points at 30 items, 10.1 at Arena-Hard’s native 500 items, and 9.8 at AlpacaEval 2’s native 805 items for one judge.Source: §5.4 · p. 8

5.5 Which judges count: the primary universe

• 0.798 is MT-Bench’s ceiling with the six current-generation judges, up from 0.600 with nine; three of four item sets then clear 0.8.The primary universe is the six current-generation judges; the nine-judge panel is reported as sensitivity.Source: §5.5 · pp. 8–9

5.6 What improvements do papers actually claim?

• 0.28 points is the median reported pointwise improvement on a 0–5 basis, below every measured floor; all 17 recovered MT-Bench improvements are below its own floor.At MT-Bench’s native 80 items, its floor is 0.54 points; the matched comparison is limited to 17 recovered improvements.Source: §5.6 · pp. 9–10

6 What published work discloses

• The audit frame contains 628 arXiv papers from 2024–2026; 320 were sampled, 312 coded successfully, and 227 reported a judged system comparison.Coding used a pinned model at temperature zero, constrained to quote evidence or answer “not stated”; spot-checked fields agreed at approximately 88%.Source: §6 · p. 10
• None of the 17 recovered MT-Bench improvements exceeds its floor at the benchmark’s native 80 items.The paper limits this check to the one benchmark whose floor it measured directly rather than transferring a floor across instruments.Source: §6 · p. 10
• The authors conclude that reported differences sit below instrument resolution while papers omit information needed to assess that gap.The disclosure audit and comparison of recovered pointwise improvements with the MT-Bench floor support this conclusion.Source: §6 · p. 10

7 Recommendations

• Report the judge universe, since a reliability or significance claim is undefined without specifying it.Source: §7 · p. 10
• Use the benchmark’s native protocol and disclose which protocol was used; on matched MT-Bench material, it moved the ceiling from 0.623 to 0.798 and reduced judges needed from 18 to 3.The text identifies this as the cheapest lever because it costs nothing per call.Source: §7 · p. 11
• Spend on judges before items once item returns are modest; past that point, the marginal return on items is near zero.Source: §7 · p. 11
• Report MDD alongside the delta, screen items for discrimination, and disclose replication and whether inference caches responses.Source: §7 · p. 11

8 Limitations

• Table 1 rows use different system panels, so cross-set comparisons are matched on judges, rubric, and scale rather than on systems.Removing the 23 null-arm replicas barely changes the ceiling or MDD, but does not remove the panel mismatch.Source: §8 · p. 11
• The judge-panel results do not establish broad panel-composition generality: the closed panel spans two families, and the open-weight panel varies neither quantisation nor serving configuration.The open-weight panel uses five models at one quantisation; Qwen3 8B was excluded after 13.0% of calls returned empty completions, leaving one family represented by a single model.Source: §8 · p. 11
• The pairwise arm measures a general protocol effect, not the deployed Arena-Hard or AlpacaEval 2 pipelines.It uses the paper’s items and judges with one fixed baseline, so benchmark-specific pipeline effects remain unresolved.Source: §8 · p. 11
• Arena-Hard contributes only 17 items, making its system variance least precisely estimated and its ceiling interval the widest among the four sets.This was a collection-budget constraint; approximately 15% of generations were truncated at the token cap, consistently penalising verbose systems across judges.Source: §8 · p. 11
• The audit’s replication field conflates repetition for variance estimation with repeated calls for order control, making 19.8% a conservative upper bound.The SummEval comparison covers one dimension of one dated summarisation task and may flatter judges.Source: §8 · p. 11

9 Conclusion

• Because system-by-judge error has no item-count term, a single-judge protocol has a reliability ceiling that additional items cannot raise.This structural result holds across judge panels, rubrics, and benchmarks.Source: §9 · p. 11
• MT-Bench’s ceiling is 0.623 under a generic rubric and 0.798 under its native protocol; across item sets, usable reliability also depends on the judge universe.Three of four item sets clear Eρ2 = 0.8 with six current-generation judges, while none does under the nine-judge universe that includes a legacy and a deliberately weak judge.Source: §9 · p. 12
• Two judges at thirty items match 77 items at one judge at Eρ2 = 0.75, while all 17 recovered MT-Bench improvements fall below MT-Bench’s own floor.On a matched design, LLM judges are more self-consistent than human experts; the limit is attributed to measurement design, not a failure of LLM judges.Source: §9 · p. 12

A Supporting analyses

• Under four judge-universe choices, MT-Bench’s ceiling ranges from 0.600 to 0.798, so reliability without a stated judge universe is uninterpretable.The ordering is stable across choices, but the level changes.Source: A Supporting analyses · p. 12
• After scale usage is standardised, open-weight judges disagree 2.17× more across model families than within them, while changing the serving route explains only 1.4%–5.0% of distinct-model disagreement.Mixing families buys independence but not a smaller panel; serving the same model behind several endpoints buys almost none of the independence of distinct models.Source: A Supporting analyses · p. 12
• Among ten current frontier models, system variance is 1.9% on screened items and 2.8% on MT-Bench, while configuration-given-model accounts for 55.7% of system variance on MT-Bench.Treating models rather than configurations as the measurement target lowers the MT-Bench ceiling from 0.600 to 0.404 and AlpacaEval 2’s from 0.706 to 0.567.Source: A Supporting analyses · pp. 12–13
• Screened-item discrimination transfers across judge families: screening lifts system share 3.16× for OpenAI-family judges and 3.82× for Anthropic-family judges.Item-discrimination rankings across random halves of the system panel correlate at Spearman ρ = 0.757, versus 0.597 for MT-Bench and 0.551 for Arena-Hard.Source: A Supporting analyses · p. 12
• Averaging k judges is equivalent to an ensemble of size k, and disjoint ensembles’ observed-to-predicted variance ratios range from 0.93 to 1.11.Replicate variation is measured at fixed decoding temperature; separating temperature effects would require crossing temperature with judge, which was not done.Source: A Supporting analyses · p. 13
• On SummEval, higher Eρ2 coincides with a smaller Kendall τ gain against experts, and LLM judges are more self-consistent than human experts on the matched design.Moving from (ni=10, nj=1) to (ni=50, nj=9) raises Eρ2 from 0.711 to 0.959 but Kendall τ from 0.596 to 0.710; recovery of the human top three stays at 23%–32%, against an 18.8% chance baseline. Rater-by-system variance per unit of system signal is 0.07–0.13 for LLM judges, 0.34 for experts, and 0.67 for crowd workers.Source: A Supporting analyses · p. 13
• Self-preference explains 0.8%–4.8% of σ2_sj, while response length explains 13%–17% of between-system variance on public benchmarks; screened-item length-score correlation is −0.004.On public benchmarks, within-item length-score correlation is 0.21–0.25; the screened items’ explicit format and length constraints remove that correlation.Source: A Supporting analyses · p. 13
• Item-only bootstrap half-widths of 0.34–0.49 points understate the true single-judge MDD of 0.58–1.24 by 1.3×–2.8×.Adjacent systems are misordered 45%–49% of the time at one judge. Under the examined call budgets, the cost-optimal judge count grows with budget, while one run remains optimal.Source: A Supporting analyses · p. 13

B Reading the main tables

• Across item sets, MDD ordering changes with the ruler: Arena-Hard is 1.57 SD, screened items 1.77 SD, AlpacaEval 2.31 SD, and MT-Bench 2.67 SD.The screened set has the largest raw MDD but the second-smallest standardized MDD; raw points suit comparing a benchmark floor with improvements reported on that benchmark, not ranking benchmarks.Source: B Reading the main tables · pp. 13–14
• Screening raises σ2_s to 0.456 from MT-Bench’s 0.044, but also raises σ2_si to 0.474 from 0.286 and σ2_sij to 0.683 from 0.267.The screened set’s single-judge ceiling is 0.746 and its standardized MDD is 1.77 SD, compared with Arena-Hard’s 0.792 and 1.57 SD.Source: B Reading the main tables · p. 14
• Single-judge MDD at native item counts is 0.41–1.24 points, while MDD at infinite items is 0.40–1.09 points.MDD@30 is 0.58–1.24 points; all three measures are single-judge, single-run, and change little because items saturate.Source: B Reading the main tables · p. 14
• System variance is defined over the studied population of systems, just as Eρ2 is defined over a universe of judges.The 42 systems comprise 14 models × 3 prompt configurations; restricting the system population to near-frontier models changes discriminating power.Source: B Reading the main tables · p. 14

• 88.0% overall agreement with blind human coding (κ = 0.73) validates the automated audit; the repeated-evaluation field reaches κ = 0.89.Validation used a second model family and a blind human-coded subsample; whether the scorer was named reached κ = 1.00.Source: C Audit and infrastructure detail · p. 14
• 16.7% of papers were coded by the human as repeating evaluation, versus 20.0% by the automated coder; the human-backed figure is reported after adjudication.The automated coders disagreed on replication (κ = 0.47); human coding aligned closely with the first coder, identifying the second coder as erroneous.Source: C Audit and infrastructure detail · p. 14
• Version pinning remains ambiguous: agreement is κ = 0.45, with 53% automated and 73% human coding.The human coder used a more generous standard for what counts as a pinned version.Source: C Audit and infrastructure detail · p. 14
• 146 of 227 papers reporting judged comparisons (64.3%) state both a judge model and item count, the minimum needed to calculate an implied floor.Among recovered deltas, 40% of fully disclosing claims versus 24% of others clear the transferred MT-Bench floor; the absolute rates are not trustworthy, but the contrast uses an identical floor.Source: C Audit and infrastructure detail · pp. 14–15
• The authors distinguish missing disclosure from incorrect conclusions and identify reporting as a low-cost fix for papers readers cannot otherwise check.They note that better-disclosed claims are not the fragile ones, while disclosure itself does not make a result survive.Source: C Audit and infrastructure detail · p. 15

• For two systems sharing items and judges, the item and judge main effects cancel, leaving system interactions; this relative error suits within-study comparisons, not fixed external standards.The variance of the difference is 2σ2δ because the systems are independent draws from the system population given the shared design.Source: D Estimation notes · p. 15
• With at least 20 items, the normal approximation is supported by the central limit theorem; at very small item counts it may be optimistic.The notes recommend substituting a t quantile on the appropriate error degrees of freedom when item counts are very small.Source: D Estimation notes · p. 15
• Expected-mean-squares ANOVA provides an unbiased, closed-form estimator that makes repeated refits cheap and deterministic, but individual variance components can be negative near zero.Components are truncated at zero for reported variance shares but left raw for computing σ2δ; REML avoids negative estimates but is iterative, so intervals use a parametric bootstrap rather than asymmetric-boundary-sensitive Wald intervals.Source: D Estimation notes · p. 15
• With 42 systems, the system facet has 41 degrees of freedom and is the least precisely estimated component, motivating bootstrap intervals for ceilings.Arena-Hard has the fewest items and therefore the widest interval, making it the cell the authors are least willing to interpret.Source: D Estimation notes · p. 15

E Additional figures

• At a fixed call budget, the MDD optimum is interior and shifts toward more judges as the budget grows.The figure plots MDD against how the budget is split between judges and items; the system-by-judge component is not addressed by adding items.Source: Figure 2 · p. 16
• Wrong-order probability for adjacent systems in the true top ten is plotted against item count at one and three judges; a third judge improves rank stability more than doubling items.Source: Figure 3 · p. 16
• The sizing nomogram shows Eρ2 contours over judge and item counts; each contour flattens, and below its judge threshold no item budget reaches the target.Source: Figure 4 · p. 17

F Artifacts

• Sieve is a 38-item measurement instrument selected for between-system discrimination, not a capability benchmark representative of a task distribution.Items were retained from a pool of 71 when between-system variance was at least 1.0 on the 0–5 scale; the main study uses 30 of the 38.Source: F Artifacts · p. 17

Artifacts.

• The released materials include 373,019 crossed judgments, Sieve item data, 4,608 pairwise judgments in both orders, a D-study calculator, and the audit materials.The datasets and item sets are CC BY 4.0, and code is MIT-licensed.Source: Artifacts · p. 17
• The datasets and code are available from the author on request; reproducing judgments against another endpoint requires readers to supply provider credentials.The records use dated model identifiers, and another endpoint may serve different snapshots behind the same names.Source: Artifacts · p. 17