PaperlayerView exact revision on arXiv

Source-linked AI summary

Pinned and Still Unstable: Within-Judge Verdict Variance and the Noise Floor of LLM-as-Judge Leaderboards

Krishna Chytanya Ayyagari

arXiv:2609.33044v2cs.CL

TL;DR

LLM-as-Judge leaderboards assume that pinned, temperature-zero judges give reproducible verdicts, but evidence on run-to-run verdict variation and its ranking effects is limited. This paper measures that instability and proposes reporting judge stability and ranking uncertainty. It finds that leaderboard precision is limited by unresolved adjacent positions, cross-judge differences, and run-to-run variation in some judges.

  • Problem

    LLM-as-Judge leaderboards treat scores as stable measurements, although the stability assumption is rarely stated and confidence intervals over judge variance are rarely reported.

    Source: §1 · p. 2

  • Method

    The paper introduces decision-focused metrics for judge re-run instability and leaderboard separability, and evaluates their effects on rankings and head-to-head orderings.

    Source: §1 · p. 3, §6.5 · p. 18, §7.3 · p. 22

  • Results

    A fifth to three-quarters of adjacent leaderboard positions are statistically indistinguishable, and only 8 of 15 expected head-to-head orderings survive every judge, versus 12 under the principal judge alone.

    Source: §9 · pp. 26–27

  • Takeaways & Limitations

    The authors propose reporting re-run stability profiles, adjacency separability, uncertainty alongside point estimates, and results from multiple judge families.

    Source: §9 · p. 27

    The study uses judges served on one platform, so precise cross-family ordering should be read as platform-specific; Arena-Hard cross-judge comparisons are also sensitive to truncated answers.

    Source: §8 · pp. 24–25

Abstract

from arXiv · show

Modern LLM evaluation assumes that pinning a judge to a fixed model version and decoding at temperature zero yields reproducible verdicts. We show this assumption fails as a property of how LLM-as-Judge is operationalized on cloud serving infrastructure, not of any particular model family. Across four frontier judges served via a single major enterprise cloud platform and three standard benchmarks (Arena-Hard, AlpacaEval 2, MT-Bench), identical inputs to the same temperature-zero judge, at a constant serving-reported model version, produce different verdicts across re-runs: per-item flip rates of roughly 5% on average and about 40% on the close-call items that decide leaderboard margins, with a per-judge magnitude spanning a 40x range (0.13% to nearly 10%). We introduce metrics tailored to this instability: per-item flip rate, a two-part stability profile (waver fraction and conditional intensity), and adjacency separability. For the principal judge the aggregate ranking is stable (0% top-K instability, 0% pooled winner flip); what degrades is precision: under a paired hierarchical bootstrap, roughly one-fifth to three-quarters of adjacent leaderboard positions are statistically indistinguishable, a noise floor driven mainly by finite prompt sampling rather than the judge. Across judges, leaderboards agree on the coarse ordering but diverge in the middle (Kendall's tau of 0.42-0.64 between Gemini and Sonnet judges on Arena-Hard, values sensitive to answers truncated at the generation cap). Of 15 expected head-to-head orderings we re-judge, 12 survive every re-run of the principal judge but only 8 survive every judge. Leaderboards thus report unhedged point estimates that overstate their precision. We propose a minimal, low-cost reporting protocol: several judge re-runs, published stability profiles and adjacency intervals, and results under at least two judges from different families.

1. Introduction

LLM-as-Judge leaderboards are treated as stable measurements, although pinning a model version and setting temperature to zero do not ensure repeatable verdicts. This paper measures the remaining judge variance and its effects on rankings, arguing for uncertainty-aware reporting.

1. Introduction

  • Repeated runs of the same pinned, temperature-zero judge can flip verdicts on identical inputs, especially on close-call items that decide leaderboard margins.The authors distinguish judge-run variance from finite prompt-sample uncertainty.Source: §1 · p. 2
  • Across three benchmarks and four judges, verdict instability varies by judge over a roughly 40× range, while noisier judges can reshuffle their own top-K sets.The reported Arena-Hard flip rates range from 0.13% for Haiku to 5–10% for Sonnet-4-5.Source: §1 · p. 3
  • Even when the reference judge’s aggregate ranking is stable, 2–7 of 9 adjacent positions are statistically indistinguishable, largely because of finite prompt sampling.The paper separates this adjacency noise floor from genuine judge effects.Source: §1 · p. 3
  • Across judges, leaderboards agree on coarse ordering but diverge on finer rankings; only 8 of 15 expected head-to-head orderings survive every judge, versus 12 under the principal judge alone.The Arena-Hard cross-judge comparisons are sensitive to candidate answers truncated at the generation cap.Source: §1 · p. 3
  • The paper proposes reporting judge re-runs, stability profiles, adjacency separability, and results from more than one judge.It argues that unhedged point estimates do not convey the leaderboard’s measurement noise floor.Source: §1 · pp. 2–3

2. Related Work

Prior work studies judge biases, reliability, inference non-determinism, and finite-sample leaderboard noise. This paper focuses on residual variance between runs of an identically configured judge and its effects on pairwise outcomes and fine-grained rankings.

Judge biases

  • Prior studies document biases within single judge runs, including position, verbosity, and self-preference biases, as well as prompt sensitivity and within-run inconsistency.The cited work also examines adversarial vulnerabilities and scalar-versus-pairwise judging protocols.Source: §2 · p. 3

Novelty

  • Reliability studies measure aggregate consistency, and drift studies compare snapshot means; this paper examines within-snapshot variance and its ranking effects.Schroeder and Wood-Doughty (2024) use a scalar psychometric coefficient but do not trace reliability through to leaderboard rankings or head-to-head claims.Source: §2 · p. 4

Determinism failures

  • Temperature-zero inference can vary across runs; Atil et al. (2024) report accuracy swings up to 15% across repeated runs on eight tasks, while this paper traces non-determinism to verdicts and rankings.The related work attributes such inference variation to floating-point non-associativity, mixture-of-experts routing, and provider-side load balancing.Source: §2 · p. 4

Leaderboard brittleness and finite-sample noise

  • Prior work finds benchmark rankings noisy from finite prompt samples, and this paper isolates judge variance layered on top of that noise.The paper credits the adjacent-position noise floor to prompt sampling rather than claiming it as novel.Source: §2 · p. 4

Gap

  • Existing reliability measures such as agreement statistics and rank correlations do not quantify how often pairwise leaderboard outcomes flip between runs of an identically configured judge.The paper identifies this as a gap alongside the lack of confidence intervals over published rankings.Source: §2 · p. 4

3. Methodology

The study evaluates judge variance across three benchmarks and four configurations served on one platform, using fixed candidate outputs and repeated runs. It defines measures for verdict instability and leaderboard separability, then tests expected orderings while limiting its claims to variance at the judge stage.

3.1 Benchmarks

  • Three benchmarks cover 80 MT-Bench prompts, 805 AlpacaEval 2 prompts, and 500 Arena-Hard prompts, with benchmark-specific scoring and fixed reference models.MT-Bench treats each turn as an independent item; AlpacaEval 2 uses raw, un-length-controlled win-rate.Source: §3.1 · pp. 4–5

3.2 Judges

  • Four judge configurations ran at temperature zero on one cloud platform, with each judge pinned to a constant serving-reported model version.Stage 2 used API-enforced structured verdicts; the study did not test other providers’ serving stacks.Source: §3.2 · pp. 5–6

3.3 Candidate Models

  • The fixed candidate pool spans four model families and capability tiers, including managed open-weight endpoints; candidate outputs were generated once and cached.Some Arena-Hard answers were truncated by the 2,048-token generation cap.Source: §3.3 · p. 6

3.4 Metrics

  • The metrics separate item-level verdict changes, aggregate ranking changes, and adjacent-position separability under a paired hierarchical bootstrap.The stability profile reports waver fraction and conditional intensity separately; the waver fraction depends on the number of runs.Source: §3.4 · pp. 7–8

3.5 Replication Study

  • The replication study re-evaluates 15 expected orderings using leaderboard-style win rates or mean scores across repeated runs and judges.The orderings come from vendor tiers, model generations, public standings, or the authors’ expectations.Source: §3.5 · p. 8

3.6 What We Are Measuring (and What We Are Not)

  • Because candidate outputs are held fixed, observed variance isolates the judge stage; unmeasured partial caching could suppress flip rates, making them lower bounds.Regenerating candidate outputs in ordinary benchmark workflows would add variance beyond what this study measures.Source: §3.6 · p. 9

4. Pilot Studies

The pilots test whether identical inputs can yield different verdicts across repeated runs, then check cross-family behavior, position swapping, and verdict-format reliability. They find item-level instability despite stable aggregate winners and motivate structured output for reliable measurement.

4.1 Pilot 1 — Single-Judge Sniff Test (Tier 0)

  • A pinned, temperature-zero Gemini judge flipped 6.4% of item verdicts across 10 runs, while the aggregate winner stayed stable.The judge wavered on 20% of items, with 32.2% conditional intensity among those items.Source: §4.1 · p. 9

4.2 Pilot 2 — Cross-Family Confirmation and Position-Swap (Tier 1)

  • Two judges from different model families both showed non-trivial item flip rates, supporting cross-family generalization of the effect on the tested platform.With candidate positions swapped, aggregate winners remained unchanged across all 10 runs for both judges; the disparate-tier pair may mask residual position bias on near-ties.Source: §4.2 · p. 10
  • The judges had distinct stability profiles: one wavered on about 21% of items at moderate intensity, while the other wavered on about 7.5% at higher intensity.The study argues that a single stability scalar would lose this distinction.Source: §4.2 · p. 10

4.3 Methodological finding: structured output is required for reliable measurement

  • Text-format parsing produced about 24% parse failures for one judge, while schema-constrained structured output empirically reduced failures to approximately 0%.The authors attribute the pilot failures to a generation pathology under the deliberately minimal thinking budget; the pilot variance also appeared with free-form parsing.Source: §4.3 · pp. 10–11
  • Structured output served as a measurement prerequisite by producing parseable verdict fields and enabling reliable variance measurement.The paper notes that format restrictions can alter reasoning-task behavior, but the pilots showed the same qualitative variance with free-form parsing.Source: §4.3 · p. 11

5. Stage 2 Experimental Design

Four Stage 2 experiments extend the pilots to benchmark scale, compare judge snapshots and families, and re-test expected head-to-head orderings. They find per-judge instability, cross-judge ranking differences, and limited survival of expected orderings across all judges.

5. Stage 2 Experimental Design

  • The four experiments extend the pilots or measure quantities that the smaller pilot samples could not address.The design includes within-snapshot variance, cross-snapshot drift, cross-family judge substitution, and replication of expected orderings.Source: §5 · pp. 11–12

5.1 Experiment 1: Within-Snapshot Variance at Scale

  • Across three benchmarks and four judges, per-item flip rates span about 40×, from 0.13% for Haiku to 5–10% for Sonnet-4-5.The full benchmark sweep used 10 independent re-runs per item and a fixed 10-candidate matrix.Source: §5.1 · p. 12

5.2 Experiment 2: Cross-Snapshot Drift Within the Claude Sonnet Family

  • Claude Sonnet 4.5 and 4.6 produce different Arena-Hard rankings and different within-snapshot stability profiles, so pinning does not eliminate re-run noise.Arena-Hard Kendall’s tau-b is 0.78; per-item flip rates are 5.16% and 3.57%, respectively.Source: §5.2 · p. 12

5.3 Experiment 3: Cross-Family Judge Substitution

  • Cross-judge Kendall’s values range from 0.42–0.87 on Arena-Hard and 0.87–0.96 on AlpacaEval, with Gemini and Sonnet reordering about 18–29% of Arena-Hard model pairs.The authors note that these Arena-Hard comparisons are sensitive to truncated candidate answers.Source: §5.3 · pp. 12–13

5.4 Experiment 4: Replication of Expected Head-to-Head Orderings

  • Of 15 expected orderings, 8 survive every judge that evaluated them, compared with 12 under the principal judge alone.The seven failures comprise three cross-judge disagreements, two within-judge instabilities, and two unanimous contradictions.Source: §5.4 · p. 13

6. Results

The results characterize within- and cross-judge variation across three benchmarks, including ranking stability, disagreement in the leaderboard middle, and how cross-judge ordering claims hold up under length-control checks. Sonnet-4-6 is excluded from AlpacaEval results because its run did not complete.

6.2 Within-judge verdict variance is real and substantial

  • 4.94% of Arena-Hard and 5.29% of AlpacaEval comparisons flip across runs; among wavering cells, conditional flip intensity is 39.6% and 39.1%, respectively.The waver fractions are 12.5% and 13.5%; the authors treat wavering as a proxy for close calls, while noting prompt ambiguity or stochastic rationale generation may also contribute.Source: §6.2 · pp. 13–14

6.3 …but the aggregate ranking does not invert on re-run

  • 0.0% aggregate pairwise winner flips and 0.0% top-K instability for K ∈ {1, 3, 5} show Gemini’s aggregate ranking does not invert across re-runs.The item-level noise largely averages out when pooling roughly 500–805 per-prompt verdicts.Source: §6.3 · p. 14

6.4 The noise floor: adjacent positions are statistically indistinguishable

  • 6 of 9 Arena-Hard and 4 of 9 AlpacaEval adjacent pairs are indistinguishable for Gemini, with the combined noise floor driven mainly by finite prompt sampling.The counts depend on this 10-model pool; the authors describe the broader finding as a substantial share of closely matched adjacent models remaining unresolved.Source: §6.4 · p. 15

6.5 Cross-judge results: variance is a per-judge property

  • 0.13–9.71% of verdicts flip across judges, a roughly 40× spread, while conditional intensity remains near-uniform at 34–40%.Haiku is nearly deterministic at 0.13–0.24%, while Sonnet-4-5 reaches 9.71% on AlpacaEval; the waver fraction, rather than conditional intensity, carries most judge-specific variation.Source: §6.5 · p. 16
  • Kendall’s τ is 0.64 for Gemini versus Sonnet-4-5 and 0.42 for Gemini versus Sonnet-4-6 on Arena-Hard, with disagreement concentrated in the middle.The 0.42 result is not statistically significant, and the Arena-Hard values are sensitive to answers truncated at the generation cap. On MT-Bench, mean absolute scores span 7.64 to 9.58, so scores are not comparable across studies unless the judge is named.Source: §6.5 · p. 17
  • 12 of 15 expected head-to-head orderings survive all Gemini re-runs, but only 8 of 15 survive every judge evaluating each claim.The strict criterion requires an ordering to hold in every re-run of every relevant judge; 10 of 15 hold under the softer majority-of-runs criterion.Source: §6.5 · p. 18
  • Qwen3-Next-80B beats Qwen3-235B under raw and both length-controlled win-rates for all three judges, despite being the longer model.The result rules out verbosity as the explanation for this ordering.Source: §6.5 · pp. 18–19
  • Claude Opus 4.5 versus Qwen3-235B is length-confounded: Opus 4.5 is ~43% shorter, and length control makes the ordering judge- and method-dependent.Gemini and Sonnet-4-5 prefer Opus 4.5 on the length-matched subset; the claim is counted as a failure but flagged as metric-dependent.Source: §6.5 · p. 19
  • Llama-4-Maverick versus Llama-3.3-70B is a near-tie (raw win-rates 0.396 vs. 0.393), making its ordering fragile to re-runs and length control.By contrast, claims #7 and #11 survive raw and both length controls for all three judges with AlpacaEval data.Source: §6.5 · p. 19
  • For claim #5, cross-judge disagreement persists on the length-matched subset, but logistic adjustment makes Haiku prefer Flash (0.863 vs. 0.854), so all three judges agree under that variant.Haiku still prefers Flash-Lite on the length-matched subset.Source: §6.5 · p. 19
  • Sonnet-4-6 is excluded from AlpacaEval results because its run was cut short; the full-scale position-swap check is optional, while pilot rates were 2.86% vs. 2.89%.Gemini, Haiku, and Sonnet-4-5 completed all assigned cells; Sonnet-4-6 completed Arena-Hard and MT-Bench only.Source: §6.5 · p. 19

7. Discussion

The least reliable part of the leaderboard is its fine-grained ordering: close-call items flip often, and many adjacent positions cannot be distinguished statistically. The discussion recommends reporting uncertainty across judge runs and prompts, while noting that the observed causes of nondeterminism remain hypotheses.

7.1 The Fine-Grained Ordering Is the Least Reliable Part

  • 2–7 of 9 adjacent positions are statistically indistinguishable, while the top-ranked model stays the same across every judge re-run.The adjacency noise floor is attributed primarily to prompt sampling; judge-run variance adds at most one further indistinguishable pair.Source: §7.1 · p. 21

7.2 Why Pinning and T=0 Are Not Enough

  • GPU floating-point non-associativity, mixture-of-experts routing, and hardware changes are proposed as possible sources of variance, but the black-box study does not establish them as causes.The authors describe these as hypothesized mechanisms because their endpoints expose effects, not the underlying serving process.Source: §7.2 · p. 22

7.3 A Minimal Protocol

  • The proposed protocol calls for at least five judge re-runs, a two-part stability profile, published per-run scores, and paired-bootstrap tests for adjacent-model ties.It also recommends reporting results under judges from at least two model families and accounting for evaluation-heavy workloads where re-runs multiply judging cost.Source: §7.3 · pp. 22–23

7.4 Implications for Existing Literature

  • Of 15 expected orderings, 7 fail to survive re-runs and judge changes; gaps below 3 percentage points on Arena-Hard or 0.2 points on MT-Bench fall within this study’s prompt-sampling noise floor.These thresholds reflect the benchmarks’ prompt counts and this candidate pool, not universal constants; fixed candidate outputs also make measured judge variance a lower bound for workflows that regenerate answers.Source: §7.4 · pp. 23–24

7.5 Why Judge Stability Is a Profile, Not a Number

  • Per-item flip rates span about 40× across judges, while conditional intensity is near-uniform at 34–40%, so the spread is carried mainly by how often judges waver.Both profile components depend on the number of runs, and Haiku’s Arena-Hard intensity estimate comes from roughly 19 wavering cells, making its precise value indicative rather than exact.Source: §7.5 · p. 24

8. Limitations

The study’s results are bounded by its fixed candidate outputs, single-platform setup, selected replication set, and single-turn evaluation design. Several comparisons also depend on generation lengths and the principal judge’s specific reasoning-budget configuration.

Scope of measured variance

  • Judge variance is measured with candidate outputs held fixed, so the estimates exclude additional variance from regenerating answers in real workflows.Some Arena-Hard answers were cut off at the generation cap, making cross-judge comparisons sensitive to how incomplete answers are scored.Source: §8 · p. 24

Replication and candidate scope

  • The replication set contains 15 expected orderings chosen by the authors rather than sampled from published comparisons, so its survival rate is not a population-wide replication rate.The candidate pool covers 10 models across four families, and generalization beyond that pool remains to be shown.Source: §8 · p. 25

Platform and judge coverage

  • All judges were served on one enterprise cloud platform, and the study does not directly measure within-snapshot variance for GPT-4-class judges hosted by OpenAI.The precise cross-family stability ordering should therefore be read as “as served on this platform”; black-box endpoints cannot separate model and serving-stack contributions.Source: §8 · p. 25

Single-turn evaluation

  • MT-Bench turn-2 items were judged without turn-1 context and were about twice as unstable as turn-1 items, so pooled MT-Bench rates partly reflect this single-turn design.The findings characterize single-turn judgments; multi-turn judging may have a different profile.Source: §8 · pp. 25–26

Position bias

  • The position-swap check used a disparate-tier pair rather than near-tie pairs, and the fixed answer order could leave a position preference unaveraged in Stage 2.A position component on the closest pairs is not fully excluded and could shift absolute win rates.Source: §8 · p. 26

Reasoning-budget configuration

  • Gemini’s stability results are specific to a minimum thinking budget of 128 tokens, which was not varied and could change the judge’s stability in either direction.The principal judge’s absolute flip rates therefore depend on this configuration.Source: §8 · p. 26

9. Conclusion

Across three benchmarks and four judges, pinned temperature-zero judgments vary across re-runs, with the magnitude differing by judge. Although this variance does not invert the reference judge’s aggregate ranking, it leaves leaderboard precision unresolved and motivates uncertainty reporting across runs, prompts, and judge families.

9. Conclusion

  • Across three benchmarks and four judges, within-judge verdict variance spans about 40×, from 0.13% for Haiku to 5–10% for Sonnet-4-5.Close-call items show about 40% flip intensity; the magnitude is a per-judge property as served on the platform.Source: §9 · p. 26
  • Between one-fifth and three-quarters of adjacent positions are statistically indistinguishable, while cross-judge comparisons reorder a significant ≈18% of Arena-Hard model pairs between Gemini and Sonnet-4-5.Of 15 expected orderings, 12 survive every principal-judge re-run but only 8 survive every judge.Source: §9 · pp. 26–27
  • The authors recommend several judge re-runs, published stability profiles and adjacency separability, multiple judge families, and uncertainty intervals alongside point estimates.They advise reading leaderboards as approximate orderings with implicit ties of unknown width and naming the judge.Source: §9 · p. 27

Appendix A — Reproducibility

The reproducibility materials describe the cost and execution setup for the pilot and full evaluation sweep. The paper reports that data, code, and related artifacts are available on reasonable request, with a cleaned public release in preparation.

Appendix A — Reproducibility

  • The pilot studies cost approximately $33 in platform API spend and ran in under three hours on a single workstation.Source: Appendix A · p. 28
  • The Stage 2 sweep covered four judges, 10 candidates, and three benchmarks at R=10, with Sonnet-4-6 completing only Arena-Hard and MT-Bench, at a cost on the order of $5,000.It ran on a single VM, with the project’s Anthropic request-rate quota limiting throughput.Source: Appendix A · pp. 28–29
  • Runner code, verdict data, candidate responses, and prompt templates are available on reasonable request, while a cleaned public release is in preparation.Source: Appendix A · p. 29

Appendix B — Per-Judge Identification (for Pilot Studies §4)

The appendix identifies the judges used in the pilot studies and clarifies how their pilot evaluations differed from the later structured-output protocol.

Appendix B — Per-Judge Identification (for Pilot Studies §4)

  • The anonymized judge mapping identifies gemini-2.5-pro and claude-haiku-4-5@20251001 as the pilot judges.The mapping specifies the platform and serving details for each judge.Source: Appendix B · p. 29
  • Pilot studies used shared prompts, candidate inputs, and evaluation pipelines, differing only in judge model identity.They used free-form verdict parsing rather than structured-output enforcement.Source: Appendix B · p. 29
  • The paper reports a 24% parse-failure rate on Tier 1, directly attributing that rate to its text-format choice, as documented in §4.3.Structured-output enforcement was adopted starting with the Stage 2 sweep (§5).Source: Appendix B · p. 29

Appendix C — Per-Claim Replication Detail (for §6.5)

Appendix C reports whether 15 expected model orderings reproduced across judge re-runs and distinguishes within-judge instability, cross-judge disagreement, and unanimous contradiction. It also documents the provenance and limits of the expected orderings.

Appendix C — Per-Claim Replication Detail (for §6.5)

  • For each claim and judge, the table reports first-run reproduction and the fraction of 10 re-runs that reproduced it; a claim survives a judge only at 1.0.Claims 1 and 4 reproduce at 1.0 for all four judges, while Claim 3 reproduces at 0.0 for all judges with data.Source: Appendix C · p. 29
  • The table includes mixed outcomes: Claim 2 reproduces at 0.9 for Sonnet-4-5, while Claim 5 reproduces at 0.0 for Haiku and 0.9 for Sonnet-4-5.Claim 8 reproduces at 0.6 for Gemini and 1.0 for Haiku and Sonnet-4-5.Source: Appendix C · p. 29
  • Within-judge instability requires a reproduced fraction strictly between 0 and 1 while the aggregate winner remains the claimed model; cross-judge disagreement means judges reach different aggregate winners.Unanimous contradiction is a reproduced fraction of 0.0 across all judges with data; Claim 6 at Haiku's 0.5 has no stable aggregate winner.Source: Appendix C · p. 30
  • The 15 expected orderings draw on vendor tiering, model generation, public leaderboard standings, or the authors’ expectations, rather than uniformly authoritative benchmark results.Claims 9–11 and 13 rely on dated public leaderboard standings and should be treated as leaderboard-plausible at access time.Source: Appendix C · pp. 30–31

Appendix D — Changes from v1

Appendix D records corrections and clarifications made in version 2, including revised replication framing, candidate-generation details, and reporting caveats.

Appendix D — Changes from v1

  • The replication set is now framed as 15 expected orderings, with 12 surviving every principal-judge re-run and 8 surviving every judge.The counts are unchanged from version 1; a second judge overturns one-third of the 12 reproduced by the principal judge.Source: Appendix D · p. 31
  • Candidate answers used a 2,048-token generation cap, and some longer Arena-Hard answers reached it, potentially affecting cross-judge comparisons.The 4,096-token limit applies to the judge, not candidate generation.Source: Appendix D · p. 31
  • Top-K reshuffle estimates are now ranges when exact ties at the K boundary affect the result; Sonnet-4-6’s top-3 set is stable.The revised text also limits the claim that all judges’ top-K sets are stable to the top-1 model.Source: Appendix D · pp. 31–32
  • Version 2 reports at least 99.8% valid Stage 2 verdicts for every judge-benchmark pair, replacing the earlier claim of zero parse errors.The per-round parse-error abort threshold is 1% for the first round and 0.5% thereafter.Source: Appendix D · p. 32
  • The adjacency bootstrap uses 20,000 resamples, and Stage 2 comparisons fixed the reference answer in slot A and the candidate in slot B.Win rates were computed by model identity, so correcting the prior slot notation does not change reported numbers.Source: Appendix D · p. 32
  • The revised framing limits cross-family claims to the judges tested on one platform and attributes the adjacency noise floor to prompt sampling rather than judge variance.It also qualifies generalization beyond the tested platform and replaces an unqualified statement about prior work.Source: Appendix D · p. 32

Related papers

Loading 2609.33044v2…