Behind Aggregate Scores: What Systematically Varying Arena-Hard's LLM Judge Reveals

• A stable aggregate does not establish stable prompt-level judgments, which can serve as preference labels, regression tests, or safety-review evidence.Source: §1 · p. 1
• The comparisons cover both instruction sensitivity and unchanged-run variation, using the test–retest disagreement of the setups involved as a reference.Source: §1 · p. 1
• The study holds 500 Arena-Hard v2.0 prompts and both models’ responses fixed while GPT-4.1 judges them under eight combinations of criteria, output format, and task framing.Source: §1 · p. 2
• Across 17 runs, only 131 of 496 complete prompts (26.4%) retain the same outcome throughout; label-only output has 28.2 percentage points more changes than the compared reruns, on average.Source: §1 · p. 2
• The authors recommend testing prespecified alternative setups, repeating each unchanged, and reporting aggregate scores and prompt-level changes alongside each setup’s repeat disagreement.Source: §1 · p. 2

2.1 Data and judge setups

• The eight GPT-4.1 setups form a 2 × 2 × 2 design varying criteria, whether the judge reasons before its verdict, and task framing.Source: §2.1 · p. 2
• The label-only package also removes reference-answer and error-checking instructions, so it does not isolate the effect of visible reasoning.Source: §2.1 · p. 2
• The coding-assistance framing fits 253 coding prompts but not 247 mathematics prompts, and the separate Claude and Gemini comparisons also vary implementation and generation settings.Source: §2.1 · p. 2
• The main October experiment repeats all eight setups twice and the default three times using pinned GPT-4.1 at temperature zero; paired comparisons retain prompts complete in every run compared.Source: §2.1 · pp. 2–3

2.2 Outcomes and analysis

• The raw win-equivalent score maps order-averaged verdicts to a 0-to-100 scale, treating five ordinal levels as equally spaced and comparing paired changes on the same prompts.Source: §2.2 · p. 3
• A prompt-level boundary change occurs when an order-averaged outcome moves among candidate win, baseline win, and tie; sensitivity counts how many of the other seven setups change it relative to default.Source: §2.2 · p. 3
• Excess over test–retest disagreement subtracts the two compared setups’ average unchanged-run disagreement from their paired boundary-change rate; the estimate assumes independent runs.Source: §2.2 · p. 3

3.1 Similar aggregate scores can conceal different prompt outcomes

• Across the eight setups, raw win-equivalent scores range from 49.36 to 53.87, yet only 131 of 496 prompts (26.4%) retain the same outcome in every run and setup.Source: §3.1 · pp. 3–4
• The criteria package changes 26.9% of prompts with a −0.49-point aggregate change, while coding-assistance framing adds 3.8 points above test–retest disagreement, not distinguishable from default disagreement alone.The framing excess is concentrated in mathematics prompts, where the description is inaccurate.Source: §3.1 · p. 4
• The label-only setup lowers the paired aggregate by 4.09 points and changes 41.6% of prompt outcomes, 28.2 points above test–retest disagreement.Most changes are movements into ties: 31.4% become ties, while the tie rate rises from 18.5% to 43.5%.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 Sensitivity concentrates near the default setup’s decision boundary

• Changed prompts have a smaller mean absolute default margin than unchanged prompts (0.69 versus 1.05), and change rates fall as the margin grows.Source: §3.2 · p. 5
• An unchanged default rerun changes 43.4% of zero-margin prompts and 33.8% of prompts at a 0.5 margin, compared with 4.3% and 5.5% at the two highest margins.Source: §3.2 · p. 5
• Thirty of 496 prompts (6.0%) change under all seven alternative setups, and their mean absolute margin is 0.65 versus 1.30 for prompts stable across all seven.Source: §3.2 · p. 5

4 Discussion

• About one point of aggregate movement accompanies 21.2% changed outcomes on an unchanged default rerun; 56 prompts move toward the candidate and 49 toward the baseline, mostly cancelling in the average.Source: §4 · p. 5
• Accuracy-prioritising criteria and Gemini each barely move the headline score, yet change 5.7 and 19.0 percentage points more prompts than a default rerun, respectively.Source: §4 · p. 5
• 39.0% of instruction-change outcomes involve ±0.5 scores, versus a 29.5% base rate; treating ±0.5 as ties lowers default disagreement to 19.3% but leaves every excess above zero.Source: §4 · p. 5
• Similar numbers of changed outcomes favor each model, so a judge can pass usual bias checks yet still change a fifth of its verdicts on rerun.Source: §4 · p. 5
• Leaderboard comparisons can remain similar while individual verdicts change; across three default runs, GPT-4.1 mini ranks 11th, 7th, and 11th among 28 models.Individual verdicts also matter when used as preference labels, safety-review flags, regression signals, or release inputs; about a quarter of prompts retain the same outcome across all 17 GPT-4.1 runs, which does not establish correctness.Source: §4 · p. 5
• Between-judge agreement of 0.435 to 0.549 should be compared with the 0.763 average agreement across unchanged default runs, not with perfect agreement.The study did not test whether combining judges improves reliability or accuracy.Source: §4 · p. 6
• Evaluations relying on individual verdicts should prespecify plausible alternative setups, repeat each unchanged, and report paired aggregate and prompt-level changes alongside each setup’s test–retest disagreement.Test–retest disagreement ranges from 2.0% to 21.2% here, so the main setup’s repeat cannot stand in for the others.Source: §4 · p. 6
• These comparisons establish neither which verdict is correct nor that one judge setup is better than another.Source: §4 · p. 6

5 Limitations

• The label-only contrast removes several instructions as well as the explanation requirement, so results apply to tested instruction packages, not individual instructions.Source: §5 · p. 6
• Each GPT-4.1 setup has only one to three repeat pairs from one day, and the study lacks human or ground-truth labels to assess verdict accuracy.The repeats measure variation but cannot distinguish genuinely ambiguous prompts from judge instability; substitution conditions also test whole implementations and were run only in May.Source: §5 · p. 6
• Results cover one benchmark, one near-50% model pair, and GPT-4.1 as the main judge, so the change rates should not be assumed typical of wider capability gaps.Response style was examined only in May; further work is proposed on another benchmark and model pair, different-day repeats, and majority verdicts.Source: §5 · p. 6

6 Conclusion

• An unchanged rerun moves the aggregate score by about one point but changes 21.2% of prompt outcomes.Source: §6 · p. 6
• The label-only setup changes outcomes on 28.2 percentage points more prompts than test–retest disagreement and lowers the aggregate by 4.09 points.Source: §6 · p. 6
• Accuracy-prioritising criteria add 6.7 percentage points of outcome changes with little aggregate movement, while coding-assistance framing adds 3.8 points.Source: §6 · p. 6
• Evaluations relying on individual verdicts should report both aggregate scores and prompt-level agreement, comparing each setup change with an unchanged repeat.Source: §6 · p. 6

A Related work

• LLM judges support scalable pairwise evaluation and are used in open pipelines and industry workflows, while prior work identifies position, length, and formatting biases [Zheng et al., 2023, Dubois et al., 2024, Li et al., 2025b].Source: A Related work · p. 9
• Answer-order swapping and style adjustment address bias within a fixed judge configuration, not verdict changes caused by changing that configuration.Source: A Related work · p. 10
• Prior analytic-flexibility research documents divergent findings across defensible analyses [Botvinik-Nezer et al., 2020, Silberzahn et al., 2018], while underspecification and predictive multiplicity show similar-performance models can vary [D’Amour et al., 2022, Marx et al., 2020, Semenova et al., 2022].This study holds model responses fixed and varies the judge, drawing on measurement validity’s distinction between a construct and its operationalisation.Source: A Related work · p. 10
• Prompt-format perturbation studies change the prompt given to the tested system and report aggregate swings; this paper changes the judging instrument and examines prompt-level changes despite stable aggregates [Sclar et al., 2024].Source: A Related work · p. 10
• Prior work measures unchanged-run variation and judge-design sensitivity, but often lacks a rerun baseline; this study reports design-change disagreement as excess over test–retest disagreement [Wei et al., 2025, Ayyagari, 2026, Vaugrante and Hagendorff, 2026, Zhang, 2027].The paper reports a 21.2% default change rate, above Ayyagari’s 0.13% to 9.7% range, while noting differences in design that could contribute.Source: A Related work · p. 10

B Additional figures

• Criteria and label-only changes produce more direct reversals than rerunning, while coding-assistance framing does not exceed rerun reversal rates.Direct reversals are 5.1% for the criteria package and 4.0% for label-only output, versus 2.0% on a default rerun; coding-assistance framing is +0.9 points versus rerunning, with an interval including zero.Source: B Additional figures · p. 10
• Figure 1 compares aggregate-change magnitudes with prompt-level change rates for three setups, against the default setup’s test–retest disagreement.The table’s excess, rather than a bar’s position relative to the dashed line, determines whether a setup exceeds test–retest disagreement.Source: Figure 1 · p. 11
• Across all seven alternative setups, prompt-level changes exceed unchanged-rerun changes at every margin, with the smallest gap at zero margin.Both rates fall as margin grows and level off at the two highest margins.Source: Figure 2 · p. 11
• Of 496 complete prompts, 162 remain unchanged under all seven alternative setups, while 30 change under all seven.Figure 3 shades each prompt by the number of alternative setups that change its outcome relative to the default.Source: Figure 3 · p. 12

C Additional robustness checks

• At the five absolute margins, excess change rates are +18.8, +21.5, +23.4, +7.6, and +10.3 points, while coding-only framing is indistinguishable from rerunning.The excess peaks at intermediate margins; the full-sample framing estimate also includes inaccurate framing for mathematics prompts.Source: C Additional robustness checks · p. 12
• Counting ±0.5 scores as ties lowers default test–retest disagreement to 19.3%; style tests do not establish that style is excluded as a mechanism.Paired complete cases prevent missing Gemini outputs from changing comparisons through sample composition; 15 parse failures show no stated differences in relative response length or markdown density and span both task categories and all default outcomes.Source: C Additional robustness checks · p. 12
• Gemini 2.5 Pro’s incomplete run covers 123 prompts paired with Flash, and reported cases show setup-dependent verdicts even when answers can be checked against supplied examples.On those prompts, Pro changes the aggregate by −2.34 points with a 38.2% boundary-change rate, versus Flash’s +0.10 points and 41.5%; one verifiably failing submission nevertheless receives a decisive default-setup win, while three checked-correct pairs receive changing verdicts.Source: C Additional robustness checks · pp. 12–13

D.1 Arena-Hard criteria, reasoning required (default setup)

• The default Arena-Hard criteria ask the judge to generate its own answer, compare responses against it, assess helpfulness, relevance, and concision, and consider creativity and missing information.The judge must give an explanation before one of five labeled verdicts.Source: §D.1 · p. 13

D.2 Arena-Hard criteria, label only

• The label-only Arena-Hard package removes reasoning and also omits self-generation, self-comparison, error correction, and clarification-on-ambiguity instructions.It therefore contrasts an instruction package, not only visible reasoning.Source: §D.2 · p. 14

D.3 Accuracy-prioritising criteria, reasoning required

• The accuracy-prioritising package makes correctness primary and treats helpfulness and safety as constraints, with reasoning before the verdict.The instructions prioritize accurate technical claims, code, calculations, and reasoning over a more helpful but less correct answer.Source: §D.3 · p. 14

D.4 Accuracy-prioritising criteria, label only

• The accuracy-prioritising label-only package keeps the priority statement but removes the explanation and the statement that helpfulness and safety are constraints.It returns one of the same five verdict labels.Source: §D.4 · p. 14

D.5 Scope preamble and case template

• Coding-assistance framing prepends a coding-task description to the user message, while the case retains Arena-Hard’s prompt-and-answer delimiters.The preamble asks the judge to evaluate responses with coding-task context in mind.Source: §D.5 · pp. 14–15

E Conditions and counts

• Table 2 covers eight October GPT-4.1 setups, with raw scores by run, tie rates, complete-prompt counts, and paired counts against the default setup.A prompt is complete when both answer orders yield parsed verdicts in every run of the setup.Source: Table 2 · p. 15, E Conditions and counts · p. 15
• 21.1%, 21.5%, and 20.9% of prompts change winner across default-run pairs, while paired score changes are +1.21, +0.68, and −0.53.The three default runs score 52.81, 54.16, and 53.52; the average linearly weighted κ between runs is 0.763.Source: E Conditions and counts · p. 15

F Verdict parsing

• Verdicts use the final matching bracketed label; invalid outputs are not imputed, and one invalid answer order excludes that prompt from comparisons involving the setup.The parser first seeks a double-bracketed label and falls back to a single-bracketed label.Source: F Verdict parsing · pp. 15–16

G Estimators

• Order-averaged scores combine recoded verdicts from both answer orders, and sign classes represent a baseline win, tie, or candidate win.A prompt lacking a parsed verdict in either order has no order-averaged score.Source: G Estimators · p. 16
• Boundary-change rates and paired aggregate changes use prompts complete under both setups; a direct reversal excludes movement into or out of a tie.The May excess measure subtracts the default setup’s test–retest disagreement from the cross-setup boundary-change rate.Source: G Estimators · p. 16
• Factor effects average across the four combinations of the other factors, while prompt sensitivity counts differing setups and κ uses five rebinned score tiers.For factor effects, setup scores are averaged over runs and boundary-change rates over every pairing of runs; κ treats ±0.5 as a tie, unlike the sign-class measure.Source: G Estimators · p. 16

H Threshold-based classification

• 132 prompts (26.4%) are unanimous, 136 (27.2%) are majority, and 232 (46.4%) are unresolved under the default thresholds across all 500 prompts.The default thresholds are a 0.6 majority share and a 0.4 tie share.Source: H Threshold-based classification · p. 17
• The unanimous share remains 26.4% across the threshold sweep by construction, while the unresolved share ranges from 33.6% to 54.4%.The authors therefore report threshold-free unanimity rather than treating the threshold-based partition as a natural division of prompts.Source: H Threshold-based classification · p. 17, Table 3 · p. 17
• The main text reports 131 of 496 unanimous prompts, while the appendix reports 132 of 500 because it classifies prompts after dropping missing runs.One of four prompts with an incomplete run is unanimous across the runs it has.Source: H Threshold-based classification · p. 17
• Restricting classification to five conditions leaves 17.6% unresolved in May, but this should be compared with equally sized subsets because fewer conditions are mechanically easier to satisfy.The subset comprises four GPT-4.1 conditions requiring reasoning and the Claude substitution.Source: H Threshold-based classification · p. 17

I Possible implications for uses of individual verdicts

• The appendix's implications are hypotheses, not results; they extrapolate from 21.2% of prompt-level outcomes changing on an unchanged rerun.Each proposed consequence would need its own experiment.Source: I Possible implications for uses of… · p. 17
• The measurement covers one benchmark, one near-50% model pair, one judge family, and repeats on one day with one pinned model version.The authors expect lower instability for model pairs with wider capability gaps because prompts where one model plainly wins never change.Source: I Possible implications for uses of… · p. 17
• Table 4 organizes settings that consume per-item judge outputs by what measured instability might and might not disturb.Its entries are hypotheses, not results.Source: I Possible implications for uses of… · p. 17, Table 4 · p. 18