Source-linked AI summary
There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items
V. S. Raghu Parupudi
TL;DR
Multiple-choice benchmark scores depend on an underexamined evaluation harness, not only on fixed questions and answers. The paper measures this dependence item by item with a matched fragility grid and finds that harness choices can determine score bands, pairwise gaps, and leaderboard winners.
Problem
Existing harness-sensitivity work reports aggregate score variance without identifying which items vary or whether those items separate models.
Method
The fragility grid evaluates 12 open-weight instruction-tuned models on 3,679 items under 26 defensible configurations while holding items, weights, and greedy decoding fixed.
Results
Harness choices make scores bands rather than points, place 95.7 percent of adjacent-model gaps on config-fragile items, and allow 4 of 12 models to reach rank one.
Takeaways & Limitations
Leaderboards should report harness-dependent score bands with robust accuracy, compare models on jointly robust items, disclose the full harness, and check compressed benchmarks for configuration stability.
Takeaways & Limitations
The study covers 12 open-weight instruction-tuned models and does not claim its reported magnitudes transfer unchanged to frontier proprietary models.
Abstract
from arXiv · showhide
Multiple-choice benchmarks fix the questions and the correct answers, but not the harness: the order of the options, the wording of the prompt, and whether a language model's answer is read from generated text or from per-option likelihoods. Work on this harness sensitivity reports it as aggregate score variance, leaving unexamined which items the variance falls on and whether they are the items that separate one model from the next. We treat the evaluation harness of large language models (LLMs) as an independent variable and resolve its effect to single items. We introduce the \textit{fragility grid}: 12 open-weight instruction-tuned LLMs from 4 families answer the same 3{,}679 items from 4 benchmarks (ARC, HellaSwag, MMLU, TruthfulQA) under 26 equally defensible harness configurations, recording one correctness bit for every model, item, and configuration. The comparison is matched, since the items, the weights, and the greedy decoding stay fixed while only the harness varies. Under the grid a model's score is a band rather than a point: gemma4-31b scores between 31 and 89 percent depending only on the harness. Three results follow. On the items that two adjacent models both answer stably the pair is tied, and config-fragile items carry 95.7 percent of a pair's gap on average. Four of the 12 models reach rank one under some configuration, so the harness selects the winner. Item discrimination, the property that benchmark-compression methods maximize, correlates with fragility at 0.28 (95 percent CI 0.25 to 0.30), so compression keeps the fragile items rather than removing them. The scoring choice, not the option order that protocols usually fix, is the load-bearing axis. We release the per-item records and the analysis script, from which every number regenerates on a CPU in seconds, and we position the fragility grid as a check a leaderboard can run before it reports an order.
1 Introduction
The paper argues that multiple-choice leaderboard scores depend on underreported harness choices, so aggregate variance must be resolved to the item level. Its fragility grid shows that these choices can determine score bands, pairwise gaps, champions, and which benchmark items compression retains.
- The fragility grid: 26 defensible harness configurations vary option order, prompt format, and scoring while holding models, items, weights, and greedy decoding fixed.The grid records one correctness bit for every model, item, and configuration.
- Score instability: 31 to 89 percent is gemma4-31b’s score range under different configurations with identical weights and items.Across the roster, 85 percent of credited answers can be flipped to wrong by an equally valid configuration.
- Ranking gaps: 95.7 percent of an adjacent pair’s gap is carried by config-fragile items on average.On items both models answer stably, five of 11 pairs are identical, four differ on one item, and two disagree in both directions with zero net difference.
- Winner selection: Four of 12 models reach rank one under some configuration, making harness choice sufficient to select the leaderboard winner.mixtral-8x7b moves from 11th under the reference harness to first under another configuration.
- Compression: 0.28 is the correlation between item discrimination and config-fragility, so benchmark compression retains rather than removes fragile items.The most discriminative items have mean fragility 0.96 versus 0.85 across the benchmark.
- Harness axes: 0.31 is mean accuracy robust to scoring choice, compared with 0.40 for prompt format and 0.60 for option order.The generation-versus-likelihood decision is the load-bearing harness axis.
2 Related Work
Prior work documents aggregate sensitivity to prompts, option order, and evaluation setups, while benchmark compression selects informative items under a fixed harness. This paper adds item-level resolution, showing that the retained items are precisely the fragile ones and that adjacent-model ordering is not identified by stable answers.
- Sensitivity of scores: Prompt formatting and paraphrased instructions can swing accuracy and reorder systems, with no uniformly best format reported.Prior multiple-choice work also documents sensitivity to option order, especially for uncertain items.
- Leaderboard perturbation: Leaderboard perturbation studies report rank movement, while holistic evaluation argues that one setup provides only a partial view.These approaches characterize setup-level movement rather than resolving variance to individual items.
- Benchmark compression: Benchmark compression uses item discrimination to select roughly a hundred informative items under a fixed harness.The unresolved question is whether those retained items remain stable when the harness changes.
- Paper position: The fragility grid resolves aggregate harness variance to items and connects it to gap decomposition, champion count, compression, and harness-axis comparisons.The paper does not claim that the models are equally capable or that multiple-choice benchmarks contain no capability signal.
3 The Fragility Grid
The fragility grid evaluates a matched roster of 12 instruction-tuned models on 3,679 four-option items across 26 configurations. It defines item stability and score-band measures from per-item correctness records, while controlling inference and reporting benchmark-construction boundaries.
- 3.1 Models: 12 instruction-tuned models from 4 families span 3 billion to 70 billion parameters, including dense and mixture-of-experts architectures.Two additional Mistral models were dropped after inference-engine initialization hangs.
- 3.2 Benchmarks and items: 3,679 pooled items come from ARC, HellaSwag, MMLU, and TruthfulQA, with four options and one correct answer per item.TruthfulQA contributes 679 filtered items and places the gold answer first in its native order.
- 3.3 The grid: 26 configurations combine six option orderings, six prompt/scoring formats, and one reference configuration of letter labels, original order, and generation scoring.Every configuration is intended to represent a defensible published evaluation setup rather than an adversarial harness.
- 3.4 Item measures: Each model-item pair receives a correctness bit under all 26 configurations, enabling robust, fragile, accuracy-band, and gap-based measures.Config-fragile items flip correctness; robust-correct and robust-wrong items remain correct or wrong under every configuration.
- 3.5 Controls and inference: Greedy zero-shot decoding, fixed items and weights, and seeded bootstrap and subset procedures make the comparison matched and deterministic.The per-item records and analysis script regenerate every number on a CPU in about ten seconds.
- 3.5 Controls and inference: Table 1 reports reference accuracy, lowest and highest single-configuration accuracy, spread, and config-lucky fraction over the same 3,679 items.Its spread is narrower than the robust-to-optimistic band shown in Figure 1.
4 Results
Across the 26 harness configurations, model accuracy forms wide bands, and fragile items account for nearly all adjacent-model leaderboard gaps. The harness can therefore change both pairwise orderings and the champion, with scoring the most consequential single axis and compression retaining fragile items.
- Fragile gaps: Five of 11 adjacent pairs agree on every jointly stable item, four differ on exactly one, and two disagree in both directions with zero net difference.The stable-item comparisons do not identify the leaderboard ordering.
- Fragile gaps: 95.7 percent of the nonzero adjacent-pair gap is fragile mass on average, while all 10 nonzero full-set gaps vanish on the jointly robust items.The qwen3-14b versus qwen3-32b pair is tied on the full set as well.
- Champion selection: Four of 12 models reach rank one under some configuration, including mixtral-8x7b moving from 11th under the reference harness to first.The two major champions win with substantial margins, whereas the two minor champions win by 1 and 3 items.
- Compression: 0.28 correlation: item discrimination is associated with config-fragility, and discrimination-selected 100-item subsets raise mean champion count to 5.7 from 4 on the full benchmark.Random 100-item subsets already raise the mean champion count to 5.1, separating sample-size volatility from the additional selection effect.
5 Limitations
The paper’s claims are bounded by its model roster, four-option benchmark format, selected harness grid, small robust-item samples, and single-seed greedy decoding. Several reported counts and magnitudes therefore depend on scope, benchmark mix, configuration choices, or unresolved sampling limitations.
- Roster scope: The study covers 12 instruction-tuned models from 4 families and does not claim its magnitudes transfer unchanged to frontier proprietary models.Mistral is represented by one model after two others were dropped for an engine-initialization hang.
- Format scope: The claims concern four-option multiple-choice benchmarks, not open-ended generation, code, or agentic tasks.The paper does not claim the fragility grid would decompose those other leaderboards in the same way.
- Benchmark construction: TruthfulQA’s constructed option order makes the gold answer first, so the four-champion count is specific to the evaluated benchmark mix.Removing TruthfulQA lowers the champion count from 4 to 3, while the paper does not claim that count is invariant to benchmark mix.
- Champion margins: Two champions win only one configuration each by 1 and 3 items, so the champion count is not four equally robust outcomes.The paper counts these models because they attain rank one under defensible configurations, while reporting every margin separately.
- Grid scope: The 26-configuration grid is a defensible judgment rather than the only possible boundary, and excluding likelihood scoring would reduce the measured effect.The authors disclose the grid so the analysis can be redrawn under a different definition of defensible harnesses.
- Robust-item evidence: The robust-item tie rests on only 72 to 336 jointly robust items per pair, and a larger robust set could separate some pairs.The bootstrap interval is consistent with the tie claim but is not independent evidence for it.
- Decoding: Greedy single-seed decoding excludes sampling variance, so the paper shows that harness variance alone can manufacture ordering without comparing the two variance sources.It does not claim harness variance is larger than sampling variance.
- Compression: The discrimination-selected 100-item subset is not uniquely defined because 64 slots are determined by tie-breaking, producing champion counts from 4 to 7.The reported mean is 5.7 across seeded tie-breaks rather than a precise single count.
6 Future Work
Future work can make fragility screening cheaper and separate answer competence from compliance effects in the scoring axis.
- Future Work: Per-item option-margin records could support a flip predictor that flags fragile items before running a full configuration sweep.Separating wrong answers from unparseable answers could also decompose the scoring axis into competence and compliance terms.
7 Conclusion
The paper argues that multiple-choice leaderboard scores measure models under a harness, not models alone. Across defensible configurations, stable items tie adjacent models, fragile items carry the gaps, and the harness can select the champion.
- 7 Conclusion: A multiple-choice leaderboard measures models under a harness, whose choices contribute more to the result than the reported number admits.The conclusion frames harness variation as part of the measurement rather than an incidental source of noise.
- 7 Conclusion: Config-fragile items carry 95.7 percent of the mean gap across all 10 adjacent pairs with nonzero full-set gaps.The same conclusion reports that four of 12 models can reach rank one and that swapping labels can flip the leading winner.
Code and Data Availability
The release provides the records and code needed to reproduce the paper’s analyses and figures locally.
- Code and Data Availability: The release contains per-item correctness records, the configuration legend, analysis scripts with unit tests, and figure-rendering code.It includes 48 files with 26 configuration bits for every item and model.
- Code and Data Availability: Every number and figure regenerates on a CPU in about ten seconds without a GPU or model download.Two runs produce byte-identical output.
A Reproducibility
The appendix makes the paper's results reproducible from released records, a fixed seed, and documented analysis choices.
- Every reported number is a deterministic function of the released per-item records and seed 1234.The appendix documents item selection, configurations, decoding parameters, and every randomized analysis step.
A.1 Released artifacts
The release provides per-item JSONL records containing benchmark metadata, gold answers, correctness across all configurations, and extension inputs.
- The release contains 48 JSONL record files, one for each benchmark-model pair.Each line identifies the benchmark item, question, gold index, and correctness bits for 26 configuration keys.
A.2 Item selection
The evaluation uses one shared, seeded item set across all 12 models, with benchmark-specific filtering and exclusions to preserve comparability.
- Seed 1234 selects the identical item set for all 12 models.MMLU contributes 1,005 shuffled test items, while ARC, HellaSwag, and TruthfulQA use shuffled four-option pools.
- The first 5 items of each shuffled benchmark are excluded as a reserved few-shot pool, although evaluation is zero-shot.The exclusion keeps the item set aligned with a possible few-shot version of the evaluation.
- ARC retains only exactly four-choice questions, while TruthfulQA is assembled from one correct answer followed by its first three distractors.TruthfulQA therefore places the gold answer first in its constructed option order.
A.3 The 26 configurations
The 26-configuration grid crosses scoring method, prompt template, and option ordering while holding inference deterministic; the resulting winners and compression statistics reveal harness-sensitive rankings.
- The 26 configurations: 26 configurations combine 24 generation settings with 2 order-invariant likelihood-cloze settings.The generation settings cross four templates with six option orderings; the reference uses letter_plain with identity ordering.
- The 26 configurations: Option orderings permute display slots, then map predictions back to original indices before scoring.The six orderings include the identity permutation and five fixed non-identity permutations sampled under seed 99.
- The 26 configurations: The four generation templates are crossed with six orderings, whereas cloze stems omit labels and cannot vary by option order.This makes scoring method and prompt format the active axes for the two likelihood configurations.
- Deterministic inference: Greedy generation uses temperature 0, top-p 1.0, seed 1234, and an 8-token cap, making correctness deterministic for fixed weights and inputs.The pipeline's stochasticity instead enters through documented analysis steps such as bootstrapping, tie-breaks, and subset sampling.
- Compression analysis: Item discrimination is the point-biserial correlation between reference correctness across 12 models and their reference accuracies.Its full-set discrimination-fragility correlation is seed-invariant at 0.2767, while bootstrap variation affects only the interval.
- Compression analysis: The discriminative subset yields more champions than the random subset under every seed, with a seed-1234 mean of 5.67 versus a full-benchmark count of 4.The discriminative-subset mean ranges from 5.59 to 5.72 across seeds, while the random baseline ranges from 4.80 to 5.08.
- Champion margins: Gemma4-31b wins 16 configurations by a median 181.5-item margin, while qwen3-14b wins 8 by 98.5 items.Plain-letter and parenthesized-letter templates favor gemma4-31b, digit-labeled templates favor qwen3-14b, and instruction-phrased templates split by order.
- Champion margins: Likelihood scoring produces close calls: llama3.3-70b wins one configuration by 1 item and mixtral-8x7b wins the other by 3.The paper treats both as valid champions but does not interpret their ordering as a finding.