PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?

Qiaoyuan Zheng, Yiqu Yang

arXiv:2609.00482v1cs.CLcs.LG

TL;DR

Small leaderboard gaps do not establish that model orderings are robust to benchmark item composition. The paper uses owner-disjoint, family-DIF-guided low-DIF scoring with matched-random controls; aggregate rankings remain stable, yet four of five benchmarks show excess reversals among near-tied pairs.

  • Problem

    Small leaderboard gaps do not establish whether the implied model ordering is robust to which benchmark items are included.

    Source: §1 · p. 1

  • Method

    The study uses owner-disjoint folds to select source- and easiness-balanced low-DIF anchors and compares their scores with equally short matched-random subtests.

    Source: §1 · p. 1

  • Results

    Four benchmarks show 30.9–47.1% near-tie reversals, exceeding matched-random subtests by 16.9–28.6 percentage points (all matched-random p = .001); WinoGrande’s excess is −0.9 points (p = .689).

    Source: §5.1 · p. 6

  • Takeaways & Limitations

    Sub-one-point leaderboard gaps should be accompanied by composition-robustness checks rather than treated as self-interpreting evidence of model superiority.

    Source: §7 · p. 8

    The five benchmarks are static, primarily closed-form, and drawn from one response collection; MMLU and MMLU-Pro share content lineage and are not independent replications.

    Source: §6 · p. 8

Abstract

from arXiv · show

Small leaderboard gaps are often interpreted as evidence that one language model is better than another, but their sign may depend on which benchmark items are included. We test this using item-level responses from five benchmarks and a family-label-free spectral approximation to multidimensional item-response theory (MIRT). In owner-disjoint folds, one owner half identifies items with low residual differential item functioning across model families (low-DIF); the resulting frozen, source- and easiness-balanced weights score models in the other half, while equally short matched-random subtests control for generic subtest variation. Full-benchmark and low-DIF rankings remain strongly correlated ($\tau_b=.900$--$.948$). Yet in four of five benchmarks, 30.9--47.1\% of cross-family pairs initially within one percentage point reverse order, exceeding their matched-random medians by 16.9--28.6 percentage points (all $p=.001$). The fifth benchmark shows no reliable excess ($-0.9$ points, $p=.689$). The pattern survives all pre-specified population perturbations, and residual item--family signatures replicate across owner halves; however, no family shows a consistent advantage across benchmarks. Thus, globally stable rankings can still leave individual near-tie orderings sensitive to benchmark composition, and sub-one-point leaderboard gaps should be accompanied by evidence that the implied ordering is composition-robust.

1 Introduction

Small leaderboard gaps do not establish that model orderings are robust to benchmark item composition. This study tests that sensitivity using family-DIF-guided scoring and finds excess near-tie reversals in four of five benchmarks despite stable aggregate rankings.

1 Introduction

  • The study asks whether near-tied model orderings remain robust when the included benchmark items change.Small leaderboard gaps do not establish that the implied ordering is robust to item composition.Source: §1 · p. 1
  • Owner-disjoint folds use family-DIF to select source- and easiness-balanced low-DIF anchors, with matched-random subtests controlling for generic shorter-subtest variation.Anchor weights learned in one owner half score the other half, after which the halves are swapped.Source: §1 · p. 1
  • Four of five benchmarks show excess near-tie reversals over matched-random controls, despite stable aggregate rankings and no consistent family advantage.The study treats family-DIF as a diagnostic rather than a causal or intrinsic family property.Source: §1 · p. 1

2 Related work

Prior work applies IRT and DIF analyses to language-model evaluation, while related studies document ranking sensitivity and measurement-validity concerns. This paper distinguishes its focus on owner-disjoint cross-family near-tie reversals from those adjacent aims.

2 Related work

  • Prior IRT work supports NLP evaluation scales, benchmark discrimination, item validation, evaluation-cost reduction, routing, and benchmark prediction, whereas these anchors diagnose composition sensitivity.The paper contrasts its spectral representation with methods for predicting performance, linking test forms, or detecting mislabeled items.Source: §2 · p. 2
  • A BBH course report found DIF-selected pools altered held-out Llama–Qwen group gaps; this study extends the setting to owner-disjoint rankings across five benchmarks and matched-random controls.It also examines individual near-tied pairs and source- and easiness-matched random controls.Source: §2 · p. 2
  • Related ranking studies examine sensitivity to weighting, prompts, protocols, output uncertainty, and statistical resolution; this paper instead estimates excess cross-family reversals among near-tied models against matched random subtests.The estimand concerns pairs initially within one percentage point and controls for subtest length and composition.Source: §2 · p. 2
  • Benchmark validity and apparent DIF can reflect multidimensional capability structure or intended subdomains, so the paper separates ranking consequences from owner-disjoint stability and semantic interpretation.Its residual effects remain conditional on a family-blind, prediction-selected spectral response representation.Source: §2 · p. 2

3 Audit design

The audit uses item-level benchmark responses and owner-disjoint cross-fitting to compare full-benchmark scores with balanced low-DIF anchors and matched-random subtests. Its design separates residual family dependence from multidimensional response structure and generic variation from shorter tests.

3 Audit design

  • The audit defines composition robustness as whether near-tied rankings survive low-DIF reweighting, using blueprint-matched random subtests to control ordinary short-subtest variation.Frozen weights learned in one owner half are applied to the other half.Source: §3 · p. 2

3.1 Data and evaluation population

  • The study analyzes item-level responses from five benchmarks and caps owner checkpoints before splitting owners into approximately equal halves.Ranking comparisons retain one checkpoint per owner–family pair, selected nearest that pair’s median full-benchmark accuracy.Source: §3.1 · pp. 2–3

3.2 Family DIF with a spectral MIRT approximation

  • A family-label-free spectral approximation adjusts for multidimensional item–model response structure before residual family effects are estimated.It is a response-derived working model for predictive nuisance adjustment and repeated held-out fitting, not a claim of superiority over likelihood-based MIRT.Source: §3.2 · p. 3

3.3 Cross-fitted residual low-DIF anchors

  • Owner-disjoint folds learn spectral directions, residual DIF, and anchor weights on one half, then score the other; three purification rounds retain the lowest-DIF items within source-by-easiness cells.The primary analysis retains 50% of items per cell, preserving each cell’s total benchmark weight; these diagnostic anchors are not scale-linking anchors.Source: §3.3 · pp. 3–4

3.4 Ranking estimand and matched control

  • The primary estimand is strict reversal among different-owner, different-family pairs within one percentage point, compared with equally short random subtests matched by source and easiness.The random controls do not select items by DIF, and the reported excess is the low-DIF reversal rate minus the random rate.Source: §3.4 · pp. 4–5

4 Validation and Diagnostic Analyses

Validation analyses probe population sensitivity, cross-half replication of residual item–family signatures, and broad content patterns. The content audit pairs stable high-DIF items with matched controls and applies prespecified confirmatory criteria to annotator labels.

4 Validation and Diagnostic Analyses

  • Validation tests whether the ranking effect survives population changes, replicates across owner halves, and aligns with broad content categories.The analyses span population robustness, item-signature replication and source attribution, and a blinded content audit.Source: §4 · p. 5

4.1 Population robustness

  • Nine population perturbations vary owner caps, checkpoint choice, lineage filtering, or family composition while preserving owner-disjoint cross-fitting and refitting most analysis stages.The stated robustness criteria require positive excess reversals in at least three benchmarks and positive matched-random p ≤.05 results in at least three.Source: §4.1 · p. 5

4.2 Item-signature replication and source attribution

  • Cross-half replication compares family-wise signature correlations, high-DIF overlap, and advantaged-family agreement using within-cell permutations; secondary analyses test source-effect replication and sign retention.For MMLU-Pro, BBH, and MMLU, the secondary analyses examine discovery-selected source effects and source contributions across folds.Source: §4.2 · p. 5

4.3 Blinded content audit

  • The blinded audit pairs 50 stable high-DIF items per benchmark with same-cell controls and uses two annotators to label ten binary and three ordinal content axes.A binary axis is confirmatory only with concordant pooled direction, BH q ≤.05, an absolute paired difference of at least eight points, and the same direction in three or more benchmarks.Source: §4.3 · p. 5

5 Results

Full-benchmark and low-DIF rankings are strongly correlated, yet near-tied orderings reverse more often than matched-random controls in four of five benchmarks. The pattern persists across wider score bands and population perturbations; residual item-family signatures replicate, but the semantic audit finds no confirmatory enrichment.

5.1 Global rankings are stable, but near-tie orderings are composition-sensitive

  • Four benchmarks show near-tie reversal rates of 30.9–47.1%, exceeding matched-random subtests by 16.9–28.6 percentage points (all p = .001), despite τb = .900–.948 overall ranking correlations.Only 2.5–4.2% of all cross-family comparisons reverse.Source: §5.1 · p. 6
  • WinoGrande shows no reliable excess: low-DIF and matched-random reversal rates are 33.8% and 34.7%, respectively (excess −0.9 points, p = .689).The observed sensitivity is localized to near ties in four benchmarks, not wholesale leaderboard reranking.Source: §5.1 · p. 6

5.2 Excess reversals persist across wider score bands

  • At a five-point score-gap threshold, all four primary-positive benchmarks still exceed matched-random controls by 5.2–19.1 percentage points; WinoGrande has no significant excess at any tested threshold.The one-percentage-point threshold is pre-specified; results outside it are descriptive sensitivity analyses.Source: §5.2 · p. 6, Figure 1 · p. 7

5.3 The result is robust to population specification

  • Across the baseline and nine population perturbations, MMLU-Pro, BBH, MMLU, and HellaSwag consistently show positive excess reversals with p ≤ .05; WinoGrande reaches that threshold only when Mistral/Mixtral models are excluded.Perturbations include owner caps, score-blind checkpoint selection, conservative lineage filtering, and leave-one-family-out analyses.Source: §5.3 · p. 6, Figure 2 · p. 8
  • Every family changes the direction of its mean anchor-minus-full score shift at least once, including in exact-common-model comparisons, so shifts are benchmark-dependent rather than intrinsic family rankings.The exact-common-model comparisons hold the evaluated model population fixed.Source: §5.3 · p. 6

5.4 Residual item signatures replicate across owners

  • Residual item-family signatures replicate above chance across owner halves on all benchmarks: median family-wise correlations are ρ = .308–.589, top-20% overlap is 31.5–38.5% versus a 20% independence baseline, and advantaged-family recovery is 44.3–72.5%.All 15 within-cell permutation tests give p = .002, Bonferroni-adjusted p = .030.Source: §5.4 · pp. 6–7, Table 2 · p. 7
  • Among three source-rich benchmarks, discovery-selected source-effect signs replicate in 58 of 60 held-out checks, while exact source contributions retain their signs in 43 of 60; these results do not identify a cognitive or training-data mechanism.Both pooled exact tests have p < .001.Source: §5.4 · p. 7

5.5 The blinded audit finds no confirmatory semantic enrichment

  • None of the ten pre-specified binary content axes meets all dual-annotator confirmatory criteria, and no exploratory family-direction association survives BH correction in either annotator.Agreement is κ = .715 on binary axes and ρ = .800 on ordinal axes; the largest reported effect is not significant after correction or replicated across annotators.Source: §5.5 · p. 7

6 Limitations and claim boundaries

The study’s observational family grouping and benchmark scope constrain what its results can establish. Its findings do not identify a causal family mechanism or extend to future items, independent benchmark replications, or claims about fairness.

6 Limitations and claim boundaries

  • Model-family labels inferred from model identifiers may conflate architecture, training, data, and uploader practices.Owner-disjoint folds and population perturbations reduce specific artifacts but do not identify a causal family mechanism.Source: §6 · p. 8
  • The five static, primarily closed-form benchmarks come from one response collection, and MMLU and MMLU-Pro share content lineage.Interactions between family grouping and other evaluation choices remain untested.Source: §6 · p. 8
  • Low-DIF anchors reweight observed items; they neither estimate future-item performance nor define a superior construct or item-removal rule.The one-point band is operational, and DIF does not imply unfairness.Source: §6 · p. 8
  • The linear spectral MIRT approximation may leave nonlinear, higher-dimensional, or unmeasured capability differences in residual effects.The dimension-selection ceiling and a two-annotator audit using broad content axes also limit interpretation; results establish conditional item–family interactions and ranking consequences, not intrinsic capabilities, contamination, or broad leaderboard invalidity.Source: §6 · p. 8

7 Conclusion

The conclusion finds that aggregate rankings can remain stable while individual near-tied orderings change with benchmark composition. The appendices examine related-work coverage, alternative score-gap thresholds, and anchor-retention settings.

7 Conclusion

  • Four of five benchmarks show excess near-tie reversals under low-DIF scoring, while the fifth does not exceed matched-random variation.Owner-disjoint estimation, matched controls, and population robustness localize sensitivity to near-tied comparisons, not a universal family advantage or broad leaderboard invalidity.Source: §7 · p. 8
  • Sub-one-point leaderboard gaps should be accompanied by composition-robustness evidence, rather than treated as self-interpreting evidence of model superiority.The study asks whether cross-family orderings separated by at most one point remain robust to benchmark item composition.Source: §7 · p. 8

A Comparison with the closest related work

B.1 All score-gap thresholds

  • At the 0.25-point threshold, MMLU-Pro has 89 eligible pairs, and only BBH, MMLU, and HellaSwag pass the matched-random test.Results outside the frozen one-point threshold are sensitivity analyses, not additional confirmatory tests.Source: §B.1 · p. 12, Table 4 · p. 12

B.2 Anchor-fraction sensitivity

  • Retaining fewer low-DIF items produces more ranking changes, while near-tie reversals remain substantial across all three tested fractions.Matched-random controls were computed only for the primary 50% specification, so confirmatory excess-reversal inference remains attached to that setting.Source: §B.2 · pp. 12–13, Table 5 · p. 13

B.3 Family/owner robustness matrix

All ten frozen owner/family specifications pass the stated direction and significance criteria. The appendix also describes exact-common-model benchmark comparisons as illustrative, descriptive checks rather than causal family rankings.

B.3 Family/owner robustness matrix

  • All ten specifications pass both robustness criteria; MMLU-Pro, BBH, MMLU, and HellaSwag remain positive and nominally significant in every specification.WinoGrande is significant only when Mistral/Mixtral is omitted, making it an unstable boundary case rather than a robust positive result.Source: §B.3 · p. 13
  • Table 6 reports excess one-point reversal rates for the primary specification and nine frozen owner/family variants, with stars marking nominal p ≤ .05.The supplied passages do not provide the table’s individual values.Source: §B.3 · p. 13

B.4 Exact-common-model directional checks

  • The three exact-common-model examples hold each family-specific model population fixed across benchmark pairs, ruling out membership changes as the explanation for directional shifts.They are illustrative rather than an exhaustive search, and the shifts are descriptive—not intrinsic or causal family rankings.Source: Table 7 · p. 13, §B.4 · p. 14

B.5 Blinded content-audit results

The blinded content audit finds no binary content axis meeting the dual-annotator confirmatory criteria. The appendix describes the owner-disjoint audit, spectral approximation, anchor construction, matched-random controls, and composition bootstrap.

B.5 Blinded content-audit results

  • Annotation reliability passes the frozen criteria, with median Cohen’s κ of .715 on non-degenerate binary axes and median Spearman ρ of .800 on ordinal axes.The audit uses 250 exact-cell matched pairs and two blinded annotators.Source: §B.5 · p. 14
  • No binary axis passes the dual-annotator confirmatory criteria: annotator A’s spatial/temporal estimate is −9.6 points, while annotator B’s is −0.8 points.For that axis, the reported corrected q-values are .106 and .902, respectively.Source: §B.5 · p. 14, Figure 3 · p. 15
  • In the exploratory diagnostic, no benchmark-stratified family-direction association survives BH correction, and no dual-annotator signal is detected.The diagnostic covers 158 high-residual-DIF items with family-direction agreement across owner halves and is not used as a semantic explanation.Source: §B.5 · p. 14

C.1 Model-family and owner construction

  • Models are assigned to families by unique name-pattern matches, owners are identified by uploader, and no owner appears in both outer halves.Models matching zero or multiple family patterns are excluded; owners contributing more than five checkpoints per family are capped at five.Source: §C.1 · p. 14

C.2 Spectral working-response construction; C.3 Nested selection of the MIRT dimension

  • The spectral MIRT working model approximates a multidimensional logistic response surface rather than fully calibrating a traditional 2PL MIRT model.One-SE dimension selection stays below the feasible candidate ceiling in every fold, while raw held-out-loss minima reach that ceiling in five of ten folds.Source: §C.2 · p. 16, §C.3 · p. 16

C.5 Anchor purification algorithm

  • Anchor purification runs for three fixed rounds, and ranking scores use the weights selected at the end of round three.Successive Jaccard overlap is recorded but does not determine when purification stops.Source: §C.5 · p. 18

C.6 Matched-random subtests

  • Matched-random controls sample items without replacement and assign sampled items weights matching the anchor cell’s item mass.The random-control replicate index is pooled across the two outer target halves.Source: §C.6 · p. 18

C.7 Composition bootstrap

  • Composition bootstraps use native sources as units when there are at least five; otherwise, items are divided into 100 contiguous, near-equal clusters.The close-pair mask is defined from observed full scores and held fixed across 2,000 bootstrap replicates.Source: §C.7 · p. 18

C.8 Implementation settings

The appendix specifies the primary audit settings, near-tie definition, matched-random controls, and uncertainty procedures. Sensitivity analyses vary the anchor fraction and score-gap threshold.

C.8 Implementation settings

  • The primary audit caps each owner–family at five checkpoints, requires 20 models per family, and uses owner-disjoint two-fold cross-fitting.The primary close-pair band is one percentage point, with 50% anchors.Source: §C.8 · p. 19
  • MIRT candidates range from 1 to 256 dimensions; selection uses the minimum K within one standard error of best loss, with benchmark- and fold-specific selected dimensions.Selected K values are 64/32 for MMLU-Pro, 128/128 for BBH and MMLU, 16/64 for HellaSwag, and 8/8 for WinoGrande.Source: §C.8 · p. 19

D.1 Controls and uncertainty

  • Sensitivity analyses vary the anchor fraction to 30% or 70% and the full-score near-tie threshold from 0.25 to 5 percentage points.Pair membership is frozen from observed full-benchmark scores at each threshold.Source: §C.8 · p. 19, §D.1 · p. 20
  • Eligible pairs differ in owner and family, are within the full-benchmark gap threshold, and are untied under either scoring rule; pooled reversal rates combine both cross-fitting directions.The primary threshold is one percentage point.Source: §D.1 · p. 19
  • Matched-random comparisons use 1,000 replicates, composition uncertainty uses 2,000 bootstrap replicates, and family-specific shift intervals resample owners.Owner-level resampling accounts for correlated shifts among checkpoints from the same owner.Source: §D.1 · p. 20

D.2 Population robustness

The appendix tests population sensitivity, cross-half item-signature replication, source attribution, and blinded content explanations. All 15 observed item-signature statistics exceed their corresponding permutation values.

D.2 Population robustness

  • Population perturbations change owner checkpoint caps, representative selection, lineage filters, or omit each of the five model families in turn.Perturbed populations retain at least 15 models per family and reconstruct owner-disjoint halves.Source: §D.2 · p. 20
  • Robustness refits the MIRT representation, residual-DIF model, and anchor purification while fixing K and retaining the primary 50% anchor fraction and one-point threshold.New perturbations use 200 matched-random replicates; K is not reselected.Source: §D.2 · p. 20

D.3 Item-signature replication and source attribution

  • Cross-half replication uses common source-by-easiness cells and evaluates family-effect rank correlations, overlap among top-20% DIF items, and agreement on the advantaged family.The same cell boundaries in both halves prevent stability from arising through independently changed stratification boundaries.Source: §D.3 · p. 21
  • All 15 item-signature statistics across five benchmarks and three measures exceed all 500 permutation values (pperm = .002); Bonferroni adjustment gives padj = .030.The permutation null preserves common blueprint cells.Source: §D.3 · p. 22
  • Source-level analyses cover MMLU-Pro, BBH, and MMLU; source-effect signs are checked across halves, while exact score-shift decomposition is secondary and does not establish item-signature replication.Pooled sign support requires at least 70% replication with a one-sided exact binomial p ≤ 0.05.Source: §D.3 · p. 22

D.4 Blinded content-audit protocol and statistical tests

  • The content audit matches 50 stable high-DIF items to controls from the same source-by-easiness cell in each benchmark, and annotators see blinded question text without family or DIF information.This yields 50 matched pairs per benchmark and 500 items total.Source: §D.4 · p. 22
  • Two annotators independently label all 500 items on ten binary content axes and three ordinal dimensions; their annotations are analyzed separately.The annotators use frozen protocols and do not solve items or predict model correctness.Source: §D.4 · p. 23
  • A confirmatory binary content explanation requires matching nonzero directions, adjusted q ≤ 0.05 and effect sizes ≥ 0.08 for both annotators, plus replication across at least three benchmarks.A later exploratory analysis finds the same advantaged-family label for 158 of 250 stable high-DIF items; it is not part of the confirmatory criteria.Source: §D.4 · p. 23

E Reproducibility and artifact boundary

The anonymized artifact includes materials for reproducing and validating the analyses.

E Reproducibility and artifact boundary

Loading 2609.00482v1…