PaperlayerView exact revision on arXiv

Source-linked AI summary

How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?

Chen Yang, Jun Chen

arXiv:2609.28177v2stat.MLcs.LG

TL;DR

Leaderboard claims may reflect selecting the best of privately evaluated variants, although their number and dependence are unknown. The paper derives Gaussian-model sensitivity curves for hidden selection. Its audit finds that most adjacent-rank claims lack statistical support before selection, while surviving claims can depend on the assumed score-matched correlation.

  • Problem

    The number and dependence of privately evaluated variants behind a published leaderboard score are unknown, leaving the effect of hidden selection on claims of advantage unresolved.

    Source: §1 · p. 1

  • Method

    The paper derives a Gaussian-model sensitivity curve giving the maximum hidden variant count consistent with certification as a function of a lower bound on within-family correlation.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    391 of 394 adjacent-rank claims lack statistical support under item resampling before selection is accounted for.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The curves make hidden-family correlation assumptions explicit without estimating the unobserved search size.

    Source: §1 · p. 1, §6 · p. 9

    Public-board verdicts remain conditional on a fixed candidate family and assumptions about its correlation; private candidate logs are unavailable, and adaptive candidate generation from benchmark feedback is outside the guarantee.

    Source: §2 · p. 3, §6 · p. 9

Abstract

from arXiv · show

LLM leaderboard gains can reflect selection among privately evaluated model variants, yet neither the number of variants nor their dependence is public. We ask how many hidden variants a published margin can support while retaining statistical evidence of a provider's advantage over a fixed comparator. For a fixed candidate family under a Gaussian margin model, we derive a sensitivity curve that reports this maximum count as a function of a lower bound on within-family correlation. The relevant correlation must match the score used for ranking and the sampling model: in a controlled family, pooled item correlation is 0.90, whereas composite-score correlation is 0.46 under item resampling and 0.92 when MMLU subjects are resampled. An item-based audit of 394 adjacent-rank claims on the Open LLM Leaderboard finds that 391 lack statistical support even before accounting for selection. Among claims that pass the uncorrected test, certification can depend on assumptions about the hidden family's correlation. The resulting curves make these assumptions explicit without estimating the unobserved search size.

1 INTRODUCTION

Leaderboard margins are often treated as if a submitted model were evaluated in isolation, although providers may select it from privately scored variants. The paper derives conditional sensitivity curves and audits how hidden selection and score-matched correlation affect claims of advantage.

1 INTRODUCTION

  • The audit evaluates a provider’s advantage over a fixed public comparator, not whether the leaderboard’s complete ranking is correct.The work also measures sibling correlations, checks calibration, and provides an implementation and validation suite.Source: §1 · pp. 1–2
  • The sensitivity curve reports the largest hidden family size for which a published margin remains certified under an assumed lower bound on sibling correlation.The hidden variant count is unobserved, so the output is a curve rather than an estimate of the provider’s search size.Source: §1 · p. 1
  • At α = .05 and comparator correlation ρb = .5, 27 correlated variants have the same critical value as 5–19 independent tries across illustrative correlation scenarios.Exact correction changes verdicts relative to both the naive test and Bonferroni on the paper’s numerical grid.Source: §1 · p. 2
  • In a controlled Qwen2.5 family, pooled item correlation is 0.90, while score-matched correlation is 0.46 under item resampling and 0.92 under MMLU-subject resampling.The correlation used for correction must match the ranked score and sampling model.Source: §1 · p. 2
  • 391 of 394 adjacent-rank claims lack statistical support even before hidden selection is accounted for.Only three exceed the uncorrected threshold under item resampling.Source: §1 · p. 2

2 HIDDEN SELECTION: SETTING AND SCOPE

The audit models a public leaderboard margin as the best score selected from a private family of variants against a fixed comparator. Its verdicts concern whether some variant beats that comparator, subject to the stated model and assumptions.

2 HIDDEN SELECTION: SETTING AND SCOPE

  • The provider submits its highest-scoring sibling, so the displayed margin is a maximum over variants whose count and within-family correlation remain private.The audit uses per-item correctness for both the submitted model and public comparator.Source: §2 · p. 2
  • The score model separates shared item difficulty, a family component shared by siblings, and idiosyncratic noise; shared item difficulty cancels in margins.The Gaussian model gives sibling scores correlation ρw and variant-comparator scores correlation ρb.Source: §2 · pp. 2–3
  • The testable null is that no provider variant truly beats the comparator; rejection supports that some variant does, not necessarily the submitted one.The same critical value provides an unconditional guarantee against certifying a submitted model that is not truly better.Source: §2 · p. 3
  • Public-board verdicts are conditional because private candidate logs do not establish applicability, and candidates generated adaptively from benchmark feedback are outside the guarantee.When the correlation domain check fails, only Bonferroni applies; this is reported as model-insufficient.Source: §2 · p. 3

3 FROM A MARGIN TO A HIDDEN-SELECTION SENSITIVITY CURVE

The paper derives a Gaussian reference tail for the maximum selected margin and inverts it into a per-claim sensitivity curve. The curve’s validity depends on correlation, model-feasibility, and nuisance-estimation assumptions.

3 FROM A MARGIN TO A HIDDEN-SELECTION SENSITIVITY CURVE

  • At the least-favorable null, every sibling is as good as the comparator, maximizing the chance that the selected maximum exceeds the observed margin.The exchangeable reference law has a one-factor representation: conditional on shared variation, the margins have independent residual noise.Source: §3 · p. 3
  • At the clone boundary, the correction reduces to the single-test law, where the hidden multiplicity is unidentifiable but irrelevant.The curve’s exact correlation domain is constrained by the margin model’s feasibility conditions.Source: §3 · p. 4
  • The certificate curve gives the largest hidden multiplicity for which an observed standardized margin remains certified at a specified within-family correlation floor.For example, a curve value of 17 at floor 0.56 would not certify 27 hidden variants.Source: §3 · p. 4
  • For heterogeneous families, the minimum pairwise sibling correlation—not the mean—provides the equicorrelated upper bound on the selected tail.The tail is nonincreasing in within-family correlation; the reported floor is a population assumption, not a universal constant.Source: §3 · p. 4
  • The reference tail is exact with known nuisances, while heterogeneous families receive a conservative bound and estimated-nuisance adjustments are pointwise-in-law asymptotic for fixed K.A reported budget of ≥100 is a reporting cap, not a finite-sample accuracy guarantee.Source: §3 · p. 5
  • At k = 27, α = .05, and ρb = .5, the independent-equivalent count ranges from 5 to 19 across illustrative scenarios.Higher within-family correlation lowers the exact threshold; 27 is a sensitivity reference, not an estimate of a provider’s search.Source: §3 · p. 5

4 CORRELATION MUST MATCH THE RANKED SCORE

Correlation for selection analysis must match the score being ranked and the sampling model. Under item resampling, pooled item correlations can substantially exceed score-matched correlations, while subject resampling can change the latter.

Two correlation estimands

  • For an equal-weight benchmark mean under item resampling, each benchmark’s score-variance contribution scales with 1/nb, unlike pooled-item weighting by item share.The two correlation estimands can differ in magnitude and family ordering; the sensitivity curves therefore use the score-matched estimator.Source: §4 · p. 6

Variance decomposition

  • In five Qwen2.5-1.5B full-FT variants, GSM8K is 8.6% of items but contributes about 91% of two-task score variance; pooled and score-matched medians are 0.894 and 0.459.Figure 2 shows pooled-item and ranked-score correlations for the worked family and across six controlled strata.Source: §4 · p. 6, Figure 2 · p. 6

Data and estimator

  • Across 12 observational families, score-matched means have median 0.707 and the minimum family minimum is 0.565, but these measurements do not establish a hidden-family floor.The item-bootstrap confidence intervals exclude subject dependence and benchmark-construction uncertainty.Source: §4 · pp. 6–7

Controlled strata

  • Across six controlled strata, pooled medians are 0.73–0.90, while score-matched medians are 0.42–0.58; the DPO family’s score-matched median is 0.81.The DPO family illustrates that one update rule cannot calibrate another’s correlation.Source: §4 · p. 7

Changing the sampling unit

  • Resampling MMLU subjects as blocks changes the worked family’s score-matched median from 0.459 to 0.921.For v1 #1, the plug-in required floor at k = 27 rises from 0.689 under item resampling to 0.861 under subject-cluster standardization.Source: §4 · p. 7

Alternative explanations

  • Controls support a pooled–score difference, but filtering changes the candidate pool and observed minimum and cannot establish a universal hidden-family floor.The controls use the combined ten-variant full-FT/LoRA pool, whose median is 0.294 rather than 0.459 for the five full-FT variants.Source: §4 · p. 7

5 AUDITING PUBLIC LEADERBOARDS

The audit evaluates published provider–comparator pairs conditionally, correcting for hidden within-family selection rather than the full ranking process. Most adjacent-rank claims lack uncorrected statistical support; verdicts for the few remaining claims depend on stated correlation and model assumptions.

Protocol

  • Only 3 of 394 v1 adjacent cross-provider claims pass the naive test; Gaussian plug-in certifies 0, while Bonferroni certifies 1.The Gaussian column withdraws support from the three naive certifications: two become not margin-certified and one model-insufficient.Source: §5 · p. 8

Top claims (Table 1)

  • The five top-claim rows show distinct outcomes: v1 #1 is k-sensitive, #2 is Bonferroni-certified, and #3 is refused by every correction at k ≥3.On v2, top-five #4 certifies throughout, while #1 is refused or model-insufficient at every adjustment.Source: §5 · p. 8

The rank-1 claim

  • At k = 27, v1 #1 requires a minimum-pairwise correlation floor of 0.849 under joint adjustment, versus 0.689 under plug-in.At primary floors 0.42 and 0.56, no v1 claim is Gaussian-certified under any adjustment.Source: §5 · p. 8

Measured-extreme stress test

  • For v1 #1, adding measured β+ excess tightens plug-in budgets at floors 0.42/0.56/0.71/0.86 to 11/13/18/66; adding R = 1.16 leaves budgets 12/16/43 at 0.56/0.71/0.86.The combined adjustments yield a joint budget of eight at the 0.56 floor.Source: §5 · pp. 8–9

Assumption sensitivity

  • The sensitivity curves use independent item resampling, with support conditional on the stated correlation floor and model assumptions; adjusted curves are reported only for K ≤100.Figure 3 plots required floors at k = 27 and boundaries with nuisance caps recomputed at each k.Source: Figure 3 · p. 9
  • A separate stress test calibrated to observed extremes raises the floor to 0.939, so none of the measured sensitivity anchors certifies the claim.The reported p-value is 0.071 even at 0.86.Source: §5 · p. 9

6 DISCUSSION AND LIMITATIONS

The certificate makes hidden-family assumptions explicit without estimating them, but its results remain conditional on the model and sampling scope. The paper proposes publishing candidate-set information and using prospective methods to address selection.

What the certificate reports

  • The sensitivity curve makes hidden-family assumptions explicit without claiming to estimate them, while ordinary sampling uncertainty already limits most dense-leaderboard comparisons.The Gaussian reference tail is exact with known nuisances; heterogeneous families receive a conservative bound, while estimated-input audits use plug-in diagnostics or fixed-K asymptotic adjustments.Source: §6 · p. 9

Assumptions and scope

  • The guarantee does not cover non-Gaussian tails, unbounded variance asymmetry, all benchmark-construction or contamination uncertainty, or candidates generated through benchmark feedback.Hidden-family floors and heterogeneity caps remain assumptions rather than estimates for frontier labs; public-board verdicts are conditional because candidate logs are unavailable.Source: §6 · p. 9

What a board should publish

  • A leaderboard operator can request a candidate count and a record of whether the candidate set was fixed before benchmark feedback, then publish a finite-range sensitivity curve.A declared count alone does not verify that protocol; disclosed candidates permit joint max-t inference, while independent confirmation data can separate selection from evaluation.Source: §6 · pp. 9–10

A ETHICS, REPRODUCIBILITY, AND AI USE

The appendix records the paper’s public-data, reproducibility, and AI-use practices, then proves the Gaussian margin results that underpin its certificate. The proofs establish least-favorable equal means and correlation monotonicity, with stated limits on the comparator assumptions.

A ETHICS, REPRODUCIBILITY, AND AI USE

  • The audit uses public releases and makes no claim about providers’ private conduct; supplementary scripts and machine-readable results support reproduction, while authors say they verified the analyses and numbers despite substantial LLM use.Reproduction requires the corresponding public datasets or model-evaluation outputs.Source: A ETHICS, REPRODUCIBILITY, AND AI USE · pp. 11–12

B PROOFS

  • The Gaussian margin vector has shared variation and independent residual noise; it matches the score-level model when ρw ≥ ρb, while extending the margin law below that range does not give a score-level factorization.The common score component cancels in each margin.Source: B PROOFS · p. 12

B.1 PROOF OF LEMMA 1 (LEAST-FAVORABILITY OF EQUAL MEANS)

  • For a fixed comparator, the null tail is maximized when all candidate means equal the comparator mean, because it is nondecreasing in each candidate’s mean shift.The conditional residuals are independent given shared factors; no selected runner-up is involved.Source: §B.1 · p. 12

B.2 PROOF OF LEMMA 2 (MONOTONICITY IN ρw)

  • The selected-margin tail is nonincreasing in within-family correlation, so evaluating the certificate at a lower correlation bound is conservative.The result follows from equal-variance Gaussian margin vectors whose off-diagonal covariance increases with ρw.Source: §B.2 · p. 12

B.3 PROOF OF PROPOSITION 1 (MONOTONICITY IN THE COMPARATOR CORRELATION)

  • On the standardized z scale, the null tail is nondecreasing in comparator correlation; heterogeneous within-family correlations require their minimum, not their mean, as the floor.The proposition assumes homogeneous comparator correlations; unequal margin variances under heterogeneous comparator correlations prevent direct use of Slepian on raw margins.Source: §B.3 · p. 13

B.4 PROOF OF LEMMA 3 (COMPARATOR HETEROGENEITY, UNEQUAL VARIANCES, ARBITRARY SELECTION)

Lemma 3 extends the selection bound to heterogeneous comparator correlations and unequal candidate variances by comparing standardized margins. The resulting certificate permits arbitrary selection rules but depends on a valid lower bound for the margin correlations.

B.4 PROOF OF LEMMA 3 (COMPARATOR HETEROGENEITY, UNEQUAL VARIANCES, ARBITRARY SELECTION)

  • The bound applies to any selection rule and controls the unconditional chance of certifying a submitted model that is not better, provided margins use their true standard deviations.It does not establish conditional validity given the realized selection, and the audit’s estimated paired standard error is a plug-in procedure rather than this theorem.Source: §B.4 · pp. 13–14
  • For unequal variances with score-SD ratios bounded by R, interval arithmetic gives certified lower bounds on the worst-case standardized margin correlation.For the audited rank-1 pair, the floor is 0.447 at R=1.1 and 0.400 at R=1.25, versus 0.492 at R=1.Source: §B.4 · p. 15
  • The certificate operates only on its stated margin-domain range; outside it, the procedure reports model-insufficient rather than evaluating the tail as a certificate.The exchangeable margin covariance can remain positive semidefinite beyond the certificate’s operating range, but the reference law is not used there.Source: §B.4 · p. 15

B.5 SELECTION-VALID NUISANCE INFERENCE

The appendix develops selection-valid nuisance bounds, tests them on replayed candidate families, and states where their guarantees do and do not extend. It also checks the clone limit and the scope of shipped-model certification.

B.5 SELECTION-VALID NUISANCE INFERENCE

  • The finite-sample empirical-Bernstein construction is vacuous here: splitting η=γ/2K across six benchmarks makes the smallest benchmark’s moment pads about 10% and saturates the β bound at 1.This does not rule out sharper finite-sample approaches.Source: §B.5 · p. 16
  • For the rank-1 claim, the required correlation floor rises from 0.689 with plug-in adjustment to 0.756 with selection-valid adjustment; joint adjustment lowers the effective margin from z=2.61 to 2.46.Source: §B.5 · p. 16
  • Selection-valid nuisance bounds use a γ/K adjustment because selecting among K candidates can make the selected bound’s miss probability exactly Kη when individual failures are disjoint.Reducing that adjustment requires information such as hidden losers’ scores, selection gaps, or a prospective holdout.Source: §B.5 · p. 16
  • Nuisance-adjusted curves are reported only through Kmax=100; a ≥100 endpoint is censored, not an unlimited budget or a simultaneous statement over correlation floors.The k=3–8 family replays do not validate coverage at K=100.Source: §B.5 · p. 16
  • At nominal 99.5%, selection-valid bounds covered selected correlations at 99.75–99.9% in all 12 replayed families, while ordinary fixed-pair bounds ranged from 99.32–99.5%.The joint rectangle covered both selected correlation and variance at 99.80–99.9% in all twelve families; replayed families had k=3–8.Source: §B.5 · p. 18

B.6 CLONE BOUNDARY (ρw →1): SANITY CHECK

  • At the clone boundary ρw=1, hidden multiplicity is unidentifiable from the margin and harmless under the provider-level null, with the critical value tending to the k=1 value.For k=27, the reported near-clone critical value is 1.6467 versus z.95=1.6449.Source: §B.6 · p. 18

B.7 SHIPPED-MODEL CLAIM (REMARK 1) AND PROOF

  • Within nonrandomized tests that threshold only on observed z, k above the sensitivity budget rules out certification of either the provider-level or submitted-model claim.Tests using runner-up item profiles, full board covariance, or family-membership information are outside this result.Source: §B.7 · p. 18

C THE FULL NUMERICAL GRID AND QUADRATURE VALIDATION

The numerical appendix reports critical values and effective multiplicities for the documented 27-variant reference case, then checks the quadrature calculation against refinements and simulation.

C THE FULL NUMERICAL GRID AND QUADRATURE VALIDATION

  • At α=.05 and ρb=0.50, Table 4 reports standardized-margin critical values and effective multiplicity for k=27.The table’s scope is the documented reference multiplicity.Source: Table 4 · p. 18
  • Refining the quadrature from 4,001 to 20,001 points changes no reported cell by more than 10^-6, and Monte Carlo agrees within 1.2 standard errors at 4×10^6 draws.Across validation cells, the 2×10^6-draw comparison is within 2.1 standard errors.Source: C THE FULL NUMERICAL GRID AND… · p. 19
  • Near the clone boundary, a 200-node Gauss–Hermite rule errs by 7 Monte Carlo standard errors, motivating the fixed fine-grid quadrature.The example uses z=2.5, k=27, ρw=0.99, and ρb=0.5.Source: C THE FULL NUMERICAL GRID AND… · p. 19

D MEASUREMENT DETAILS

The measurement details compare observed sibling-family correlations while emphasizing that pooled-item patterns cannot be transported to a different score functional or treated as a hidden-family floor.

D MEASUREMENT DETAILS

  • Observed pooled correlations rise from base-to-instruct pairs (0.44–0.65) through merges/LoRA (≈0.71) and version series (≈0.78) to controlled checkpoints (≈0.90).This ordering does not transfer across score functionals: the same checkpoint family is 0.90 pooled and 0.556 on its audited composite.Source: D MEASUREMENT DETAILS · p. 19
  • The census minimum is not a reliable large-family floor: truncation at six members makes small-family minima optimistic, and dropping one family raises the ≥7B-base floor from 0.42 to 0.705.Census families were truncated to six, while curated families range from k = 3–8.Source: D MEASUREMENT DETAILS · p. 19
  • The curated-family rules require a shared organization prefix, an explicit shared variant-series token, and at least three members, with rules frozen before correlations were computed.A fixed-estimator comparison reports MMLU medians of 0.676 for curated families, 0.746 for census families, and 0.778 for version-series pairs; it is not a de-confounding.Source: D MEASUREMENT DETAILS · p. 19
  • Figure 4 reports item-resampling score-matched minima and means for all twelve curated v1 families, while 17 of 45 org-disjoint MMLU-Pro census-family minima fall below the 0.56 sensitivity anchor.The populations differ in tasks, construction rules, and family sizes; their observations do not establish a hidden-family floor.Source: Figure 4 · p. 20

E SUBJECT-CLUSTER SAMPLING SENSITIVITY

This sensitivity analysis replaces item resampling for MMLU with resampling its 57 subjects as exchangeable blocks, changing the sampling population. It finds that the score-matched correlation depends strongly on that choice, while subject-cluster results remain diagnostics rather than a new joint-coverage guarantee.

E SUBJECT-CLUSTER SAMPLING SENSITIVITY

  • The alternative sampling scheme resamples MMLU’s 57 subjects as exchangeable blocks while retaining shared-row item resampling for other benchmarks.It does not assume subjects represent all possible knowledge domains, and it leaves within-GSM8K template dependence and contamination unmodeled.Source: E SUBJECT-CLUSTER SAMPLING SENSITIVITY · p. 21
  • The block bootstrap carries each sampled subject’s items, recomputes the original item-weighted score and covariance, and preserves between-subject score contrasts during null replays.Subject means are not equally weighted or separately centered.Source: E SUBJECT-CLUSTER SAMPLING SENSITIVITY · p. 21
  • The controlled family’s score-matched median correlation is 0.459 under item resampling and 0.921 under subject resampling, versus a pooled median of 0.894.The pairwise minima are 0.417 and 0.914, respectively; MMLU’s median share of composite variance rises from 9.0% to 86.7% under subject sampling.Source: E SUBJECT-CLUSTER SAMPLING SENSITIVITY · p. 21
  • Under subject sampling, the rank-1 plug-in required correlation floor is 0.861, and the reported cluster calculations are sampling-model sensitivity diagnostics rather than a new joint-coverage guarantee.The rank-3 bootstrap one-sided tail estimate is 0.0497, described as borderline; item-level nuisance intervals are not transferred to 57 subject clusters.Source: E SUBJECT-CLUSTER SAMPLING SENSITIVITY · pp. 21–22
  • Across twelve curated-family replays, the Gaussian reference tail is at least the empirical tail in 54/60 threshold cells and in every cell for 8/12 families.None of the six pointwise exceptions is resolved by simultaneous intervals; these observed-family checks are not a theorem for arbitrary subject dependence or hidden families.Source: E SUBJECT-CLUSTER SAMPLING SENSITIVITY · p. 22

F MASKING-VALIDATION DETAILS

The masking-validation details compare the certificate with bootstrap, conditional-test, and board-reselection checks. The reported validations support calibration in the tested configurations, while leaving explicit limits on what those checks certify.

F MASKING-VALIDATION DETAILS

  • In board-level replays, the empirical studentized tail fell when both winner and comparator were reselected, but this is calibration for the observed board under the global sharp null, not a uniform theorem.The per-claim certificate remains pair-conditional; the global sharp null is not least-favorable for board-level selection.Source: F MASKING-VALIDATION DETAILS · p. 22
  • The certificate is more conservative than the full-observability conditional p-value in all 12 unmasked families, and both tests reject at rates consistent with α = 0.05 under the sharp null.In the small-benchmark calibration, the worst min-pairwise cell is within Monte Carlo bands.Source: F MASKING-VALIDATION DETAILS · pp. 22–23
  • For five audited claims, empirical bootstrap tails track Gaussian tails across the tested grid; at t = 2.5, empirical tails are 0.0061–0.0068 versus Gaussian 0.0062.The reported skew and excess kurtosis are small, and the passage concludes the marginal Gaussian approximation is not the binding assumption at these sample sizes.Source: F MASKING-VALIDATION DETAILS · p. 23
  • The public submission history averages 6.0 submissions per organization, but sparse base-model metadata and rarely observed deletions make these descriptive counts, not lower bounds on a claim’s hidden pool.The census is limited to the 45 largest qualifying families, truncates each at six members, and drops 22/209 attempted members for missing or unscorable repos.Source: F MASKING-VALIDATION DETAILS · pp. 23–24
  • With simultaneous Monte Carlo bands, the audit-aligned min column resolves no anti-conservative cells across 60 cells, while the single-benchmark min column resolves 78/96 conservative cells and 18 consistent cells.The mean-correlation column has four statistically resolved failures; the min-column’s worst point estimate remains within bands.Source: F MASKING-VALIDATION DETAILS · pp. 23–24
  • The dyadic training-factor analysis preserves shared-variant and shared-item dependence by resampling items with common indices across all correctness columns and refitting the full correlation matrix.Its item-bootstrap intervals hold the trained variants fixed and address item-sampling precision, not model-level significance.Source: F MASKING-VALIDATION DETAILS · pp. 24–25

G CONTROLLED FAMILIES UNDER ITEM RESAMPLING

Controlled-family experiments show that pooled-item correlation can substantially exceed the correlation of the composite score under item resampling, and test whether the certificate remains conservative across families and settings. The worked example illustrates how an individual claim’s certification boundary depends on the assumed correlation floor and adjustment.

G CONTROLLED FAMILIES UNDER ITEM RESAMPLING

  • The training-factor regression treats dyads as dependent, and its permutation results associate shared training data with higher correlation and mismatched adapter capacity with lower correlation.Settings were pre-specified rather than randomized; the step-count permutation test does not reject (p = 0.573).Source: G CONTROLLED FAMILIES UNDER ITEM… · p. 25
  • In a ten-variant Qwen2.5-1.5B family, pooled medians are 0.89/0.85 for full-FT/LoRA, while composite medians are 0.46/0.42.For full-FT against an external comparator, the homogeneous tail is resolved as anti-conservative at z = 1.5; the joint variance-aware construction restores domination across all five cells.Source: G CONTROLLED FAMILIES UNDER ITEM… · pp. 25–26
  • The composite-correlation gap persists among top-scoring and accuracy-matched variants, and the combined ten-variant pool has a composite min-pairwise correlation of 0.170.The passage attributes the pattern to composite variance being dominated by the small generative benchmark where siblings disagree.Source: G CONTROLLED FAMILIES UNDER ITEM… · p. 27
  • For the five full-FT variants, the certificate tail is at or above the empirical selection null at every tested margin; three of five cells resolve as conservative and none as anti-conservative.The base comparator is nearly as correlated with variants as they are with one another, at approximately 0.90.Source: G CONTROLLED FAMILIES UNDER ITEM… · pp. 27–28
  • Across six strata, score-matched medians are 0.52–0.58 for two added SFT lineages and 0.81 for DPO on the same base.The authors report that preference optimization preserves more of the base’s error structure than the measured SFT strata.Source: G CONTROLLED FAMILIES UNDER ITEM… · p. 28
  • Across six controlled strata, the pooled-versus-score-matched gap appears in every row, and the β+-robust column dominates pointwise in 17/18 family-stratum/comparator replays.The only resolved anti-conservative cell is the full-FT equal-variance cell; incorporating unequal variances restores domination in the remaining replay.Source: G CONTROLLED FAMILIES UNDER ITEM… · p. 28

H WORKED EXAMPLE: AUDITING ONE CLAIM WITH SELECTIVE-EVALS

  • For the worked v1 claim, the plug-in verdict certifies k = 27 only under ρw ≥ 0.69, while the joint adjustment requires a floor of 0.849.The claim is not margin-certified at the two primary anchors, 0.42 and 0.56.Source: H WORKED EXAMPLE: AUDITING ONE CLAIM… · p. 29

I ARENA, PROVENANCE, AND METADATA

No claim passes the shown Gaussian adjustments, and the audit’s verdicts remain conditional because private candidate logs are unavailable. The board examples show that certification can vary with assumed multiplicity and correlation, while related work distinguishes this retrospective correction from other evaluation safeguards.

I ARENA, PROVENANCE, AND METADATA

  • In the measured-extreme joint stress test, rank 1’s required correlation floor reaches 0.939 and every anchor refuses, even at the 0.86 mean-band reference.Budgets fall to M-I/6/8/13; the authors state these measured extremes do not bound a named hidden family.Source: I ARENA, PROVENANCE, AND METADATA · p. 30
  • Without complete private candidate logs, none of the 394 v1 claims or v2/Arena illustrations has verified fixed-family applicability; verdicts are conditional scenarios.Measured public-family correlations do not bound hidden-family correlation, and overlapping claims make board counts descriptive.Source: I ARENA, PROVENANCE, AND METADATA · pp. 30–31
  • No claim passes the shown Gaussian adjustments; v1 claim #2 is Bonferroni-supported despite model insufficiency.The figure reports all 394 overlapping v1 adjacent claims.Source: Figure 12 · p. 30
  • At the curated floor, the v1 aggregate changes from (1, 360, 33) at k = 5 to (0, 361, 33) at k = 27; only rank 1 is newly certified at smaller k.Rank 1 is certified at every adjustment through k ≤5 and refused everywhere from k = 20.Source: I ARENA, PROVENANCE, AND METADATA · p. 31
  • Arena’s five same-provider score-pair correlations are −0.05 to −0.02, while the v2 top-30 extension certifies 5/30 naively and 2/30 by Bonferroni at k = 27.The Arena rank-1 gap is refused at k = 27 near the near-zero-correlation corner; in the v2 extension, denser pairing shrinks margins for nine top-15 claims.Source: I ARENA, PROVENANCE, AND METADATA · pp. 30–31

J EXTENDED RELATED WORK

  • Crowdsourced-vote manipulation attacks comparisons, while evaluation-UQ corrections address variance sources such as judge choice, decoding temperature, and prompt design.The cited account quantifies 56 Elo at documented k = 27, reducible to 32 through pipeline design; pipeline correction and claim certification compound rather than substitute.Source: J EXTENDED RELATED WORK · pp. 31–32
  • Selective-inference methods such as zoom correction require observing every candidate’s score, an assumption this audit’s hidden-candidate setting drops.The paper instead uses rank-verification machinery where candidates are observed and treats retrospective inference as the setting of the audit.Source: J EXTENDED RELATED WORK · p. 32
  • Ex-ante mechanisms complement ex-post certification; unlike behavior-calibrated search thresholds, this paper leaves k and ρ−w as declared axes because family correlation is not a searcher property.The related-work comparison notes that behavioral calibration cannot recover correlation among searches.Source: J EXTENDED RELATED WORK · p. 32
Loading 2609.28177v2…