Source-linked AI summary
Rank Confidence Sequences:Anytime-valid Leaderboards
Hamed Khosravi, Xiaoming Huo
TL;DR
Rank intervals for shared-item leaderboards are usually fixed-sample, although results are inspected repeatedly and existing anytime-valid methods do not provide ranks for this setting. The paper constructs rank confidence sequences from pairwise betting evidence and tests over possible orderings. Its experiments report validity under monitoring and performance close to fixed-sample methods at a pre-planned halfway look.
Problem
Existing rank intervals assume a single pre-planned look, while no existing method provides anytime-valid rank sets for all models scored on the same items.
Source: §1 · p. 1Method
Results
At a pre-planned halfway look, the method certifies about as many comparisons as the strongest fixed-sample procedure; under 200 looks, fixed-sample rank sets err in 30% to 51% of runs.
Source: §1 · pp. 2–3Takeaways & Limitations
A model’s evaluation can stop once its question is answered, such as whether it is in the top three, while the guarantee survives data-dependent stopping.
Source: §1 · p. 2Models arriving over time and adaptive item selection are not covered, and validity applies to the benchmark as given rather than transfer to other tasks.
Source: §2 · p. 3, §6 · p. 10
Abstract
from arXiv · showhide
Leaderboards rank models by their average scores on benchmark items, and they are consulted repeatedly while the evaluation is still running. Existing confidence intervals for a model's rank control their error rate only if they are computed once, after a number of items chosen in advance. If they are recomputed as results arrive, and the evaluation stops once they look decisive, their error rate exceeds its nominal level. Anytime-valid methods keep their guarantees at all sample sizes simultaneously and hence under any stopping rule. They exist for the accuracy of one model, for one pair of models and for the set of models that may be best. For pairwise battles they also give ranks. None gives ranks when all models are scored on the same items, which makes their scores dependent. We construct rank confidence sequences: for every model, a set of ranks that contains its true rank, simultaneously for all models and at all times, at a chosen error level $\alpha$, in finite samples. The construction combines betting e-processes, one for each ordered pair of models, with closed testing over the possible orderings of the models. It allows any dependence between the models' scores on an item. The method has two advantages. A leaderboard can be inspected after every item without inflating its error rate. The evaluation of each model can stop as soon as the question asked about it is answered, which saves compute. When results are examined only once, halfway through or later, little power is lost relative to fixed-sample methods. The paper quantifies these advantages in simulations and on public leaderboard data.
1 INTRODUCTION
Existing rank intervals assume a single pre-planned look, while leaderboard results are often inspected repeatedly as evaluation proceeds. This section motivates anytime-valid rank sets and introduces a method for producing them under shared-item dependence.
1 INTRODUCTION
- Repeated looks and stopping when intervals seem decisive can push the chance of a false report above the nominal level.Each examination creates another opportunity for error, and evaluation may stop exactly when an interval is wrong.Source: §1 · p. 1
- After 200 looks, fixed-sample rank sets at nominal 5% make at least one false statement in 30% to 51% of runs with six equally accurate models.The guarantee covers one look, not repeated monitoring.Source: §1 · p. 2
- Anytime-valid rank sets must cover all models simultaneously at every sample size, despite unknown dependence from scoring models on the same items.This guarantee remains valid when stopping depends on observed data.Source: §1 · p. 2
- The paper constructs rank confidence sequences by combining pairwise betting evidence with testing over possible model orderings.The guarantee holds in finite samples without assumptions on dependence among models’ scores.Source: §1 · p. 2
- The method supports anytime inspection, model-by-model stopping, and halfway-look comparisons close to the strongest fixed-sample procedure.It can report certified comparisons, rank sets, top-k sets, and tiers; evaluations can stop once a model-specific question is answered.Source: §1 · pp. 2–3
2 PROBLEM SETTING
The problem setting defines model ability as mean score and rank by the number of models with strictly greater ability. It considers shared-item scores under finite-benchmark and superpopulation sampling, with rank sets required to cover all models at all times.
2 PROBLEM SETTING
- A model’s ability is its scalar mean benchmark score, and its rank is one plus the number of models with strictly greater ability.Tied models share a rank; rank 1 is best.Source: §2 · p. 3
- Every item yields a score in [0, 1] for each model, and the models’ scores may be dependent.Binary correctness is a common case, but scores may take any value in the interval.Source: §2 · p. 3
- The main setting evaluates a fixed benchmark in a random order shared by all models; the guarantee also covers i.i.d. items from a population.For a fixed benchmark, the only data randomness is the order of evaluation; for the population setting, the joint score law across models is unrestricted.Source: §2 · p. 3
- A rank confidence sequence is a time-adapted set for each model whose true rank is covered for every model and time with probability at least 1−α.The guarantee is required for every data-generating law.Source: §2 · p. 3
3 RANK CONFIDENCE SEQUENCES
The method updates betting evidence for ordered model pairs, eliminates incompatible weak orderings, and reports statements shared by all surviving orderings. Its guarantees hold under the stated score and sampling conditions, with several options for computing and reporting results.
3 RANK CONFIDENCE SEQUENCES
- Certified comparisons are those on which every surviving ordering agrees, and reports can include rank intervals, top-k sets, and tiers.Enumeration can produce exact rank sets, which may be smaller than the rank intervals.Source: §3 · pp. 4–5
- Pairwise betting wealths measure evidence that one model scores higher than another, using bets whose sizes are selected from a finite grid.The method uses the score sequence, the preselected error level, and betting-fraction grid; under the null, the wealth is an e-process and its threshold crossing remains valid at all times.Source: §3 · p. 4
- The procedure eliminates a weak ordering when the average wealth for the pairwise hypotheses it implies reaches 1/α.The true ordering is retained with probability at least 1−α, even when the pairwise wealths are dependent.Source: §3 · pp. 4–5
- The exact enumeration method is practical up to about eight models; polynomial-time and integer-programming alternatives can be used at larger sizes.The polynomial-time shortcut may certify fewer comparisons, while the integer-programming method solved every experiment instance.Source: §3 · p. 5
- Models may be retired using observed scores, after which wealths for pairs involving them are frozen and remain in subsequent reporting.Examples include retiring a model once its top-k status is certified or its rank interval reaches a target width.Source: §3 · p. 5
- The method requires a fixed model set, non-adaptive item selection, and scores for every model not yet retired on each item.Pairwise-battle data are outside the stated scope; the per-item update costs O(M^2).Source: §3 · p. 6
4 WHAT CAN BE ESTABLISHED, AND WHY IT MATTERS
The section establishes simultaneous validity for certified comparisons, derives guarantees for rank-related reports, and shows that model-by-model retirement preserves validity. These results rely on pairwise e-processes sharing a filtration, without requiring independence among them.
4.1 ALL CERTIFIED DOMINANCES ARE CORRECT, AT ALL TIMES
- Pairwise e-processes and Ville’s inequality control false dominances uniformly over all times, even when the wealth processes are dependent.The true ordering is never eliminated with probability at least 1−α, and the guarantees also hold at arbitrary random times.Source: §4.1 · p. 6
4.2 RANKS, TOP-k SETS AND TIERS INHERIT THE GUARANTEE
- Rank intervals, exact rank sets, and top-k reports inherit the certified-set guarantee; a tier index is a lower bound on true rank, not a pairwise certificate.A model’s tier index never decreases as certified evidence accrues, and tier 1 can include models with no certified comparisons.Source: §4.2 · p. 7
4.3 EVALUATION CAN STOP EARLY, MODEL BY MODEL
- Freezing pairwise wealths when either model retires preserves the validity assumption, allowing evaluation to stop separately for each model.The model’s last scored item remains in its record, and retirement is permanent.Source: §4.3 · pp. 7–8
- Under monitoring, Figure 1 contrasts fixed-sample curves that exceed α with e-process curves that stay below α as looks accumulate.The probability plotted is that some report is wrong; both axes are logarithmic.Source: Figure 1 · p. 8
5 EXPERIMENTS
The experiments find that rank confidence sequences remain reliable under repeated monitoring, support partial rankings early, and can reduce evaluation when models stop individually. At a pre-planned halfway look, they match or exceed the strongest fixed-sample procedure on five of six benchmarks.
E1
- 30%–51% of monitored runs contain a false fixed-sample rank statement, versus 1.6% with the exact test at 200 looks and nominal 5%.With all six models tied, both fixed-sample procedures are valid at one pre-planned look; the inflation is caused by monitoring, not the normal approximation.Source: §5 · p. 8
E2
- Table 1 summarizes the percentage of true dominances certified and mean tiers at halfway; it reports that no report was wrong in these runs.The caption identifies the summary measures and halfway results but does not provide cell values.Source: Table 1 · p. 9
- Figure 2 compares the fraction of true dominances certified against the fraction evaluated for the twenty highest-scoring models, with mean tiers at halfway.The method is checked every 1%, while Holm and Romano–Wolf are checked at one look.Source: Figure 2 · p. 9
- At halfway on MMLU, 50% of 190 true dominances are certified among the top twenty models, which form 3.0 certified tiers.When twenty models are spread across the MMLU leaderboard, 91% of their true dominances are certified halfway; across 900 top-eight runs, one report is wrong.Source: §5 · pp. 8–9, Table 1 · p. 9
E3
- 6.2% of the full MMLU evaluation suffices to certify top-3 membership for all 395 released models, versus 31% for the twenty highest-scoring models alone.In simulation, retiring models individually costs 22% of evaluation against 84% when all stop together; across all 395-model runs, 86% of true MMLU dominances are certified halfway and 99.6% by the end, with no wrong reports.Source: §5 · p. 9
E4
- At a pre-planned halfway look, the method matches or exceeds bootstrap step-down rank sets on five of six benchmarks and trails by four percentage points on TruthfulQA.The later separation increasingly reflects the finite-benchmark offset, not a pure power comparison.Source: §5 · pp. 9–10
6 LIMITATIONS AND CONCLUSION
The paper identifies boundaries on the method’s scope and concludes that rank confidence sequences provide simultaneous, anytime-valid rank statements for shared-item leaderboards. Its experiments show reliability under monitoring, competitive halfway comparisons, and reduced evaluation for top-3 decisions.
6 LIMITATIONS AND CONCLUSION
- The method does not cover models arriving over time or adaptive item selection, and its validity is for the benchmark as given, not contamination or transfer to other tasks.At an early pre-planned look, fixed-sample procedures certify more.Source: §6 · p. 10
- Rank confidence sequences provide rank statements valid for all models at all times on shared-item leaderboards.The conclusion reports that the method stayed valid under monitoring where fixed-sample rank sets did not.Source: §6 · p. 10
- At a pre-planned halfway look, the method matched or exceeded fixed-sample procedures on five of six benchmarks, and all-model top-3 status required 6.2% of a full MMLU evaluation.These are experimental outcomes reported in the conclusion.Source: §6 · p. 10
A METHOD DETAILS
The method eliminates weak orders that conflict with accumulated pairwise evidence, then derives certified comparisons and rank sets from the orders that remain. Pairwise betting supplies anytime-valid evidence even when model scores depend on one another, while bet choices and certification algorithms govern power and computation.
A METHOD DETAILS
- Enumeration checks every ordering directly and is practical while the number of orderings remains manageable; Algorithm 1 implements it.The appendix also gives an integer-programming approach and a closed-form shortcut.Source: A METHOD DETAILS · p. 15
- The shortcut uses transitivity to combine direct wealth with indirect evidence through a third model, runs in O(M) operations per pair, and certifies a subset of the exact set.Its statistic adds E_jl to the sum over third models m of min{E_jm, E_ml}.Source: A METHOD DETAILS · p. 15
- Algorithm 3 tests whether any surviving weak order places l at least as high as j, and computes the exact certified set in exact arithmetic for any M.Its integer program encodes weak orders through binary pairwise relations; Proposition A.1 says each inner loop ends within t solves.Source: A METHOD DETAILS · pp. 15–16
- Exact certification is coNP-hard in the worst case, although Algorithm 3 solved every instance in the reported experiments.The integer program has M(M−1) binary variables and M(M−1)(M−2) transitivity constraints.Source: A METHOD DETAILS · p. 16
B SUPPORTING THEORY
- For every predictable bet, pairwise wealth is an e-process under the null, regardless of other models’ parameters or dependence between their scores.The bet choice affects power but not validity.Source: §B.1 · p. 17
B.1 VALIDITY AND GROWTH OF EACH PAIRWISE WEALTH
- A constant bet can have negative expected log-growth when the ability gap is small relative to discordance, so the pair may never be certified.Predictable alternatives include a plug-in bet, online Newton steps, and mixtures; a fixed-grid mixture covers only the range of gap-to-discordance ratios spanned by its grid.Source: §B.1 · pp. 17–18
- For binary scores, a finite bet grid is inconsistent for gaps with Δ/ρ below its threshold; for the default grid, the threshold is about 0.015.Consistency holds when the smallest grid bet has positive growth, but no finite grid is consistent for every positive gap.Source: §B.1 · p. 18
- When 0.06 ≤ Δ/ρ ≤ 1, the default grid yields certification in O(ρ Δ^-2 log(|Λ|M^2/(αε))) items; the exact test certifies no later than the shortcut, which certifies no later than e-Bonferroni.The stated probability and expected-time bounds also apply to the shortcut and exact test.Source: §B.1 · pp. 18–19
B.2 SUPPLEMENT TO THE VALIDITY THEOREM
The supplement develops validity guarantees for order-based testing, including an α-level bound for the event in (8) and validity at arbitrary random times. It also gives conditions for preserving power, guarantees for tiers and retirement, and further structural, computational, and comparative details.
B.2 SUPPLEMENT TO THE VALIDITY THEOREM
- Testing only exactly realizable intersections T(W) avoids testing most intersections, and the resulting relation is a strict partial order unless the confidence set is empty, an event with probability at most α.For M = 3, 13 of 63 intersections remain; for M = 4, 75 of 4095 remain. The empty confidence set reveals an error.Source: §B.2 · p. 19
B.3 TESTING ONLY ORDERINGS LOSES NOTHING, AND THE SHORTCUT IS VALID
- The ordering-restricted test includes every rejection of the unrestricted closed test on every sample path, while the shortcut is valid, costs O(M) per pair per update, and includes e-Bonferroni rejections.The shortcut threshold remains conservative relative to the exact test.Source: §B.3 · p. 19
- With M = 3, E13 = 60, E12 = 42, and E23 = 5 at α = 0.1, the restricted test certifies 1 ≻ 2 while the full-closure statistic is 9.4 against 10 and e-Bonferroni is 42 against 60.More generally, this strict advantage holds when c < 1/α and 4/α ≤ a < 5/α − c.Source: §B.3 · p. 20
- About 7% more true dominances are certified than by e-Bonferroni after 500 items, at most 3% more by 4000; on the leaderboard, 2–5% more are certified up to three quarters of the benchmark, with nearly the same number at the end.These comparisons concern the exact test; the leaderboard result is for its eight highest-scoring models.Source: §B.3 · p. 20
B.4 TIERS AND RETIREMENT
- On the good event, tiers partition models into antichains, satisfy Rj(θ⋆) ≥ Lj,t ≥ tiert(j), and are monotone as certified dominances accumulate; their number is at most the number of distinct model scores.The tiers form a minimum-size antichain partition, but being in a lower tier does not itself certify dominance over every model in a higher tier.Source: §B.4 · pp. 20–21
- Under (S) and the stated pairwise-betting assumptions, all-pairs retirement has a cost bound, but a model tied with another is never retired on the good event and incurs the full planned cost.The stated bound does not cover the top-k retirement rule.Source: §B.4 · p. 21
C.1 PAIRWISE WEALTHS
- Under condition (c), the result gives 1/aλ ≤ min{74.7, 270ρ}Δ^-2; for binary scores, it identifies the exact growth at λ = 0.03 via formula (b).The stated constant is c0.03 = 0.0600.Source: §C.1 · p. 24
C.2 VALIDITY, STRUCTURE AND POWER
- The event in (8) is contained in {τ⋆ < ∞}, whose probability is at most α by Ville’s inequality.This yields the stated error bound for that event.Source: §C.2 · p. 25
- Because both events hold for all times, their corresponding statements hold at any N₀-valued random time, even if it is not a stopping time.Source: §C.2 · p. 25
- When C_t is nonempty, D_t is irreflexive, asymmetric, and transitive, so it defines a strict partial order.The proof uses the relations imposed by any W in C_t.Source: §C.2 · p. 25
- Example B.11 shows that the inclusion {∃t: C_t = ∅} ⊆ {∃t: W⋆ ∉ C_t} can be strict.The configuration was checked by enumerating 13 weak orders and 63 subsets.Source: §C.2 · p. 25
- A common time s for all W permits the running maximum over B_jl; e-Bonferroni rejections are contained in D_sc, at a cost of one sum of M − 2 minima per pair.Source: §C.2 · p. 26
C.3 CLOSURE, RANKS, TIERS AND RETIREMENT
The appendix shows how certified pairwise comparisons yield rank sets and tiers, while retirement can reduce evaluation cost under specified conditions. It also establishes that deciding exact certification is computationally hard for some wealth matrices.
C.3 CLOSURE, RANKS, TIERS AND RETIREMENT
- Certified comparisons form a strict partial order, and the resulting rank bounds contain each model’s rank on the good event.The bounds are L_j,t = 1 + the number certified above model j, and U_j,t = M − the number certified below it; ties are counted on neither side.Source: §C.3 · pp. 26–27
- With M = 3, a model’s projected rank set can be {1, 3} even when its enclosing rank interval is [1, 3].The examples also show that a top tie can give model 1 rank 1 in every surviving order although its upper rank bound is 2.Source: §C.3 · p. 27
- Tiers partition the models into incomparable groups, and tier numbers cannot decrease as certified comparisons accumulate.The highest tier is bounded above by the model’s true rank on the good event.Source: §C.3 · pp. 27–28
- Under the all-pairs rule, a model with a tied partner is never retired on the good event and incurs the full planned evaluation cost.The expected-cost bound is developed for the all-pairs rule under (S), with the stated untied-pair conditions for the sharper bound.Source: §C.3 · p. 29
C.4 INTEGER PROGRAMMING AND HARDNESS OF EXACT CERTIFICATION
- Deciding whether an ordered pair is exactly certified is coNP-hard over wealth matrices the update can produce with predictable bets.The reduction uses bets in {0, 1}; ties in the constructed ordering do not help, since tied classes can be refined while keeping at most half their internal arcs backward.Source: §C.4 · pp. 29–30
D SCOPE OF THE GUARANTEE AND OPEN PROBLEMS
The paper identifies settings its guarantees and cost analyses do not cover, and reports finite-input implementation checks. Several computational and retirement-cost questions remain open.
D SCOPE OF THE GUARANTEE AND OPEN PROBLEMS
- The machine checks cover finite inputs and do not replace the proofs.They include exhaustive enumerations for small model counts and checks on random wealth matrices and finite-benchmark item orders.Source: D SCOPE OF THE GUARANTEE AND OPEN… · pp. 30–31
- The guarantee requires scalar model abilities, a fixed model set, and an item order that does not depend on observed data.Pairwise-battle win probabilities need not be transitive, and data-dependent item choice breaks the stated sampling conditions.Source: D SCOPE OF THE GUARANTEE AND OPEN… · p. 31
- The top-k retirement rule has no established cost bound, and the paper leaves open a rate under (F) and a sharper exact-test rate.The available cost bound covers the all-pairs rule under (S), while validity under either retirement rule follows separately.Source: D SCOPE OF THE GUARANTEE AND OPEN… · p. 31
E EXPERIMENTS
The experiments evaluate rank certification on simulated designs and public leaderboard data, using a shared definition of error and comparisons against fixed-sample procedures.
The real leaderboard
- The real-leaderboard study uses per-item results from 395 models across six benchmarks, with random item orders and selected subsets as well as all models.The truth is defined by mean score over the full benchmark; each setting is redrawn 50 times.Source: E EXPERIMENTS · p. 31
The simulations
- Simulations cover shared item difficulty, ties and near ties, and finite benchmarks with several model counts and accuracy gaps.The first design uses six models; the benchmark design uses 5,000 items and either six or twenty models.Source: E EXPERIMENTS · p. 32
What counts as an error
- A run counts as wrong if any report at any monitored time certifies a false dominance, which also captures rank intervals that miss a true rank.This error measure corresponds to an analyst free to stop at any look.Source: E EXPERIMENTS · p. 32
Competitors and implementation
- Fixed-sample competitors use Holm or Romano–Wolf step-down procedures, with paired tests or a bootstrap, and are recomputed at monitored looks.The implementation follows Section 3, and a separate simulation reports that the Theorem 4.2 bound is nearly attained.Source: E EXPERIMENTS · p. 32
E.1 DOES MONITORING BREAK A FIXED-SAMPLE RANK SET, AND DOES OURS SURVIVE IT?
Repeated monitoring substantially raises the false-statement rate of fixed-sample rank sets, while the proposed method remains below the nominal level in the reported settings.
Setup
- The comparison evaluates six models on 2,000 items, with procedures seeing the same data across 5,000 replications.Fixed-sample sets are recomputed at equally spaced looks; the proposed method is checked at those same looks.Source: §E.1 · p. 32
Results
- At 200 looks, false statements occur in 30% of runs for exact McNemar and 51% for paired z-tests, versus 1.6% for the proposed exact test.At one pre-planned look, the fixed-sample procedures are valid at 3.3% and 3.6%, respectively.Source: §E.1 · p. 32
- The excess errors persist with tied pairs and near ties: fixed-sample rates are 9.7% and 18% versus 0.3%, and 4.6% and 11.7% versus 0.2%.The near-tie setting has gaps of 0.004 and no exact ties.Source: §E.1 · p. 32
E.2 HOW MUCH OF A REAL LEADERBOARD CAN BE CERTIFIED?
On the full 395-model leaderboard, the method certifies many true dominances, with certification varying across benchmarks and evaluation progress. The analysis also explains why end-of-benchmark comparisons can favor its sets.
Study scope
- The study includes selected subsets and all 395 released models; selecting top-scoring models is valid under (F) but is not a prospective operator choice.The unselected run is included for that reason.Source: §E.2 · p. 33
Certification across the leaderboard
- At half a benchmark, certification reaches 26% to 86% of true dominances across the full leaderboard, and up to half among close models.At three quarters of MMLU, the top-twenty subset has 63% certified against 51% for Holm and 54% for Romano–Wolf.Source: §E.2 · p. 33
The whole leaderboard
- With all 395 models on MMLU, 86% of 77,802 true dominances are certified halfway through and 99.6% by the end, versus Holm’s 83% and 88%.The method produces 225 tiers at the end, compared with Holm’s 26; no false dominance is certified in 300 runs.Source: §E.2 · p. 33
Interpreting end-of-benchmark results
- At the benchmark’s end, fixed-sample methods certify only 55–58% of top-twenty MMLU dominances despite the ranking being known exactly.The paper says the fair comparison is at fractions up to one half, where methods are close and fixed-sample sets are ahead early.Source: §E.2 · p. 33
E.3 HOW MUCH COMPUTE DOES RETIREMENT SAVE?
Retiring models as soon as their questions are answered can sharply reduce evaluation cost, while real leaderboard results show that some top-k goals remain unsettled. At halfway, anytime-valid ranks retain much of the fixed-sample power while avoiding errors from repeated monitoring.
E.3 HOW MUCH COMPUTE DOES RETIREMENT SAVE?
- 22% of a full run certifies all twenty simulated models’ top-3 status under per-model retirement, versus 84% when stopping everything together.For six models and k = 2, the three top-k rules cost 44%, 22% and 30%; certifying the entire ranking costs 51% for six models and 90% for twenty.Source: §E.3 · p. 34
- 21–24% of a full run suffices for top-k certification at k = 2 through 16 among twenty close simulated models; the goal is reached in 97.5–99.5% of runs.Costs fall to 18% at k = 1 and 16% at k = 19, where only one side of the boundary must be resolved.Source: §E.3 · p. 34
- 31% of the twenty-model MMLU run, 57% of GSM8K, and 68% of HellaSwag suffice for the real-leaderboard top-3 goal, reached in 96–100% of orders.On ARC-Challenge and WinoGrande, where the top twenty are indistinguishable, retirement saves almost nothing; across all 395 models, 97.7–99.5% are settled even though the all-or-nothing goal is reached in only 0–2% of orders.Source: §E.3 · p. 34
- McNemar’s sign test on TruthfulQA targets the median paired difference, not the mean-score estimand, so its certifications are not comparable.The passage identifies this as a test mismatch, not a monitoring effect.Source: §E.3 · p. 35
- At equal cost, the uncorrected Pocock design reaches the goal in 56% of runs versus ours at 99%, but errs in 2.7%; Bonferroni reaches 3% at 32% cost.The uncorrected design also errs in 3.7% of runs without retirement; the corrected design supplies multiplicity correction but removes the saving.Source: §E.3 · p. 35
E.4 WHAT DOES ANYTIME VALIDITY COST?
- At halfway, ours matches or exceeds Romano–Wolf on five of six real benchmarks and trails by four points on TruthfulQA; monitored fixed-sample sets err in up to 4% of orders.At one quarter, fixed-sample sets lead by up to eight percentage points; that lead is halved by halfway. The comparison uses the valid fixed-sample look and six Romano–Wolf checkpoints.Source: §E.4 · p. 35
E.5 DOES THE GUARANTEE DEPEND ON HOW MODEL ERRORS ARE CORRELATED?
Across the tested dependence regimes, anytime error remains controlled while certification speed changes. The exact multiplicity correction generally certifies more pairs than e-Bonferroni, while the bet choice can matter more than the correction.
E.5 DOES THE GUARANTEE DEPEND ON HOW MODEL ERRORS ARE CORRELATED?
- At α = 0.05, anytime false-statement rates stay between 1.5% and 2.0% for six tied models and between 0.4% and 0.6% for two tied pairs across four dependence regimes.With evenly spread abilities, top-3 certification costs 16% under strong positive dependence and 46% under mixed-sign dependence; the latter reaches the goal in only 26% of runs.Source: §E.5 · p. 36, Table 5 · p. 36
- At half the items, strong positive dependence certifies 96% of true dominances with mean rank-interval width 0.22, versus 70% and width 1.49 under independence.With near ties, independence and the two positive regimes certify 74%, 79% and 80%; the latter is a practical ceiling because three gaps of about 0.004 are rarely resolved within 4000 items.Source: §E.5 · p. 36
E.6 WHAT DOES THE MULTIPLICITY CORRECTION CONTRIBUTE?
- After 500 simulated items, the exact test certifies about 7% more true dominances than e-Bonferroni, narrowing to at most 3% more by 4000 items.For near ties, the counts are 5.10 versus 4.77 of 15 after 500 items and 12.01 versus 12.00 by 4000; Table 6 compares certifiers using identical wealths.Source: §E.6 · p. 36, Table 6 · p. 37
- On real wealths, the exact test certifies 2–5% more than e-Bonferroni among the eight highest-scoring models through three quarters of a benchmark, and nearly the same by the end.The shortcut and e-Bonferroni with closure are nearly identical in these experiments; adjacent-level weights lose most early power because they give zero weight to non-adjacent pairs.Source: §E.6 · pp. 36–37
- At twenty models, the shortcut trails the exact test by 1.4 true dominances at halfway on average and by 0.3 at the end; the exact test solved every instance.The exact test uses integer programming beyond eight models; a run with looks every 1% of a benchmark needs about eight seconds on one CPU core.Source: §E.6 · p. 37
- A fixed bet of 0.25 never certifies gaps of about 0.02 in the tested settings, while the mixture bet certifies 14.0 of 15 true dominances after 5000 items.The mixture bet was the best adaptive bet at every checkpoint in every setting, motivating its use as the default.Source: §E.6 · p. 37
F RELATED WORK IN DETAIL
Existing rank methods use fixed-sample tests, while the closest anytime-valid approaches differ in what they certify and whether they exploit relations among pairwise hypotheses. This appendix situates the paper’s rank inference among those methods and the statistical tools behind it.
F RELATED WORK IN DETAIL
- Fixed-sample rank methods set critical values for a chosen sample size, whereas this method uses an anytime-valid certified set and exact enumeration can yield tighter rank sets than counting.Al Mohamad et al. (2021) use the partitioning principle for rank sets at one fixed sample size; the paper describes its exact test as their procedure’s anytime-valid counterpart.Source: F RELATED WORK IN DETAIL · pp. 38–39
- Group-sequential methods require prescheduled looks and a normal approximation, while active-ranking methods guarantee only their final grouping; this method guarantees intermediate reports under any stopping rule.Arviv et al. (2026) leave multiplicity across pairs open, and active-ranking algorithms choose comparisons adaptively.Source: F RELATED WORK IN DETAIL · p. 39
- Among anytime-valid approaches, Gu et al. (2026) use Bonferroni over ordered pairs and assume i.i.d. prompts; Arnold et al. (2026) certify only which models may be best.For fixed benchmarks, this paper’s validity rests only on random item order; its shortcut never certifies less than Gu et al.’s Bonferroni correction on the same wealths.Source: F RELATED WORK IN DETAIL · p. 39
- The construction draws on multiple comparisons with the best, step-down resampling, betting martingales, time-uniform confidence sequences, and Ville’s inequality.Certified tier 1 is the anytime-valid analog of multiple comparisons with the best; time-uniform guarantees cover when to stop or report an evaluation, not which model variants to disclose.Source: F RELATED WORK IN DETAIL · p. 39