PaperlayerView exact revision on arXiv

Source-linked AI summary

Anytime-Valid LLM Leaderboards via Benchmark-weighted and Block-Factorized e-Processes

Hongfu Gao, Songxin Zhang, Zejian Xie, Bingyi Jing, Zhou Wang, Yiming Liu

arXiv:2609.32248v1cs.LG

TL;DR

Run-to-run variability makes leaderboard score differences insufficient to establish model superiority, especially when leaderboards are repeatedly updated. BB-EDGE builds an anytime-valid directed graph of pairwise benchmark advantages using block-factorized evidence and weight-proportional stakes. Experiments report anytime FWER control and high statistical power, while the guarantees remain scoped to the fixed benchmark and protocol.

  • Problem

    Run-to-run variability means a higher empirical mean alone does not establish higher expected performance, while existing approaches lack anytime-valid guarantees under repeated monitoring or data-dependent stopping.

    Source: §1 · pp. 1–2

  • Method

    BB-EDGE factorizes empirical-Bernstein evidence over protocol-defined blocks, uses weight-proportional stakes, and applies direct e-Holm to certify pairwise advantages as directed edges with anytime FWER control.

    Source: §1 · p. 2

  • Results

    BB-EDGE achieves anytime-valid FWER control with the highest statistical power in the reported synthetic and real-world experiments.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The resulting confidence graph supports anytime-valid Top-k certification and simultaneous rank intervals for model selection and ranking uncertainty quantification.

    Source: §1 · p. 2

    BB-EDGE’s anytime-valid guarantees are conditional on the fixed benchmark and protocol and do not extend to out-of-distribution tasks.

    Source: §7 · p. 10

Abstract

from arXiv · show

Large language model (LLM) leaderboards compare model capabilities by ranking models according to their mean performance on fixed benchmarks. However, variability in evaluation outcomes across runs may produce unsupported claims of model superiority on the benchmark, a risk compounded by leaderboard updates. In this paper, we propose BB-EDGE (Benchmark-Weighted and Block-Factorized e-processes for Directed Graph Evaluation), a principled framework that represents an LLM leaderboard as a directed graph whose edges certify pairwise mean-performance advantages, with anytime-valid family-wise error rate (FWER) control. Concretely, for each direction, BB-EDGE constructs an empirical-Bernstein e-process by factorizing evidence over protocol-defined blocks and assigning stakes proportional to the corresponding block weights, then applies direct e-Holm across these $e$-processes to certify directional advantages as edges. Theoretically, we characterize weight-proportional linear stakes under heterogeneous benchmark-average nulls and prove anytime FWER control under arbitrary within-block and cross-pair dependence. BB-EDGE further supports anytime-valid Top-$k$ certification and simultaneous rank intervals. Extensive experiments on synthetic data and four real-world benchmarks demonstrate that BB-EDGE maintains anytime FWER control while achieving high efficiency.

1 INTRODUCTION

LLM leaderboard scores vary across runs, so higher empirical means alone do not establish expected superiority. BB-EDGE provides anytime-valid pairwise certifications with family-wise error control and performs well in synthetic and real-world evaluations.

1 INTRODUCTION

  • Across runs, stochastic responses and evaluation scores mean a higher empirical mean alone does not establish higher expected performance, risking unsupported superiority claims.Repeated leaderboard updates make anytime-valid comparisons important; fixed-budget approaches do not provide guarantees under repeated monitoring or data-dependent stopping.Source: §1 · p. 1
  • An i.i.d. item-stream procedure can falsely certify benchmark-average advantages under heterogeneous item differences or dependent dialogue observations, and same-model comparisons reveal false certifications on real benchmarks.The paper also extends BB-EDGE to anytime-valid Top-k certification and simultaneous rank intervals.Source: §1 · p. 2
  • BB-EDGE builds a directed confidence graph from pairwise mean-performance advantages, using block-factorized empirical-Bernstein e-processes, weight-proportional stakes, and direct e-Holm for anytime FWER control.Its theory establishes that weight-proportional stakes are necessary and sufficient for nonpositive linear drift over the full benchmark-average null.Source: §1 · p. 2
  • 41 of 45 GSM8K pairwise comparisons (91.1%) were certified after 2 replicates, using 67% fewer replicates than the strongest competing baseline.Across simulations, BB-EDGE maintained anytime-valid FWER control and had the highest power among FWER-controlling baselines in the examined scenarios.Source: §1 · p. 2

2 RELATED WORK

Prior leaderboard work addresses benchmark construction, ranking, and uncertainty, while online inference methods address repeated evaluation in other settings. The paper distinguishes its target as anytime-valid inference for pairwise mean scores on fixed benchmarks.

LLM Leaderboards

  • Common leaderboard practice reports a ranking from a single average score without quantifying which pairwise advantages are statistically supported.Evaluation outcomes can vary across runs because of sampled decoding and stochastic LLM-based judging.Source: §2 · p. 2

Model Ranking

  • Existing ranking and uncertainty methods generally target a fixed evaluation budget, so they do not protect against repeated examination of rankings.These include alternative ranking criteria, Holm-adjusted paired tests, and conformal rank-uncertainty methods.Source: §2 · pp. 2–3

Anytime-Valid Inference

  • Online e-process methods evaluate individual LLMs or infer pairwise human-preference probabilities, but their i.i.d. item-stream formulation does not directly accommodate heterogeneous fixed-benchmark effects or arbitrary within-dialogue dependence.This paper instead targets pairwise mean-performance differences on fixed benchmarks and controls false edges across model pairs and replicates.Source: §2 · p. 3

3 STATISTICAL FRAMEWORK FOR ANYTIME-VALID LLM LEADERBOARDS

The framework evaluates models sequentially on a fixed benchmark and certifies directed pairwise advantages rather than treating empirical rankings as definitive. It defines anytime validity as simultaneous error control at every completed replicate.

Sequential Evaluation for LLM Leaderboards

  • Models are evaluated on a fixed benchmark under a standardized protocol, with successive complete replicates scoring every model on every item.The protocol specifies prompts, decoding configuration, available tools, grading rules, and dataset structure.Source: §3 · p. 3
  • Single-run rankings can change across runs; on GSM8K, the single-run ranking differed from the 20-replicate ranking in 9 of 20 replicates, with reversals among three closely matched models.The framework therefore targets statistical certification, with a prespecified margin τ for practically meaningful advantages.Source: §3 · p. 3

Ranking via Certified Edges

  • A certified directed edge indicates that one model exceeds another’s fixed-benchmark mean performance by more than the prespecified margin τ; pairs without a certified direction remain unresolved.A directed path of length d supports a difference greater than dτ.Source: §3 · pp. 3–4

Anytime Validity and Power

  • Anytime validity requires FWER_r ≤ α at every replicate, allowing the graph to be examined after each completed replicate while maintaining simultaneous error control.Efficiency is measured by the expected proportion of true edges certified.Source: §3 · p. 4

4 METHODOLOGY

BB-EDGE addresses arbitrary dependence within protocol-defined blocks and heterogeneous item-level advantages while preserving the benchmark-average target. Its construction factorizes evidence across blocks, assigns weight-proportional stakes, and applies direct e-Holm to certify edges.

4 METHODOLOGY

  • BB-EDGE addresses two challenges in fixed-benchmark inference: arbitrary within-block dependence and item-level performance differences that vary across the benchmark.The directional null constrains the benchmark-weighted average, not each block mean individually.Source: §4 · pp. 4–5
  • The benchmark is partitioned into prespecified blocks based on protocol-induced dependence, changing evidence resolution without changing the benchmark-level estimand.One block permits arbitrary within-replicate dependence; singleton blocks permit item-level factorization under conditional independence; intermediate partitions accommodate within-block dependence.Source: §4 · p. 4
  • Because block means need not individually satisfy the null, arbitrary block-specific stakes can induce positive drift; proportionality to benchmark weights is necessary and sufficient for nonpositive drift across admissible block means.The construction uses predictable block estimates in its empirical-Bernstein contributions.Source: §4 · p. 5
  • BB-EDGE factorizes empirical-Bernstein evidence across protocol-defined blocks, using proportional stakes to preserve the heterogeneous benchmark-average null and block-specific predictable estimates for variance adaptation.This construction does not change the target hypothesis.Source: §4 · p. 5
  • Direct e-Holm combines evidence across the directional family and remains valid under arbitrary dependence among pairwise e-processes; certified edges form a graph supporting anytime-valid Top-k certification and rank intervals.Reachability can imply a larger mean-performance difference, while unreachable pairs remain unresolved.Source: §4 · p. 5

5 THEORETICAL ANALYSIS

The theory makes weight-proportional stakes valid for heterogeneous benchmark-average nulls and combines the resulting e-processes with direct e-Holm for anytime false-edge control. Its guarantees depend on the stated sampling and design conditions, while the partition determines evidence resolution rather than the estimand.

Validity and assumptions

  • The guarantees assume independent, identically distributed complete panels across runs and conditionally independent within-run block vectors, with design choices fixed in advance or selected using independent data.Predictions must be fixed before each replicate and lie in [0, 1].Source: §5 · p. 6
  • Weight-proportional stakes are necessary and sufficient among nonnegative allocations for nonpositive linear drift over the full benchmark-level null.Individual blocks may favor either model even when the benchmark-weighted average satisfies the null.Source: §5 · p. 6

Anytime graph guarantees

  • With Assumption 1, direct e-Holm yields an acyclic graph whose edges and path implications are true margin claims with probability at least 1 − α.The bound applies across directions and replicates without requiring independence among directional e-processes.Source: §5 · p. 6

Resolution and design constraints

  • A partition changes evidence resolution, not the benchmark estimand: one block allows arbitrary within-replicate dependence, while singleton blocks require justified item-level independence.Intermediate partitions accommodate within-block dependence; resolution must be fixed in advance, not selected after seeing which graph reports more edges.Source: §5 · p. 6

Certification speed

  • A positive-weight grid component with persistently positive average conditional log growth yields eventual certification of an alternative direction.The growth expression combines benchmark-average advantage with a term accounting for blockwise prediction error.Source: §5 · p. 7

6 EXPERIMENTS

In synthetic experiments, BB-EDGE maintained anytime FWER control and gained power faster than the compared factorization and penalty baselines. Real-benchmark analyses produced certified comparison graphs and reported fewer queries for a given GSM8K analysis, alongside same-model diagnostics and Top-k and rank-interval results.

6.1 SYNTHETIC DATA ANALYSIS; 6.2 REAL DATA ANALYSIS

  • Synthetic experiments used 10 models on 100 items, with repeated configurations and dependence and heterogeneity mechanisms; the real evaluation covered 10 open-source models on four benchmarks with task-specific metrics.The synthetic design included within-block and cross-model dependence, while the real benchmarks included MMLU-Pro, GSM8K, MT-Bench, and ACEBench.Source: §6.1 · p. 7, §6.2 · p. 8

6.1 SYNTHETIC DATA ANALYSIS

  • 90% power arrived within 56 replicates under item-level heterogeneity for BB-EDGE, versus 95 for BFH-e-Holm; BB-EDGE also maintained empirical anytime FWER control across examined settings.Figure 1 compares FWER and power over replicates across i.i.d., dependence-only, heterogeneity-only, and combined settings.Source: §6.1 · p. 7, Figure 1 · p. 8
  • BB-EDGE continued to perform well with L = 20, benchmark sizes N = 50 or 500, and margins τ = 0.01 or 0.03; under varied correlation and block size, it controlled FWER with highest power.BB-EDGE-PI performed similarly, while AsympCS-Bonf and SERPANT-R under within-block dependence exceeded the target in the reported comparison.Source: §6.1 · pp. 7–8

6.2 REAL DATA ANALYSIS

  • On GSM8K, Granite-4.1-8B tops the certified graph, while Gemma-4-E4B-It, Llama-3.1-8B-Instruct, and MiMo-7B-RL remain mutually unresolved; on ACEBench, the highest-mean Gemma and Qwen models remain uncertified against each other.Increasing τ to 0.01 removes the Granite-to-Phi edge because their exact-match difference, 0.0071, does not exceed the required margin.Source: §6.2 · p. 8, Figure 2 · p. 9
  • BB-EDGE used 66.7% fewer queries than BFH-e-Holm on GSM8K; on ACEBench it certified Top-4 after 4 replicates, versus 17 for BFH-e-Holm and no certificate from CR-EDGE within 20.The GSM8K comparison used six replicates and 79,140 queries; reported rank intervals were narrower for BB-EDGE.Source: §6.2 · p. 9, Figure 3 · p. 9
  • Across 100 same-model splits, SERPANT produced more false certifications than BB-EDGE; this reused-replicate diagnostic does not estimate repeated-evaluation FWER.The split compared two groups of 10 from each model’s 20 observed replicates.Source: §6.2 · pp. 8–9

7 CONCLUSION

The conclusion presents BB-EDGE as an anytime-valid method for certifying model mean-performance advantages on fixed-benchmark leaderboards. Its guarantees do not extend to out-of-distribution tasks.

7 CONCLUSION

  • BB-EDGE partitions benchmarks by protocol, builds block-factorized empirical-Bernstein e-processes with weight-proportional stakes, and applies direct e-Holm to certify advantages.The method targets fixed-benchmark leaderboards.Source: §7 · p. 9
  • Experiments on synthetic and real-world benchmarks demonstrate anytime-valid FWER control with the highest statistical power.Source: §7 · p. 10
  • The guarantees are conditional on the fixed benchmark and protocol and do not extend to out-of-distribution tasks.Item-level anytime-valid inference accommodating heterogeneous item effects and cross-item dependence is left for future work.Source: §7 · p. 10

A ANYTIME-VALID INFERENCE FOR TOP-k CERTIFICATION

The pilot-targeted procedure selects a candidate set, tests its cross-set comparisons using independent confirmatory replicates, and certifies it only after rejecting the required hypotheses. It controls false Top-k certification, and experiments report faster certification than baselines that often inflate FWER.

A ANYTIME-VALID INFERENCE FOR TOP-k CERTIFICATION

  • An independent pilot fixes the candidate set, and confirmatory testing covers only the k(L − k) directions from that set to models outside it.Pilot scores are excluded from the confirmatory evidence products.Source: A ANYTIME-VALID INFERENCE FOR TOP-k… · p. 15
  • The procedure applies direct e-Holm to the cross-set family and certifies the candidate set only when all required hypotheses are rejected.Theorem 2 preserves anytime FWER control for the pilot-selected family.Source: A ANYTIME-VALID INFERENCE FOR TOP-k… · p. 15
  • An incorrect pilot candidate is allowed, but its probability of being certified is at most α.This guarantee does not bound the error probability conditional on a certificate being issued.Source: A ANYTIME-VALID INFERENCE FOR TOP-k… · p. 16
  • Both BB-EDGE variants had estimated FWER_R = FCP_R = 0; relative to the strongest baseline, full-graph BB-EDGE reduced mean CR by 44.7%–69.9%.Pilot-targeted BB-EDGE reduced mean confirmatory CR by 62.3%–83.7% while testing 24 cross-set directions.Source: A ANYTIME-VALID INFERENCE FOR TOP-k… · p. 16, Table 1 · p. 16

B ANYTIME-VALID INFERENCE FOR RANK INTERVALS

BB-EDGE constructs rank intervals from the confidence graph to represent ranking uncertainty rather than force a unique ordering when differences are not significant. The intervals simultaneously cover all models’ true benchmark ranks at every completed replicate with probability at least 1 − α.

B ANYTIME-VALID INFERENCE FOR RANK INTERVALS

  • The rank interval uses certified comparisons in the graph: its lower endpoint counts models certified to outperform the target, and its upper endpoint excludes models it certifies it outperforms.Rank 1 denotes the highest benchmark mean.Source: B ANYTIME-VALID INFERENCE FOR RANK… · p. 16
  • With probability at least 1 − α, the intervals simultaneously cover all models’ true benchmark ranks at every completed replicate.The guarantee holds conditional on (B, π) when assumptions hold for the entire sequence.Source: B ANYTIME-VALID INFERENCE FOR RANK… · p. 17
  • At replicate r = 15 under item-level heterogeneity, BB-EDGE achieved anytime coverage 1.00 with mean width 1.633, 62.6% narrower than valid baseline BEB-Holm (4.369).Among methods maintaining the desired anytime coverage, BB-EDGE produced the narrowest intervals.Source: B ANYTIME-VALID INFERENCE FOR RANK… · p. 17

C.1 UPDATED BENCHMARK ITEMS

For changing benchmarks, BB-EDGE freezes each expanded version, restarts its evidence process, and allocates error across versions. It also supports later model additions through staged error spending, under an independence condition on their inclusion and design.

C.1 UPDATED BENCHMARK ITEMS

  • Each updated benchmark is frozen for its epoch, and a fresh e-process with version-specific budget α_v is restarted; requiring cumulative budgets no greater than α extends FWER control across versions.Each claim remains specific to the benchmark version on which it was tested.Source: §C.1 · pp. 17–18
  • After benchmark updates from 40 to 70 and then 100 items, both epoch-reset and fixed-benchmark BB-EDGE kept empirical FWER below 0.05 in all four settings.Epoch-reset power began lower after resets but caught up with and eventually exceeded the fixed-benchmark reference, especially under dependence and combined dependence and heterogeneity.Source: §C.1 · p. 18, Figure 5 · p. 18

C.2 NEWLY ADDED MODELS

  • For anticipated model additions, prespecified stage-wise error spending supports combined-graph FWER control when inclusion and design choices are independent of the inference panels.Previously certified edges are retained; the current experiments allocate the full error budget to the initial leaderboard.Source: §C.2 · pp. 18–19

D.2 THE CONDITIONAL EMPIRICAL-BERNSTEIN INEQUALITY

  • A conditional empirical-Bernstein inequality supplies the bounded-variable ingredient for the predictable block-factorized construction.The stated proof gives a scalar argument to specify the normalization used.Source: §D.2 · p. 20

D.3 PROOF OF THEOREM 1

This section proves anytime-valid guarantees for BB-EDGE’s directional and pilot-selected tests, including rank intervals and adaptive or staged certification, while allowing dependent directional processes and averaging over independent initial design or pilot information. It also establishes valid acyclicity, rank, and top-set conclusions from certified graph edges, and gives a fixed-grid condition for eventual certification of a true directional advantage.

D.3 PROOF OF THEOREM 1

  • Theorem 1’s directional processes are nonnegative supermartingales, yielding anytime FWER at most α under arbitrary dependence among directions.The proof conditions on initial design information, applies a finite stopping argument, and extends the bound to any finite-time crossing.Source: §D.3 · pp. 22–24
  • On the simultaneous guarantee event, certified edges are acyclic, imply margin claims along paths, and yield rank intervals and valid top-set certificates.The rank definition counts strictly better means and permits ties; ties within a certified top set or its complement are allowed.Source: §D.3 · p. 24

D.4 PROOF OF THEOREM 2

  • For Theorem 2, pilot-selected cross-set nulls are tested against the benchmark-average mean, without requiring each block mean to satisfy the null separately.Pilot runs determine the selected family but do not contribute confirmatory evidence; confirmatory runs retain the conditional evidence-process property.Source: §D.4 · pp. 25–26
  • Theorem 2 controls false-rejection probability at α conditional on the pilot and benchmark, and also after averaging over pilot outcomes.The argument uses a direct e-Holm implication and a finite stopped-sum bound; dependence among directional processes does not invalidate the sum argument.Source: §D.4 · pp. 26–27
  • A certified pilot-selected set has every member’s benchmark mean above every outsider’s, while an incorrect pilot selection is allowed without assuming the selected set meets the margin.The conditional bound controls issuing a certificate when the candidate fails the margin, including by any run up to R.Source: §D.4 · p. 27
  • The guarantee bounds the probability of an incorrect certificate, not the error probability conditional on certification; tied true benchmark means prevent the corresponding direction from being rejected.This distinction holds after conditioning on the pilot data.Source: §D.4 · p. 27

D.5 PROOF OF COROLLARY 1

  • For the first R runs, the true rank lies in the simultaneous rank interval, and the certified graph is acyclic.The bounds also hold simultaneously for all run counts when the assumptions hold for an entire sequence.Source: §D.5 · pp. 28–29

D.6 PROOF OF THEOREM 3

  • Across adaptive epochs, the probability of any false certificate is bounded by the total error budget, even without independence across epochs.The bound holds for finite run budgets and when evaluation stops after a data-dependent number of runs.Source: §D.6 · p. 32

D.7 PROOF OF COROLLARY 2

  • Replaying augmented panels supports valid staged certifications despite reuse of stored outcomes; on the no-error event, combined-graph path implications, rank intervals, and Top-k certificates are valid.The stage error events may be arbitrarily dependent, with their combined error controlled by countable subadditivity.Source: §D.7 · p. 33

D.8 THE GRID-DEPENDENT POWER STATEMENT

  • When a tested direction has ΔB_a,b > τ and a positive-probability fixed-grid component has positive conditional expected log increments almost surely, its rejection probability tends to one.The direction is certified at every sufficiently large completed replicate almost surely.Source: §D.8 · pp. 33, 35

E ABLATION STUDY

The experiments examine BB-EDGE under varied model counts, item sizes, and prespecified margins; the leaderboard results also show that close mean scores and item-level heterogeneity can leave model comparisons unresolved.

E ABLATION STUDY

  • The ablations examine a larger model set (L = 20), benchmark sizes N = 50 and N = 500, and prespecified margins τ = 0.01 and τ = 0.03.The provided figure captions identify these conditions but do not report their outcome values.Source: Figure 6 · p. 36, Figure 7 · p. 36, Figure 8 · p. 37
  • Figure 11 compares BB-EDGE with BB-EDGE-PI, AsympCS-Bonf, and SERPANT-R.The supplied figure caption names the methods but does not state the comparison results.Source: Figure 11 · p. 38
  • Figure 12 varies within-block correlation from 0.1 to 0.7 across blocks and prespecified block sizes from 1 to 10.The caption specifies the heterogeneous block structures examined, not their results.Source: Figure 12 · p. 38

F.1 LEADERBOARD

  • Gemma-4-E4B-It and Qwen3.5-9B score 0.7751 and 0.7790 on ACEBench, with a gap smaller than their respective standard deviations of 0.0104 and 0.0085.BB-EDGE leaves the pair unresolved at r = 20; item-level advantages can also differ in sign despite near-zero benchmark-average differences.Source: §F.1 · p. 39
  • Tables 2 and 3 report mean accuracy and standard deviation on MMLU-Pro and mean exact match and standard deviation on GSM8K, respectively.The supplied table captions do not include model values or rankings.Source: Table 2 · p. 39, Table 3 · p. 39

F.2 FULL-RANKING CONFIDENCE GRAPH

The confidence graphs resolve many model comparisons while retaining uncertainty for close pairs, and the paper extends these comparisons to Top-4 certification, rank intervals, and closed-source APIs. The reported resolution depends on benchmark, margin, and evaluation budget.

F.2 FULL-RANKING CONFIDENCE GRAPH

  • At r = 20, MMLU-Pro forms a fully resolved chain led by Qwen3.5-9B, while MT-Bench leaves three pairs unresolved after multiplicity correction.With τ = 0.01, fewer edges are certified; MMLU-Pro becomes a partial order, though clear partial rankings remain across all four benchmarks.Source: §F.2 · pp. 39, 41, Figure 15 · p. 41, Figure 16 · p. 42
  • On GSM8K, the one-block graph resolves 24 of 45 comparisons at r = 100, while the item-factorized graph resolves 41 after two replicates.The one-block approach requires roughly two orders of magnitude more replicates and is recommended only when conditional independence across blocks cannot be justified.Source: §F.2 · p. 41, Figure 17 · p. 42

F.3 TOP-k CERTIFICATION

  • Qwen3.5-9B and Phi-4 appear in every benchmark’s certified Top-4, while Granite-4.1-8B appears in three of four.The remaining certified slot varies by benchmark.Source: §F.3 · p. 43, Table 6 · p. 43
  • The two BB-EDGE variants require the fewest replicates for Top-4 certification; full-graph BB-EDGE is fastest on MMLU-Pro and GSM8K, while the pilot-targeted variant is faster on MT-Bench and ACEBench.On MT-Bench, the pilot-targeted variant certifies in 3 replicates versus 5 for full-graph BB-EDGE.Source: §F.3 · p. 43, Table 7 · p. 43

F.4 RANK INTERVAL

  • BB-EDGE uniquely determines every model’s rank on MMLU-Pro, but retains unresolved ranks on GSM8K, MT-Bench, and ACEBench.For example, unresolved intervals on GSM8K include ranks 5–7; the figure depicts inferred rank intervals, with overlaps indicating unresolved ranking.Source: §F.4 · p. 43, Figure 18 · p. 44

F.5 CLOSED-SOURCE APIS

  • The closed-source API evaluation is restricted to ACEBench Normal Multi-Turn because of limited resources.The reported graph uses five API models at r = 20.Source: §F.5 · p. 43
  • On ACEBench, GLM-5.3-Flash and Qwen3.8-Flash jointly occupy the top tier, with each certified above one of two middle-ranked APIs, which are both above MiniMax-M3.The graph has a symmetric two-branch structure that converges on a single lowest-ranked model.Source: §F.5 · p. 44, Figure 19 · p. 44

G.1 DATASET

The appendix describes benchmark-specific block structures, comparison baselines, evaluation metrics, experimental settings, and query templates. Its inference setup permits arbitrary dependence within blocks while relying on conditional independence across separately executed blocks.

G.1 DATASET

  • The four public benchmarks span different task types and block structures: singleton-item blocks for MMLU-Pro and GSM8K, dialogue blocks for MT-Bench, and self-contained dialogue blocks for ACEBench.For MT-Bench, the two turns of each dialogue form a block; ACEBench blocks contain two or three scored tool-call steps.Source: §G.1 · p. 45, Table 9 · p. 45
  • The dataset partitions allow arbitrary within-block dependence and assume conditional independence across separately executed blocks; permutation tests found no significant cross-block dependence for any model after Holm correction.The smallest raw p-value was 0.073 for GSM8K; the text says the tests support, but do not establish, the prespecified partitions.Source: §G.1 · p. 45

G.2 BASELINES

  • The baselines include uncorrected empirical means, fixed-time tests, alternative e-process constructions, and item-stream preference procedures, which differ in uncertainty handling, factorization, stakes, or monitoring.For example, BFH-e-Holm replaces BB-EDGE’s empirical-Bernstein penalty with a Hoeffding penalty, while CR-EDGE treats a complete benchmark replicate as one block.Source: §G.2 · p. 46
  • SERPANT’s item-stream adaptation need not control the benchmark-average target under heterogeneous item advantages, and on graded MT-Bench its preference target need not equal the mean-score target.Its certification times are descriptive when its item-stream assumptions or estimand do not match the benchmark-average target.Source: §G.2 · p. 46

G.3 EVALUATION METRIC

  • The evaluation metrics define Top-k certification events, false-certification probability, certification probability and time, and anytime coverage and width of rank intervals.Pointwise 95% binomial Monte Carlo intervals quantify simulation uncertainty, not an additional anytime guarantee.Source: §G.3 · p. 47

G.4 EXPERIMENT DETAILS

  • The experiments use 41 equally spaced stakes over [0, 0.95] with uniform mixture weights, and initialize block-specific running-mean predictions at µ0 = (1 + τ)/2.The protocol choices support, but do not by themselves establish, the conditional-independence assumptions.Source: §G.4 · p. 48

G.5 QUERY TEMPLATE

  • The appendix supplies query templates for MMLU-Pro, GSM8K, MT-Bench, and ACEBench, plus a judge-prompt structure for MT-Bench single-answer grading.The supplied captions identify the template topics but do not give their contents.Source: Table 10 · p. 48, Table 11 · p. 49, Table 12 · p. 49
Loading 2609.32248v1…