Equivalence testing and run-count design for stochastic machine-learning experiments

• Small-run failures to reject difference do not establish equivalence, while run and benchmark-item variability can be comparable to reported improvements.Source: §1 · p. 2
• At n=2, the joint-energy-model methods’ 0.006 AUROC difference yielded TOST p=0.33, with simulated certification probability near 3%.Source: §1 · p. 2
• The protocol treats runs as units, separates seed and item variance across independent-draw and shared-benchmark settings, and combines Welch-TOST/CI duality with a two-level bootstrap for composite metrics.Source: §1 · p. 2
• The design formula for required runs is validated on a 270-cell simulation grid and benchmarked against exact pooled-model power.Source: §1 · p. 3
• Certifying ±0.01 at 80% power requires about 12 runs per method in the joint-energy-model case, even when the methods are truly identical.Source: §1 · p. 3
• The second case study tests the formula against real, non-Gaussian seed laws and shows the protocol resolving comparisons left undecided by two or three runs.Source: §1 · p. 3

• Multi-reader, multi-case ROC work combines hierarchy, equivalence, and sample-size methods, but treats human readers rather than retraining randomness of a learning method.Source: §2 · pp. 3–4
• Earlier ROC equivalence tests address fixed scoring rules and item-level uncertainty, while the present setting adds randomness from retraining.Source: §2 · p. 3
• Machine-learning benchmarking research studies difference testing, run-level variation, power, and resampling, but these lines generally target differences or uncertainty rather than equivalence certification.Source: §2 · p. 4
• The paper positions its contribution as transferring classical ingredients to training runs, making the shared-benchmark estimand explicit, and validating finite-sample behavior at small run counts.Source: §2 · pp. 4–5

3.1 Two-level model

• The two-level model separates seed-to-seed AUROC variation from item-level noise for independent test draws, while shared-test inference targets the realised collection and cannot isolate method–sample interaction.The seed-level sample variance captures conditional total per-run variance on a shared test set; the benchmark itself may require multiple collections or an explicit model of benchmark generation to assess.Source: §3.1 · pp. 5–7

3.2 Seed-level equivalence test

• TOST rejects a relevant difference only when both one-sided tests reject, equivalently when the (1 − 2α) confidence interval lies inside the pre-specified margin.The Welch components are approximate, and their realised size is checked by simulation; the paper recommends reporting the interval with the margin.Source: §3.2 · pp. 7–8

3.3 Two-level bootstrap

• The two-level bootstrap resamples runs and shared test-item indices to report uncertainty for composite per-run statistics, rather than to make the equivalence decision.Its finite-sample coverage is assessed by simulation, not claimed to be generally consistent: it is close to nominal when shared-test seed variance is comparable to residual image variance, while crossed resampling can overcount interaction variance and few-run seed-dominant settings can be anti-conservative.Source: §3.3 · pp. 8–9

4 Power analysis and the required number of runs

• The design formula inverts the power expression to find the smallest run count for a target power at a specified equivalence margin and true difference.For the stated formula, the design assumes equal allocation and a common per-run variance.Source: §4 · p. 9
• The shifted-t refinement substitutes central-t quantiles and distribution functions, using ν = 2(n − 1) as a design surrogate rather than exact TOST t theory.Against exact pooled-model power, its shortfall is at most 0.045 at n = 5, 0.018 at n = 8, and 0.014 at n = 12 in the design-relevant range.Source: §4 · p. 10
• At δ = 0.01, the design requires 12 runs per method for 80% power; required runs grow roughly as δ^-2 at tight margins.Once seed-level variance dominates, enlarging the test set cannot remove the seed-level floor.Source: Figure 1 · p. 11, §4 · p. 12
• At δ = 0.01, uncertainty in the case-study variance estimate yields an approximate 95% run-count interval spanning 4 to 3542 per method.The figure marks the point estimate of 12 runs per method, while recommendations below five are not relied upon.Source: Figure 1 · p. 11
• For approximate power at least 0.2 and n ≥ 5, simulation errors are at most 0.025, while recommendations below five runs should be raised to five.At n = 3, errors run in both directions; at n = 2, the approximation can predict near-zero power when the realised rate is 0.18.Source: §4 · pp. 11–12

5 Simulation study

• Power validation uses independent test draws; the shared-test-set variance change is absorbed by the plug-in variance, but its conditional-law shape is not separately validated.The non-Gaussian robustness grid tests centred exponential and scaled t3 seed effects.Source: §5 · p. 13
• Across exponential and heavy-tailed t3 seed laws, boundary size never exceeds 0.053, and design-range power overstatement stays within Monte Carlo resolution.The formula understates power by up to 0.05 under skewness and 0.125 under heavy tails in the design-relevant range.Source: §5 · p. 13
• Under a shared test set, the two-level interval is near nominal or mildly conservative when variance levels are balanced, but can under-cover with few runs and dominant seed variation.At σ = 0.01 and n = 2, coverage is 0.835 ± 0.012; it recovers to 0.94–0.95 by n = 5–8.Source: §5 · p. 13
• The naive per-image bootstrap under-covers when seed-level variation matters; at σ = 0.01 its coverage stays near 0.53 for every run count.Its interval width does not respond to σ, whereas the two-level interval reflects both seed and item variation.Source: §5 · p. 13
• The shifted-t approximation tracks realised TOST power where power is practically useful, is conservative near zero power, and misses in both directions at n = 3.The power grid uses δ = 0.01 and α = 0.05, with independent test draws for each run; realised size at the equivalence boundary is also assessed.Source: §5 · p. 12, Figure 2 · p. 14

Case-study setup

• The study compared two SGLD and two predictor–corrector runs using each run’s macro-average AUROC across five OOD test sets, with δ = 0.01.The retrospective margin was adopted at analysis time, rather than pre-registered.Source: §6 · pp. 14–15

Equivalence test

• At two runs per method, the Welch-TOST returned p = 0.334 and could not declare equivalence within ±0.01.Its 90% interval, [−0.032, +0.043], crossed the margin; the two-level bootstrap interval was treated as illustrative at this sample size.Source: §6 · p. 16

Retrospective power

• With the variance calibrated to the case-study runs, two-run certification probability was about 2.6%, making non-certification largely a design outcome.The calibrated simulation estimate was 0.026 ± 0.004; this is a model-based plug-in estimate, not a property of the design alone.Source: §6 · pp. 16–17

Design answer

• 12 runs per method are required for 80% power at δ = 0.01 under exact equality; the calibrated simulation achieved 0.811 power.The design formula predicted 0.812; both calculations assume a common per-run variance.Source: §6 · p. 17, Table 2 · p. 17
• At α=0.05, true Δ=0, and case-study s=0.0080, relaxing the margin to 0.02 brings cost to five runs per method (raw formula values 4–5), while certifying 0.005 takes 45.Source: §6 · p. 17, Table 2 · p. 17

Design uncertainty

• The plug-in run-count estimate is highly uncertain with two runs; a 4–6-run pilot and an upper confidence bound on pilot variance can reduce undersizing, at the cost of more runs.At five runs per method, the stated multiplicative 95% uncertainty in n* shrinks to ×0.5–×3.7 under the working model.Source: §6 · pp. 17–18

7 Second case study: real seed laws in two public collections

• The case study uses 1,000-seed CIFAR-10 and 25-run MultiBERTs collections; it measures accuracy, with each MultiBERTs run fixing its fine-tuning seed.The authors select RTE, SST-2, and MNLI to span test sets from 277 to 9,815 items.Source: §7 · pp. 18–19

7.1 Two-level structure

• Seed dispersion ranges from 0.002 on MNLI to 0.061 on RTE; for RTE, 80% power at a one-point margin requires 636 runs per method.On SST-2 and MNLI, adding item-level redraw variance to the shared-benchmark sizing variance would inflate the requirement from five to 20 runs per method on SST-2.Source: §7.1 · p. 19

7.2 The design formula under a real seed law

• The real-law formula’s error stays within ±0.05 in its design-relevant range; at zoo δ=0.0173, n*=12 gives 0.794 realised versus 0.826 predicted power.Below eight runs, the formula can be conservative; SST-2 agrees with predictions from three runs upward.Source: §7.2 · p. 20

7.3 Two-method comparisons

• At 10 matched pairs, both zoo contrasts certify equivalence at δ=0.01; MultiBERTs’ SST-2 learning-rate gap (+0.0084) is unresolved at 0.01 but equivalent at 0.02.For the learning-rate comparison, the interval is [+0.0067, +0.0101], just crossing the 0.01 margin.Source: §7.3 · pp. 20, 22
• With two matched pairs, zoo equivalence certifications occur in 2–4% of subsets, rising to 3–7% with three pairs.At five pairs, the shares are 7% and 9%; these are exact shares of the observed finite pool, not fresh-replication rates.Source: §7.3 · p. 21
• Point-estimate pilots certify in 0.662 of replications at δ=0.0173 and 0.598 at δ=0.01; upper-bound variance sizing raises these rates to 0.802 and 0.779, respectively.The upper-bound rule remains below the fixed-size reference at the tighter margin and increases mean total runs.Source: §7.3 · p. 22
• Unpaired Welch tests preserve the reported verdicts but narrow intervals for negatively correlated contrasts; the eight comparisons at two margins have no familywise control.Separate verdicts across multiple contrasts therefore require multiplicity adjustment.Source: §7.3 · pp. 22–23

7.4 Scope of this case study

• The real-law checks are limited by the tiny CIFAR-10 testbed, only 25 MultiBERTs runs, and unverified independence of GLUE test sets from model selection.Subsampled rates quantify resampling uncertainty within the observed pool, not uncertainty about the seed law.Source: §7.4 · pp. 23–24

Approximations in the power theory

• The power formula is accurate in its design-relevant range, but under the heavy-tailed zoo seed law its error reached 0.05 conservatively and 0.03 in the other direction.Realised size stayed near α; recommendations below five runs are raised to five.Source: §8 · p. 24

Bootstrap validity is regime-dependent

• The two-level bootstrap has near-nominal coverage only with balanced variance levels; at n ≤3, the authors recommend a seed-level Welch interval instead.The bootstrap assessment covers one binormal generator family, one test-sample size, B = 400, and equal method dispersions; the Welch interval is approximate.Source: §8 · p. 24

Dependence between test sets

• The macro-average is a single-statistic estimand, not a certification of equivalence on individual OOD sets, whose per-set tests remain dependent and lack a multiplicity claim.Mean per-set differences reach +0.028 for SVHN and −0.012 for LSUN-R, opposite in sign and beyond the margin; their per-set equivalence claims need their own margins and multiplicity-aware procedure.Source: §8 · pp. 24–25

Uncertainty of the plug-ins

• At n = 2, the pooled s2 has roughly one effective degree of freedom, making the resulting run-count estimate an order-of-magnitude statement.A two-stage pilot using a point estimate achieved 0.60–0.66 against nominal 0.80; using the upper 80% variance confidence bound raised this to 0.78–0.80 at about 1.5 times the point-estimate run count, in one collection, two margins, and ∆=0.Source: §8 · p. 25

Beyond AUROC

• The seed-level model and TOST extend beyond AUROC, but strongly skewed or heavy-tailed run means at small n need simulation checks before trusting the design formula.Rare large-shift failure mixtures can lead five-run studies to miss failures and certify too often; inspect per-run values, since a short failure-free pilot cannot exclude a rare component.Source: §8 · p. 25

Choice of the margin

• The margin δ is a domain decision fixed before the experiment; reporting the (1−2α) interval alongside it keeps the conclusion transparent under any reader’s margin.The paper treats δ as the smallest difference that would change the practical decision, not as a resource-based fallback.Source: §8 · pp. 25–26

9 Conclusions

• About 12 runs per method are needed to certify ±0.01 at 80% power even under exact equality; the published n = 2 comparison had roughly 3% realised certification probability.On two public collections, the formula tracks realised certification probability within 0.05 in its design-relevant range and resolves comparisons left undecided by two or three runs.Source: §9 · p. 26
• The two-level protocol combines a pre-fixed margin, per-run AUROCs, Welch-TOST and its dual confidence interval, and a two-level bootstrap for composite statistics.Its power theory gives a run-count formula; once image-level noise is subordinate, the price is dominated by seed-level dispersion, which test-set enlargement cannot reduce.Source: §9 · p. 26
• In the two-stage experiment, using an upper confidence bound for pilot variance raised achieved rates from 0.60 and 0.66 to 0.78 and 0.80 at about half again the point-estimate rule’s run count.The paper identifies multiplicity-aware equivalence, adaptive sequential designs, and inner-level estimators beyond AUROC as continuations.Source: §9 · p. 26

Funding.

• The authors received no specific funding for this work.Source: Funding · p. 27

Competing interests.

• The authors declare no competing interests.Source: Competing interests · p. 27

Ethics approval.

• The study involved no human participants or animals and used simulated data and publicly available benchmark datasets only.Source: Ethics approval · p. 27

Consent to participate.

• No human participants were involved in the study.Source: Consent to participate · p. 27

Consent to publish.

• The manuscript contains no data, images, or videos relating to an identifiable person.Source: Consent to publish · p. 27

Data availability.

• Case-study predictions, checkpoints, derived per-run results, and the listed benchmark datasets are available from the cited public sources.The MultiBERTs prediction files carry no stated licence; the accompanying repository is Apache-2.0.Source: Data availability · p. 27

Code availability.

• The equivtest package, simulation and case-study scripts, and figure builder are available in a GitHub repository archived on Zenodo.Reproducing the checkpoint-based zoo analysis additionally requires PyTorch and torchvision.Source: Code availability · p. 27

Use of AI-assisted tools.

• The first author used Claude and GPT for drafting, editing, literature organization, and developing and debugging simulations and case-study code.Source: Use of AI-assisted tools · p. 27
• The authors state that the research ideas, methods, analyses, results, and conclusions are their own and that they take full responsibility.Source: Use of AI-assisted tools · p. 27