Accounting for Variance in Machine Learning Benchmarks

• Uncontrolled factors can dominate meaningful performance differences between algorithms, undermining benchmark conclusions.Source: §1 · p. 1
• Robust comparisons require repeatedly training complete pipelines across hyperparameter optimization and random seeding, but computational budgets often permit only a few model fits.Source: §1 · p. 1
• Prior comparison methods often focused on data-sampling variance while ignoring extrinsic variation, including hyperparameter choices and pipeline procedures.Source: §1 · pp. 1–2
• The paper frames uncontrolled hyperparameter optimization statistically through explicit bias and variance, then evaluates decision criteria and their error rates under noise.Source: §1 · p. 2
• The recommendations randomize as many variation sources as possible, account for variance when comparing average performance, and favor resampling over fixed test sets.Source: §1 · p. 2

• Benchmark performance is a realization of a random variable because both the dataset and the learning pipeline contain uncontrolled fluctuations.Source: §2 · p. 2
• The framework models variation from data sampling, initialization, ordering, augmentation, and hyperparameter optimization across the complete learning pipeline.Source: §2.1 · pp. 2–3
• Hyperparameters govern regularization, architecture, and optimizer settings, while their optimization introduces additional arbitrary variation through search procedures and data splits.Source: §2.1 · pp. 2–3
• Bootstrapping the data is generally the largest variance source, whereas model initialization is typically less than 50% of bootstrap variance and comparable to stochastic-gradient-descent ordering.Source: §2.2 · p. 4
• A binomial model provides a simple account of classification sampling noise, but correlated errors reduce effective degrees of freedom and widen the distribution.Source: §2.2 · p. 4
• Hyperparameter choice induces sizable variance, with three optimization methods averaging as much variance as commonly studied weight initialization.Source: §2.2 · p. 4
• Total benchmark variance is on the order of published state-of-the-art increments, so it must be accounted for in pipeline comparisons.Source: §2.2 · p. 5

3.1 Multiple data splits for smaller detectable improvements

• Fixed training and test sets estimate expected empirical risk noisily because finite data sampling limits information about performance on new data.The resulting uncertainty can be comparable to typical improvements, and test-set size may limit estimation quality.Source: §3.1 · p. 5
• Increasing the number of splits k enables detection of smaller performance differences, but the ideal estimator remains computationally expensive.The estimator averages performance over multiple train–test splits, while reliable comparisons must account for its uncertainty.Source: §3.1 · p. 5

3.2.1 Ideal estimator: sampling multiple HOpt

• The ideal estimator randomizes all learning and hyperparameter sources independently, is unbiased, and requires O(k · T) model fittings.Its variance decreases as k grows, with limk→∞ Var(ˆµ(k)) = 0.Source: §3.2.1 · p. 6

3.2.2 Biased estimator: fixing HOpt

• The biased estimator reuses hyperparameters from one optimization, requires O(k + T) fittings, and can remain noisy as k increases because fixed sources induce correlation.When the average correlation ρ is large, the correlation term can dominate its variance, so additional splits do not adequately improve the estimate.Source: §3.2.2 · p. 6
• Accounting for more variation sources reduces the biased estimator’s correlation and moves it closer to the ideal estimator in most case studies.Randomizing initialization alone gives only a small improvement, while randomizing data and all non-HOpt sources performs substantially better.Source: §3.2.2 · pp. 6–7, §3.3 · p. 7

3.3 The cost of ignoring HOpt variance

• IdealEst(k = 100) takes 1 070 hours, whereas each FixedHOptEst(k = 100) takes 21 hours.The study reports that the cheaper estimator is better than the status quo and suitable for statistical model comparisons without prohibitive costs.Source: §3.3 · p. 7

• The average-difference criterion does not account for false detections, making random chance difficult to distinguish from true impact.Source: §4.1 · p. 8

Probability of outperforming

• Probability of outperforming compares performance distributions across fluctuations and declares A significantly better when P(A > B) ≥ γ.The method uses a threshold γ for how often A must outperform B.Source: §4.1 · p. 8
• The proposed test uses confidence intervals and requires both statistical significance and meaningful improvement to conclude that A is better than B.The lower confidence bound tests whether P(A > B) exceeds 0.5, while the upper bound tests whether it exceeds the meaningfulness threshold.Source: §4.1 · p. 8

• The single-point comparison has approximately 10% false positives and 75% false negatives, while average comparison has below 5% false positives and approximately 90% false negatives.The probability-of-outperforming method achieves approximately 5% false positives and 30% false negatives.Source: §4.2 · p. 9
• Adjusting thresholds using performance variance improves control of false negatives compared with a fixed average-comparison threshold.Source: §4.2 · p. 9
• The probability-of-outperforming test controls error rates well with a biased estimator, though its performance is worse than with the ideal estimator.The authors state that it is a major improvement over the commonly used comparison method at no additional cost, without guaranteeing nominal control.Source: §4.2 · p. 9

• The recommendations are designed to be simple to implement and suitable for limited computational budgets.Source: BENCHMARKS WITH A BUDGET · p. 9

Randomize as many sources of variations as possible

• Varying arbitrary pipeline choices produces less-correlated test-set measurements and can reduce error in expected-performance estimates.The paper relates this effect to variance reduction from bagging.Source: Randomize as many sources of… · p. 9

• Figure 6 compares detection rates across true P(A > B), methods, and estimator costs, showing more balanced errors for probability-of-outperforming than average comparison.Source: Figure 6 · p. 9

Use multiple data splits

• Multiple data splits improve statistical power because a single test subset provides limited information about performance on wider samples from the same distribution.Source: Randomize as many sources of… · pp. 9–10

Account for variance to detect meaningful improvements

• Using P(A > B) with γ = 0.75 separates benchmarking fluctuations from published improvements across the five case studies considered.The authors also recommend highlighting procedures within the significance bounds, not only the best-performing procedure.Source: Randomize as many sources of… · p. 10

Comparing models instead of procedures

• The framework does not apply when models are fixed and their training process and sources of variation cannot be controlled.In such cases, only the test data remains as a source of variation.Source: §6 · p. 10

Benchmarks and competitions with many contestants

• Comparing many algorithms introduces multiple-comparison challenges, requiring stricter decision thresholds and making rankings partly dependent on test-set choices.Source: Benchmarks and competitions with many… · p. 10

Comparisons across multiple dataset

• Comparisons across datasets must account for dataset-specific variance and error levels.Source: Benchmarks and competitions with many… · p. 10
• Wilcoxon and Friedman tests have limited statistical power when studies include only three to five datasets.Source: Benchmarks and competitions with many… · p. 10

Non-normal metrics

• The study focuses on model performance metrics, while practical evaluation may also involve training time or memory footprint.Performance metrics are generally averages over samples and are therefore treated as amenable to a reasonable normality assumption.Source: Benchmarks and competitions with many… · p. 10

• Variance arises from data sampling, hyperparameter choices, weight initialization, and other uncontrolled sources.Source: §7 · p. 10
• Data-split perturbations contribute more variance than weight initialization in the deep-learning evaluations.Weight initialization contributes a small amount, comparable to residual post-optimization hyperparameter variance.Source: §7 · p. 10
• Reliable benchmarking requires randomizing as many variation sources as possible, preferring multiple random splits, and accounting for variance when comparing algorithms.Source: §7 · p. 10

• One image-segmentation case study retained uncontrolled numerical noise because the pipeline could not be made perfectly reproducible.Source: A NOTES ON REPRODUCIBILITY · p. 14
• Bootstrap sampling is preferred to cross-validation for simulating independent data sampling because cross-validation can underestimate variance through induced correlations.Source: B OUR BOOTSTRAP PROCEDURE · p. 14
• Randomize data splits, augmentation, model initialization, data order, and hyperparameter optimization whenever possible.The paper notes that avoiding fixed seeds is a simple way to randomize variation sources.Source: §C.1 · p. 15
• Multiple hyperparameter optimizations may be unaffordable, but fixing hyperparameters after one optimization degrades conclusion reliability.Source: §C.1 · p. 15
• Pairing matched runs reduces variance from σA + σB to σA−B, enabling detection of smaller differences without increasing sample size.Pairing reuses the same seed for algorithms A and B in each pair.Source: §C.2 · p. 15

• The sample size N denotes the number of runs used to estimate P(A > B), distinct from the dataset size n.Source: §C.3 · p. 16
• The procedure sets γ, α, and β, recommending γ = 0.75 and β = 0.05 for strong statistical power.α controls false positives and β controls false negatives.Source: §C.3 · p. 16
• P(A > B) estimation is equivalent to a Mann–Whitney test, enabling Noether’s sample-size method.Source: §C.3 · p. 16
• 29 trainings are required for γ = 0.75 with β = 0.05, whereas detecting P(A > B) below 0.6 becomes impractical.Below γ = 0.55, the requirement exceeds 700 trainings; the figure reports sample sizes above 500 below 0.6.Source: §C.3 · p. 16, Figure C.1 · p. 16
• Percentile bootstrap confidence intervals are computed by repeatedly resampling N pairs and taking the α/2 and (1−α/2) percentiles.Source: §C.5 · p. 16

• A comparison is statistically meaningful only when the observed probability of improvement exceeds a practical threshold γ, not merely 0.5.Source: §C.6 · p. 16
• If CImax ≤ γ, the result is not statistically meaningful even when CImin > 0.5.Source: §C.6 · p. 16
• The evaluation used five machine-learning case studies spanning image classification, NLP classification, and image segmentation.Source: §D.1 · pp. 16–17, §D.3 · p. 18, §D.4 · p. 18

Task and data

• The MLP-MHC task predicts normalized binding affinity from concatenated MHC-allele and peptide sequences.Source: §D.5 · pp. 18–19

Data and encoding

• The study uses one-hot amino-acid encoding instead of the BLOSUM62 encoding used by NetMHCpan4 and MHCflurry.Source: §D.5 · p. 19

Model

• The model uses one hidden layer, ReLU nonlinearity, and Glorot weight initialization.Source: §D.5 · p. 19

Hyperparameter search

• The hyperparameter search varies hidden-layer sizes from 20 to 400 and L2 regularization over a log-uniform range between 0 and 1.Source: §D.5 · p. 19

Performance comparison

• The authors report performance comparable to NetMHCpan4 and MHCflurry on HPV external-test and NetMHC-CVsplits datasets.Source: §D.5 · p. 19
• Comparison on NetMHC-CVsplits may overestimate MHCflurry because that dataset may overlap with its training data.The authors identify retraining MHCflurry on each fold as the proper comparison, but leave it for later work.Source: §D.5 · pp. 19–20

Estimator variability

• Figure H.4 provides a complete comparison of the standard deviations of different performance estimators.Source: H RANDOMIZING MORE SOURCES OF VARIANCE… · pp. 20–21

Sources of variation

• The supplementary analyses include hyperparameter-optimization curves and performance distributions conditional on individual or jointly randomized variation sources.Source: Figure F.2 · p. 21, Figure G.3 · p. 21

Estimator variability

• Figure H.5 decomposes estimator mean-squared error into bias, variance, correlation, and total MSE components.Source: H RANDOMIZING MORE SOURCES OF VARIANCE… · p. 21

Experimental analysis

• The experiments vary sample size and threshold γ, converting γ into an equivalent performance difference δ = Φ−1(γ)σ.Source: I ANALYSIS OF ROBUSTNESS OF COMPARISON… · p. 21

Estimator variability

• Figure H.4 shows estimator standard error across sample counts for classification accuracy, Intersection over Union, and Area Under the Curve.Source: Figure H.4 · p. 22

Detection robustness

• Figure I.6 reports detection rates as functions of sample size and threshold γ across simulations with different true probabilities of A outperforming B.Source: Figure I.6 · p. 23