Source-linked AI summary
Accounting for Variance in Machine Learning Benchmarks
Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Gaël Varoquaux, Pascal Vincent
TL;DR
Machine-learning benchmarks can be dominated by uncontrolled variation, including data sampling, initialization, and hyperparameter choices. The paper models this variance and evaluates comparison criteria, finding that probability-of-outperforming tests improve error control and motivate recommendations for more reliable benchmarks.
Problem
Uncontrolled factors can dominate meaningful performance differences, making benchmark comparisons non-reproducible and potentially unfair.
Method
The paper models benchmark variance across data sampling, weight initialization, hyperparameter optimization, and other fluctuations, then evaluates decision criteria for detecting improvements.
Results
Across deep-learning benchmark analyses, probability-of-outperforming comparisons control error rates well with biased estimators and substantially improve on commonly used comparisons.
Takeaways & Limitations
Randomize as many variation sources as possible, account for variance when comparing average performance, and favor resampling over fixed held-out test sets.
Takeaways & Limitations
The framework does not apply when models are given and their training processes and variation sources cannot be controlled, leaving only test data as a variable.
Abstract
from arXiv · showhide
Strong empirical evidence that one machine-learning algorithm A outperforms another one B ideally calls for multiple trials optimizing the learning pipeline over sources of variation such as data sampling, data augmentation, parameter initialization, and hyperparameters choices. This is prohibitively expensive, and corners are cut to reach conclusions. We model the whole benchmarking process, revealing that variance due to data sampling, parameter initialization and hyperparameter choice impact markedly the results. We analyze the predominant comparison methods used today in the light of this variance. We show a counter-intuitive result that adding more sources of variation to an imperfect estimator approaches better the ideal estimator at a 51 times reduction in compute cost. Building on these results, we study the error rate of detecting improvements, on five different deep-learning tasks/architectures. This study leads us to propose recommendations for performance comparisons.
1 INTRODUCTION: TRUSTWORTHY BENCHMARKS ACCOUNT FOR FLUCTUATIONS
Reliable machine-learning benchmarks must account for uncontrolled fluctuations across data sampling, initialization, augmentation, and hyperparameter optimization. Because exhaustive repeated pipeline training is often unaffordable, the paper develops practical recommendations for limited computational budgets.
- Uncontrolled factors can dominate meaningful performance differences between algorithms, undermining benchmark conclusions.
- Robust comparisons require repeatedly training complete pipelines across hyperparameter optimization and random seeding, but computational budgets often permit only a few model fits.
- Prior comparison methods often focused on data-sampling variance while ignoring extrinsic variation, including hyperparameter choices and pipeline procedures.
- The paper frames uncontrolled hyperparameter optimization statistically through explicit bias and variance, then evaluates decision criteria and their error rates under noise.
- The recommendations randomize as many variation sources as possible, account for variance when comparing average performance, and favor resampling over fixed test sets.
2 THE VARIANCE IN ML BENCHMARKS
Benchmark performance is a random quantity shaped by data splits and stochastic learning-pipeline choices, with data sampling often contributing the largest variance. The study models these sources jointly and shows that benchmark variance can be comparable to published improvements.
- Benchmark performance is a realization of a random variable because both the dataset and the learning pipeline contain uncontrolled fluctuations.
- The framework models variation from data sampling, initialization, ordering, augmentation, and hyperparameter optimization across the complete learning pipeline.
- Hyperparameters govern regularization, architecture, and optimizer settings, while their optimization introduces additional arbitrary variation through search procedures and data splits.
- Bootstrapping the data is generally the largest variance source, whereas model initialization is typically less than 50% of bootstrap variance and comparable to stochastic-gradient-descent ordering.
- A binomial model provides a simple account of classification sampling noise, but correlated errors reduce effective degrees of freedom and widen the distribution.
- Hyperparameter choice induces sizable variance, with three optimization methods averaging as much variance as commonly studied weight initialization.
- Total benchmark variance is on the order of published state-of-the-art increments, so it must be accounted for in pipeline comparisons.
3 ACCOUNTING FOR VARIANCE TO RELIABLY ESTIMATE PERFORMANCE ˆRP
The paper distinguishes an unbiased ideal estimator that randomizes all learning and hyperparameter sources from a cheaper biased estimator that reuses one hyperparameter optimization. Although repeated splits can reduce uncertainty, fixing hyperparameters induces correlation and bias; randomizing more sources moves the cheap estimator closer to the ideal one.
- 3.1 Multiple data splits for smaller detectable improvements: Fixed training and test sets estimate expected empirical risk noisily because finite data sampling limits information about performance on new data.The resulting uncertainty can be comparable to typical improvements, and test-set size may limit estimation quality.
- 3.1 Multiple data splits for smaller detectable improvements: Increasing the number of splits k enables detection of smaller performance differences, but the ideal estimator remains computationally expensive.The estimator averages performance over multiple train–test splits, while reliable comparisons must account for its uncertainty.
- 3.2.1 Ideal estimator: sampling multiple HOpt: The ideal estimator randomizes all learning and hyperparameter sources independently, is unbiased, and requires O(k · T) model fittings.Its variance decreases as k grows, with limk→∞ Var(ˆµ(k)) = 0.
- 3.2.2 Biased estimator: fixing HOpt: The biased estimator reuses hyperparameters from one optimization, requires O(k + T) fittings, and can remain noisy as k increases because fixed sources induce correlation.When the average correlation ρ is large, the correlation term can dominate its variance, so additional splits do not adequately improve the estimate.
- 3.2.2 Biased estimator: fixing HOpt: Accounting for more variation sources reduces the biased estimator’s correlation and moves it closer to the ideal estimator in most case studies.Randomizing initialization alone gives only a small improvement, while randomizing data and all non-HOpt sources performs substantially better.
- 3.3 The cost of ignoring HOpt variance: IdealEst(k = 100) takes 1 070 hours, whereas each FixedHOptEst(k = 100) takes 21 hours.The study reports that the cheaper estimator is better than the status quo and suitable for statistical model comparisons without prohibitive costs.
4 ACCOUNTING FOR VARIANCE TO DRAW
Average-performance comparisons can mistake benchmark fluctuations for improvements because they ignore variance. The paper proposes testing whether one algorithm outperforms another with sufficiently high probability, yielding better-balanced error rates.
- The average-difference criterion does not account for false detections, making random chance difficult to distinguish from true impact.
- Probability of outperforming: Probability of outperforming compares performance distributions across fluctuations and declares A significantly better when P(A > B) ≥ γ.The method uses a threshold γ for how often A must outperform B.
- Probability of outperforming: The proposed test uses confidence intervals and requires both statistical significance and meaningful improvement to conclude that A is better than B.The lower confidence bound tests whether P(A > B) exceeds 0.5, while the upper bound tests whether it exceeds the meaningfulness threshold.
- The single-point comparison has approximately 10% false positives and 75% false negatives, while average comparison has below 5% false positives and approximately 90% false negatives.The probability-of-outperforming method achieves approximately 5% false positives and 30% false negatives.
- Adjusting thresholds using performance variance improves control of false negatives compared with a fixed average-comparison threshold.
- The probability-of-outperforming test controls error rates well with a biased estimator, though its performance is worse than with the ideal estimator.The authors state that it is a major improvement over the commonly used comparison method at no additional cost, without guaranteeing nominal control.
5 OUR RECOMMENDATIONS: GOOD
The paper recommends randomizing major sources of benchmark variation and accounting for variance when judging improvements. These practices improve estimation and support more reliable decisions under limited computational budgets.
- The recommendations are designed to be simple to implement and suitable for limited computational budgets.
- Randomize as many sources of variations as possible: Varying arbitrary pipeline choices produces less-correlated test-set measurements and can reduce error in expected-performance estimates.The paper relates this effect to variance reduction from bagging.
- Figure 6 compares detection rates across true P(A > B), methods, and estimator costs, showing more balanced errors for probability-of-outperforming than average comparison.
- Use multiple data splits: Multiple data splits improve statistical power because a single test subset provides limited information about performance on wider samples from the same distribution.
- Account for variance to detect meaningful improvements: Using P(A > B) with γ = 0.75 separates benchmarking fluctuations from published improvements across the five case studies considered.The authors also recommend highlighting procedures within the significance bounds, not only the best-performing procedure.
6 ADDITIONAL CONSIDERATIONS
The study has important scope boundaries for model access, the number of algorithms, dataset aggregation, and evaluation metrics. These settings introduce challenges that the proposed framework does not fully resolve.
- Comparing models instead of procedures: The framework does not apply when models are fixed and their training process and sources of variation cannot be controlled.In such cases, only the test data remains as a source of variation.
- Benchmarks and competitions with many contestants: Comparing many algorithms introduces multiple-comparison challenges, requiring stricter decision thresholds and making rankings partly dependent on test-set choices.
- Comparisons across multiple dataset: Comparisons across datasets must account for dataset-specific variance and error levels.
- Comparisons across multiple dataset: Wilcoxon and Friedman tests have limited statistical power when studies include only three to five datasets.
- Non-normal metrics: The study focuses on model performance metrics, while practical evaluation may also involve training time or memory footprint.Performance metrics are generally averages over samples and are therefore treated as amenable to a reasonable normality assumption.
7 CONCLUSION
Benchmark performance fluctuates across multiple sources, and reliable comparisons must account for that variance rather than focusing only on weight initialization.
- Variance arises from data sampling, hyperparameter choices, weight initialization, and other uncontrolled sources.
- Data-split perturbations contribute more variance than weight initialization in the deep-learning evaluations.Weight initialization contributes a small amount, comparable to residual post-optimization hyperparameter variance.
- Reliable benchmarking requires randomizing as many variation sources as possible, preferring multiple random splits, and accounting for variance when comparing algorithms.
A NOTES ON REPRODUCIBILITY
Reproducible and statistically reliable comparisons require controlling or randomizing variation throughout the pipeline, while pairing can reduce comparison variance when feasible.
- One image-segmentation case study retained uncontrolled numerical noise because the pipeline could not be made perfectly reproducible.
- Bootstrap sampling is preferred to cross-validation for simulating independent data sampling because cross-validation can underestimate variance through induced correlations.
- Randomize data splits, augmentation, model initialization, data order, and hyperparameter optimization whenever possible.The paper notes that avoiding fixed seeds is a simple way to randomize variation sources.
- Multiple hyperparameter optimizations may be unaffordable, but fixing hyperparameters after one optimization degrades conclusion reliability.
- Pairing matched runs reduces variance from σA + σB to σA−B, enabling detection of smaller differences without increasing sample size.Pairing reuses the same seed for algorithms A and B in each pair.
C.3 Sample size
Sample size is chosen to estimate P(A > B) reliably under specified false-negative and threshold requirements, with γ = 0.75 recommended as a practical setting.
- The sample size N denotes the number of runs used to estimate P(A > B), distinct from the dataset size n.
- The procedure sets γ, α, and β, recommending γ = 0.75 and β = 0.05 for strong statistical power.α controls false positives and β controls false negatives.
- P(A > B) estimation is equivalent to a Mann–Whitney test, enabling Noether’s sample-size method.
- 29 trainings are required for γ = 0.75 with β = 0.05, whereas detecting P(A > B) below 0.6 becomes impractical.Below γ = 0.55, the requirement exceeds 700 trainings; the figure reports sample sizes above 500 below 0.6.
- Percentile bootstrap confidence intervals are computed by repeatedly resampling N pairs and taking the α/2 and (1−α/2) percentiles.
C.6 Statistical test with P(A > B)
The statistical test evaluates both whether A’s advantage exceeds chance and whether it is large enough to be meaningful, using confidence bounds for P(A > B).
- A comparison is statistically meaningful only when the observed probability of improvement exceeds a practical threshold γ, not merely 0.5.
- If CImax ≤ γ, the result is not statistically meaningful even when CImin > 0.5.
- The evaluation used five machine-learning case studies spanning image classification, NLP classification, and image segmentation.
D.5 Major histocompatibility class I-associated peptide binding prediction with shallow MLP
This section evaluates a shallow MLP for MHC-associated peptide binding prediction and compares it with NetMHCpan4 and MHCflurry. It also highlights methodological constraints in hyperparameter search and cross-model evaluation.
- Task and data: The MLP-MHC task predicts normalized binding affinity from concatenated MHC-allele and peptide sequences.
- Data and encoding: The study uses one-hot amino-acid encoding instead of the BLOSUM62 encoding used by NetMHCpan4 and MHCflurry.
- Model: The model uses one hidden layer, ReLU nonlinearity, and Glorot weight initialization.
- Hyperparameter search: The hyperparameter search varies hidden-layer sizes from 20 to 400 and L2 regularization over a log-uniform range between 0 and 1.
- Performance comparison: The authors report performance comparable to NetMHCpan4 and MHCflurry on HPV external-test and NetMHC-CVsplits datasets.
- Performance comparison: Comparison on NetMHC-CVsplits may overestimate MHCflurry because that dataset may overlap with its training data.The authors identify retraining MHCflurry on each fold as the proper comparison, but leave it for later work.
H RANDOMIZING MORE SOURCES OF VARIANCE INCREASE THE QUALITY OF THE ESTIMATOR
This section examines estimator variability across tasks and sources of variation. It reports standard-deviation and mean-squared-error analyses intended to explain why broader randomization can improve estimation.
- Estimator variability: Figure H.4 provides a complete comparison of the standard deviations of different performance estimators.
- Sources of variation: The supplementary analyses include hyperparameter-optimization curves and performance distributions conditional on individual or jointly randomized variation sources.
- Estimator variability: Figure H.5 decomposes estimator mean-squared error into bias, variance, correlation, and total MSE components.
I ANALYSIS OF ROBUSTNESS OF COMPARISON METHODS
This section studies the robustness of comparison methods under changes in sample size and decision threshold. The analyses examine detection rates across different true probabilities that algorithm A outperforms algorithm B.
- Experimental analysis: The experiments vary sample size and threshold γ, converting γ into an equivalent performance difference δ = Φ−1(γ)σ.
- Estimator variability: Figure H.4 shows estimator standard error across sample counts for classification accuracy, Intersection over Union, and Area Under the Curve.
- Detection robustness: Figure I.6 reports detection rates as functions of sample size and threshold γ across simulations with different true probabilities of A outperforming B.