Source-linked AI summary
Nested cross-validation when selecting classifiers is overzealous for most practical applications
Jacques Wainer, Gavin Cawley
TL;DR
Selecting a classification algorithm requires choosing both the algorithm and its hyperparameters, but it is unclear whether unbiased nested cross-validation is necessary for that practical choice. This paper compares nested and flat cross-validation across classifier-selection scenarios and finds that flat cross-validation generally selects an algorithm of similar practical quality when few hyperparameters require tuning.
Problem
Practitioners must select both a classification algorithm and its hyperparameters, while current machine-learning understanding cannot predict in advance which algorithm will perform best for a particular problem.
Method
The study compares nested and flat cross-validation when selecting among three strong classifiers and among 12 classifier families.
Results
Flat cross-validation generally selects an algorithm that performs as well as the algorithm selected by nested cross-validation for most practical purposes.
Takeaways & Limitations
Nested cross-validation is probably not needed for classifier selection when the candidate algorithms have only a limited number of hyperparameters to tune.
Takeaways & Limitations
The practical-equivalence claim is limited because all tested algorithms had only 1 to 3 hyperparameters, and it applies to the best-ranked algorithm rather than the full ranking.
Abstract
from arXiv · showhide
When selecting a classification algorithm to be applied to a particular problem, one has to simultaneously select the best algorithm for that dataset \emph{and} the best set of hyperparameters for the chosen model. The usual approach is to apply a nested cross-validation procedure; hyperparameter selection is performed in the inner cross-validation, while the outer cross-validation computes an unbiased estimate of the expected accuracy of the algorithm \emph{with cross-validation based hyperparameter tuning}. The alternative approach, which we shall call `flat cross-validation', uses a single cross-validation step both to select the optimal hyperparameter values and to provide an estimate of the expected accuracy of the algorithm, that while biased may nevertheless still be used to select the best learning algorithm. We tested both procedures using 12 different algorithms on 115 real life binary datasets and conclude that using the less computationally expensive flat cross-validation procedure will generally result in the selection of an algorithm that is, for all practical purposes, of similar quality to that selected via nested cross-validation, provided the learning algorithms have relatively few hyperparameters to be optimised.
1 INTRODUCTION
Selecting a classifier requires jointly choosing an algorithm and its hyperparameters, but flat and nested cross-validation make different accuracy estimates and computational trade-offs. The study evaluates whether nested cross-validation’s greater reliability justifies its cost for practical model selection.
- Motivation: Classifier selection and hyperparameter tuning are dependent steps because an algorithm’s performance can change substantially with its hyperparameter values.The paper frames both choices as a joint optimization problem for expected future performance.
- Flat cross-validation: Flat cross-validation tunes hyperparameters and then reuses the resulting estimate to select the model, making it computationally inexpensive but optimistically biased.The bias arises because evaluation data also influence hyperparameter tuning.
- Nested cross-validation: Nested cross-validation tunes hyperparameters inside each outer fold and uses held-out outer data to estimate performance, removing flat cross-validation’s bias at substantially higher computational cost.Its outer estimate targets the performance of a model-fitting procedure that includes cross-validation-based tuning.
- Selection criterion: Both procedures estimate expected classifier performance and select the algorithm with the highest estimate, although their numerical estimates will generally differ.The paper distinguishes unbiased nested estimates from positively biased flat estimates while assessing their usefulness for selection.
- Study aim: The study evaluates nested versus flat selection across 115 real-life datasets and 12 classification algorithms, focusing on the practical accuracy gain from nested selection.The central comparison concerns whether differing selections translate into meaningful future-accuracy differences.
2 DATA AND METHODS
The experiments compare nested and flat cross-validation across repeated train/test splits, measuring selection agreement and future-accuracy differences on binary classification datasets. The analysis tests whether any nested-selection advantage exceeds a dataset-specific irrelevance threshold across narrow and broad classifier scenarios.
- Experimental design: Each dataset was evaluated using six repetitions of a stratified 50% train/test split.The repeated splits served as separate experiments for estimating accuracy gain and irrelevance thresholds.
- Cross-validation procedures: Within each training split, the study computed 5-fold-within-5-fold nested-CV and 5-fold flat-CV estimates for 12 classification algorithms.Flat CV also supplied the hyperparameters used for evaluating future accuracy.
- Outcome measurement: The nested and flat procedures selected algorithms n(i,r) and f(i,r), whose future accuracies were measured on the corresponding held-out test subset.The analysis compared the selected algorithms rather than only comparing their cross-validation estimates.
- Accuracy gain: Accuracy gain was defined as the future accuracy of nested selection minus the future accuracy of flat selection, then averaged across the six repetitions for each dataset.A positive gain would favor nested selection under the paper’s definition.
- Irrelevance threshold: The irrelevance threshold captures unavoidable accuracy-estimation variation and is defined using the smaller estimation error of the two selected algorithms.The threshold is dataset- and algorithm-dependent, with smaller datasets expected to show larger unavoidable changes.
- Scenarios: The scenarios compare selection among random forest, RBF-kernel SVM, and gradient boosting machines versus selection across 12 classifier families.The three-algorithm scenario reflects the study’s identified strongest average-ranked classifiers.
3 RESULTS
Across the evaluated scenarios, flat and nested cross-validation often selected the same classifier, and any nested-selection accuracy advantage was generally below the threshold of practical irrelevance.
- Algorithm rankings: The top-three algorithm ranking agreed with the order reported in [10].Table 1 reports mean algorithm rankings based on nested cross-validation accuracies across repetitions and datasets.
- Top 3 and full scenarios: In the top-3 scenario, most accuracy-gain points fell below the y = x line, indicating irrelevance thresholds usually exceeded accuracy gains.Figure 3 compares absolute accuracy gains with dataset-specific thresholds of irrelevance.
- Top 3 and full scenarios: 71% of top-3 selections matched between flat and nested cross-validation, compared with 33% for random selection.The top-3 scenario considered random forest, SVM with RBF kernel, and gradient boosting machines.
- Top 3 and full scenarios: 62% of full-scenario selections matched between flat and nested cross-validation, compared with 8% for random selection.The reported statistical tests again found the accuracy gain below the corresponding irrelevance threshold.
- Statistical analysis: Table 2 evaluates agreement, Wilcoxon-test p-values, mean accuracy-gain differences, and 95% confidence intervals for the top-3 and full scenarios.The mean is defined as the difference |accgain(i)| −δ(i).
4 DISCUSSION
The paper claims flat cross-validation is generally practically equivalent to nested cross-validation for selecting classifiers with few hyperparameters, but several limits constrain this conclusion.
- Nested cross-validation is probably unnecessary when choosing among the three top-performing classifier families studied.The top-3 scenario comprises random forest, SVM with Gaussian kernel, and gradient boosting machines.
- Nested cross-validation is probably unnecessary for classifier sets with only a limited number of hyperparameters to tune.This broader claim is conditioned on the algorithms having few hyperparameters.
- The study evaluated the first claim on 115 datasets, so generalization depends on whether those datasets represent practitioners’ future problems.The paper identifies dataset representativeness as a requirement for extending the finding beyond the tested datasets.
- The tested datasets were medium-sized, binary, and excluded text classification, limiting direct generalization to other data regimes.The authors note that dataset size may matter because flat-procedure bias generally decreases as dataset size increases.
- 32 datasets with at least 2000 data points showed weaker statistical evidence but similar effect sizes and more frequent agreement between procedures.The authors therefore believe practical equivalence may extend to larger datasets than those tested.
- The conclusions remained stable under two alternative analysis methods and a different irrelevance-threshold definition, but not when algorithm selection was skipped.When selection was skipped, mean accuracy gain was significantly larger than the irrelevance threshold.
5 CONCLUSION
The paper concludes that flat cross-validation generally selects an algorithm performing similarly to nested cross-validation when candidate algorithms have few hyperparameters. This conclusion is framed for practical algorithm selection, not proof of superiority or full-ranking agreement.
- Three algorithms—random forest, SVM with Gaussian kernel, and gradient boosting machine—are identified as the strongest average performers across the 115 tested datasets.
- Flat cross-validation can generally select an algorithm that performs as well as the nested-CV choice for practical purposes.
- Across candidate algorithms with few hyperparameters, flat cross-validation can simultaneously tune hyperparameters and select the algorithm without practically meaningful loss relative to nested cross-validation.
THRESHOLD
The paper examines alternative ways to aggregate repeated accuracy estimates and to define the irrelevance threshold. These alternative analyses produce results consistent with the paper’s claims.
- The original analysis treats each repetition as an independent experiment and aggregates repetitions only at the final step.
- One alternative averages all measured accuracies across six repetitions before applying the selection procedure.
- Another alternative selects flat and nested algorithms once per dataset rather than separately for each repetition.
- A further alternative treats each repetition as an independent experiment and aggregates only when computing the Wilcoxon test.
- The alternative analyses and a threshold defined by the minimum standard deviation produce results consistent with the paper’s claims.
APPENDIX B SHOULD ONE SELECT THE ALGORITHM AT ALL?
The paper tests whether selecting among algorithms is necessary by comparing random forests alone with nested selection. The results indicate that algorithm selection provides an accuracy gain with practical consequence.
- The authors compare using only random forests with selecting the best classifier through nested cross-validation.
- The observed accuracy gain from selecting an algorithm is above the threshold of irrelevance.
- Random forests alone cannot be assumed to produce a classifier within the irrelevance threshold of the best option.