On Cross-Validation for Hyperparameter Optimization of Deep Learning Image Classifiers

Ljubomir Buturovic

Source-linked AI summary

On Cross-Validation for Hyperparameter Optimization of Deep Learning Image Classifiers

Ljubomir Buturovic

arXiv:2608.14705v1Aug 10, 2026cs.CVcs.LG

TL;DR

It is unclear how to derive reliable validation signals for hyperparameter optimization of deep image classifiers, particularly with small medical-imaging datasets. This study compares fixed holdout, reshuffled holdout, and five-fold cross-validation, finding that cross-validation consistently reduces performance-estimation error on medical datasets, especially at small sample sizes.

Problem
There is little empirical guidance on whether cross-validation or fixed validation sets are preferable for deep-learning image-classifier HPO on small datasets.
Method
The study compares fixed holdout, reshuffled holdout, and five-fold cross-validation while holding the HPO search, training, architectures, and test data constant.
Results
Cross-validation consistently reduced AEE versus holdout methods on medical datasets, with the largest advantage at small sample sizes, while AEE was negligible across protocols on Tiny ImageNet.
Takeaways & Limitations
For small-sample deep image classification, cross-validation-based HPO is recommended when computational resources permit because it provides more reliable subsequent test-performance estimates.
Takeaways & Limitations
Only five development subsamples per dataset and sample size were evaluated, limiting precision in estimating variability across possible development samples.

Abstract

from arXiv · show

Hyperparameter optimization (HPO) can materially affect the performance of deep learning (DL) image classifiers, but there is little empirical guidance on how to derive the validation signal that drives it, especially for the small sample sizes common in fields such as medical imaging. We compared three HPO protocols in terms of {\em absolute performance-estimation error} (AEE; the absolute difference between the winning configuration's validation AUROC and its test AUROC): fixed holdout (F), reshuffled holdout (R), and 5-fold cross-validation (C). The search space, sampler, training procedure, architecture, and test set were held identical across protocols. We evaluated the protocols on three public datasets spanning two regimes: binary medical imaging (RSNA pneumonia radiographs and binarized HAM10000 skin lesions) and 200-class natural imaging (Tiny ImageNet), across a range of development set sizes $n$ and two backbones (ResNet-18 on all datasets, Vision Transformer (ViT-S/16) on RSNA). On the medical datasets, every point estimate favored cross-validation over both holdout protocols, with reductions in AEE largest at small sample sizes and diminishing as $n$ increased. This pattern remained robust under conservative family-wise adjustment. On Tiny ImageNet, AEE was negligible under all three protocols. Test AUROC was generally similar among protocols. Fixed holdout had lower mean AEE than reshuffled holdout in 11 of 12 medical conditions, although this secondary finding was less uniformly supported. For small-sample medical image classification, we recommend cross-validation-based HPO when computational resources permit because it trades additional computation for a more reliable development-time estimate of subsequent test performance.

1 Introduction

Cross-validation commonly evaluates hyperparameter configurations in classical machine learning, but its merits for deep-learning image-classification HPO remain under-studied. This paper compares held-out validation and cross-validation to inform the trade-off between tuning bias and computational cost.

HPO can significantly improve predictive-model performance, and classical machine learning commonly evaluates configurations with cross-validation estimates such as AUROC.
For smaller datasets, including medical-image classification, cross-validation may be feasible, but empirical guidance comparing it with fixed validation sets for deep-learning HPO is lacking.
The paper compares held-out validation and cross-validation for deep-learning image classifiers to guide choices balancing tuning bias against HPO computational cost.

2 Methods · 2.1 Overview

The study compares fixed holdout, reshuffled holdout, and 5-fold cross-validation for deep-image-classifier HPO while holding the search and training setup constant. It evaluates whether cross-validation’s added computation improves validation-signal reliability, rather than comparing methods under equal computational budgets.

2.1 Overview: The study compared fixed holdout (F), reshuffled holdout (R), and 5-fold cross-validation (C) for tuning deep image classifiers.The comparison used Optuna’s Tree-structured Parzen Estimator sampler without an early-stopping scheduler.
2.1 Overview: F reused one train/validation split to score every HPO trial.
2.1 Overview: R drew a fresh random train/validation split for each HPO trial.
2.1 Overview: C scored each trial using the mean validation metric across a fixed set of 5 folds.
2.1 Overview: The training pool comprised the training and validation data together.
2.1 Overview: Each winning configuration was retrained once on the full training pool and evaluated on a held-out test set.The goal was rigorous HPO-method comparison, not state-of-the-art test performance.
2.1 Overview: The number of HPO trials, rather than total compute, was held constant across protocols.Five-fold CV used approximately five times as many model fits per trial as either holdout protocol.
2.1 Overview: Cross-validation’s additional computation was treated as the mechanism for reducing validation-signal variance and evaluated as a compute-for-reliability trade-off.The comparison was not conducted under equal computational budgets.

2.2 Datasets

The study selected three publicly available datasets to cover binary medical and many-class natural image classification, with emphasis on small-sample medical settings. Each dataset was partitioned into development data and a shared independent test set for comparing validation protocols.

Dataset selection: The dataset selection prioritized medical imaging while including a non-medical dataset to assess whether conclusions generalize beyond medicine.The selection also targeted publicly available data, binary and multi-class problems, non-saturated off-the-shelf CNN performance, and sufficiently large independent test sets.
Datasets: Three datasets—RSNA, BHAM, and TIN—spanned binary medical imaging and many-class natural imaging.RSNA comprised binary chest-radiograph classification of pneumonia versus not, with pneumonia prevalence of 23.0%.
Data partitioning: Each dataset used a large held-out test set and a smaller training pool, with the test set shared across all methods.Within each subsample, methods F and R used an 80/20 train/validation holdout, whereas method C used all development data in 5-fold cross-validation.

2.3 Image pre-processing

All images underwent standardized preprocessing to preserve parity across benchmarking environments and prevent preprocessing from confounding comparisons. Medical images were deterministically converted to standardized arrays, while all datasets used ImageNet normalization.

Standardization: All raw images were converted to a standardized format before model exposure to preserve strict parity across benchmarking environments.Medical datasets specifically required deterministic mapping from native source formats to standard image arrays.
RSNA preprocessing: RSNA DICOM images were polarity-corrected, per-image min-max normalized to [0,1], converted to 8-bit [0,255], and resized to 224 × 224.MONOCHROME1 images were inverted to MONOCHROME2, and resizing used bilinear interpolation.
RSNA preprocessing: No dataset-level intensity normalization was applied to RSNA images, avoiding the use of cross-image statistics in stored images.Normalization was performed per image before conversion to 8-bit format.
Dataset-wide normalization: The standard ImageNet normalization pipeline was applied across all three evaluation datasets, using µ = [0.485, 0.456, 0.406] and σ = [0.229, 0.224, 0.225].This was intended to prevent data preprocessing from serving as a confounding variable in the controlled benchmarking environment.

2.4 Architecture and training

The study used an ImageNet-pretrained ResNet-18 as its primary backbone across all datasets and tested robustness with a ViT-S/16 on RSNA. HPO varied optimization, regularization, augmentation, and training-duration settings while keeping batch size fixed and resampling augmentation each epoch.

Backbones and preprocessing: ImageNet-pretrained ResNet-18 was used across all three datasets, with a full F/R/C comparison additionally run using ViT-S/16 on RSNA.Batch-normalization parameters remained unfrozen; only the final classification layer was randomly initialized and replaced to match the class count.
Hyperparameter search: The HPO space varied learning rate, weight decay, label smoothing, dropout, RandAugment, mixup, cutmix, optimizer, and training duration.Batch size was fixed at 32, while epochs ranged from 10 to 50 and the optimizer was AdamW or SGD.
Hyperparameter search: TPE proposed 50 trials for medical datasets and 30 trials for Tiny ImageNet, reflecting practicality constraints.Augmentation was sampled afresh at every epoch under all protocols.

2.5 Validation protocols · 2.6 Factorial design and metrics

The study compared fixed, reshuffled, and five-fold cross-validation validation signals within a controlled factorial design, using AUROC-based HPO and absolute performance-estimation error. Medical experiments used lesion-aware grouping where required, while TIN used ImageNet-pretrained transfer learning and a secondary descriptive top-1 accuracy metric.

2.5 Validation protocols: Fixed holdout used one 80/20 split per subsample, reshuffled holdout a fresh 80/20 split per trial, and cross-validation five class-stratified folds.For BHAM, splits and folds were additionally grouped by lesion_id.
2.6 Factorial design and metrics: Each dataset, sample size, and architecture used a 5 × S factorial of five training subsamples crossed with S random seeds.S was 5 for medical datasets and 3 for TIN, yielding 25 and 15 cells per protocol, respectively.
2.6 Factorial design and metrics: The seed controlled head initialization, data ordering, augmentation, dropout, validation-split derivation, and the HPO sampler within each subsample–seed cell.The three protocols were compared within these matched cells.
2.6 Factorial design and metrics: TIN evaluated protocols under standard ImageNet-pretrained transfer learning rather than from-scratch learning on data independent of ImageNet pretraining.This qualification defines the TIN experimental regime.
2.6 Factorial design and metrics: HPO optimized AUROC; for 200-class TIN, this was macro-averaged one-vs-rest AUROC, and AEE was defined as |AUROCv − AUROCt|.AUROCv denotes the winning configuration’s validation score, while AUROCt denotes its test score.
2.6 Factorial design and metrics: Protocol contrasts paired runs within subsample–seed cells and modeled paired test-AUROC and AEE differences with crossed random effects for subsample and seed.The model intercept represented the mean protocol difference.
2.6 Factorial design and metrics: The primary inferential family comprised 24 cross-validation-versus-holdout AEE comparisons across medical datasets, with Bonferroni-adjusted simultaneous intervals controlling family-wise error at 0.05.Individual reported confidence intervals were pointwise and unadjusted for multiplicity.
2.6 Factorial design and metrics: TIN top-1 accuracy was reported only as a secondary descriptive metric; macro-averaged one-vs-rest AUROC alone selected models and hyperparameters.Top-1 accuracy played no role in HPO and was included to show the problem was not saturated.

2.7 Computational considerations · 2.8 Code availability

Five-fold cross-validation required substantially more model-training computation than either holdout protocol, while the study’s analysis materials and HPO implementations were made available for reproducibility. Computations ran on a single NVIDIA GeForce RTX 5090 GPU over several weeks.

2.7 Computational considerations: Five-fold cross-validation required five model fits per HPO trial, compared with one fit under either holdout protocol.Each cross-validation fit used 80% of the development sample for training.
2.7 Computational considerations: With the number of HPO trials fixed, cross-validation produced approximately five times the nominal model-training workload.The higher workload followed from fitting one model for each of five folds.
2.7 Computational considerations: Computations ran on a home-built desktop computer using Ubuntu 25.10 and a single NVIDIA GeForce RTX 5090 GPU over several weeks.
2.8 Code availability: Supplementary materials provide the analysis scripts, result-level data, dataset-partitioning code, and reproduction instructions for statistical analyses and manuscript figures.
2.8 Code availability: The image-classifier HPO implementations for holdout and cross-validation are available as tunic and cvic, respectively.
2.8 Code availability: The cvic implementation is available at the project’s GitHub repository.The repository address is provided in the paper’s code-availability statement.

3 Results

Cross-validation consistently reduced absolute performance-estimation error relative to holdout protocols on medical imaging, especially at small development-set sizes, with findings robust to multiplicity adjustment. On Tiny ImageNet, AEE and test-AUROC differences were negligible, while final test AUROC was generally similar across protocols.

Test performance: Test AUROC differences were generally smaller and less consistent than AEE differences, with no practically meaningful protocol advantage on Tiny ImageNet.On BHAM, no pairwise test-AUROC difference exceeded 0.009 AUROC; on Tiny ImageNet, none exceeded 0.002 AUROC in magnitude.
Absolute performance-estimation error: Cross-validation had lower AEE than both holdout protocols in every medical dataset, architecture, and sample-size condition.All 24 point estimates favored cross-validation, with the largest reductions at small n.
Absolute performance-estimation error: −0.150 [−0.186, −0.116] was the RSNA ResNet-18 C −R AEE difference at n = 100, declining to −0.025 [−0.030, −0.020] at n = 3000.The corresponding C −F difference declined from −0.116 [−0.152, −0.081] to −0.012 [−0.018, −0.007].
Robustness and sample size: Cross-validation’s medical-dataset AEE advantage diminished as development sample size increased and remained robust after family-wise adjustment.After Bonferroni adjustment, all 12 C −R intervals and 8 of 12 C −F intervals remained entirely below zero.
Natural-image classification: On Tiny ImageNet, the largest pairwise AEE difference was 0.003 AUROC, and every confidence interval included zero.AEE was negligible under all three protocols at every sample size.
Holdout comparison: Fixed holdout had lower mean AEE than reshuffled holdout in 11 of 12 medical conditions, although confidence intervals excluded zero in only 6 conditions.The sole point estimate favoring reshuffling was BHAM at n = 100: F −R = +0.003 [−0.060, +0.066].

4 Discussion

Five-fold cross-validation consistently reduced absolute performance-estimation error on the medical datasets, especially at small sample sizes, while AEE was negligible on Tiny ImageNet. The recommendation is tempered by higher computational cost and limitations in development-sample coverage and evaluation design.

Performance findings: Cross-validation consistently reduced AEE relative to holdout methods on medical datasets, with the largest advantage at small n and diminishing advantage as n increased.All 24 point estimates favored cross-validation, and most conservatively adjusted simultaneous confidence intervals remained entirely below zero.
Performance findings: AEE was practically zero under every HPO approach on Tiny ImageNet, leaving little distinction among protocols.
Performance findings: The AEE advantage of cross-validation was virtually unchanged when using ViT-S/16 instead of ResNet-18.
Protocol comparison: Reshuffling the validation split generally increased AEE, providing no empirical justification over reusing a fixed holdout in this setting.
Practical considerations: Five-fold cross-validation required approximately five times the nominal model-training workload because each trial involved five model fits rather than one.Actual elapsed-time cost was not measured under controlled conditions and depends on hardware, implementation, and parallel execution.
Practical considerations: Using all development data within cross-validation can provide a more reliable validation signal while allowing every image to contribute to training and validation across folds.The authors propose this instead of further dividing a limited dataset when acquiring more data is prohibitively expensive.
Limitations: The analysis reused the same held-out test set across models, so test-performance estimates were correlated rather than hundreds of independent evaluations.The predefined, automated runs and lack of test-performance use during experimentation do not by themselves imply test-set overfitting.
Limitations: Only five development subsamples per dataset and sample size limited precision for estimating variability across possible development samples.Crossed random-effects models and sensitivity analyses produced consistent primary AEE conclusions, but additional independent subsamples would improve precision.

5 Conclusions

Cross-validation substantially reduces absolute performance-estimation error on small medical image datasets, with an advantage that diminishes as sample size increases and disappears when error is negligible. The authors recommend cross-validation-based HPO when computational resources permit, while noting generally similar final test performance and a less-uniform fixed-holdout advantage over reshuffled holdout.

5 Conclusions: Cross-validation substantially reduces absolute performance-estimation error on small medical image datasets, with benefits diminishing as sample size increases.The conclusion remains robust to conservative family-wise adjustment and disappears when estimation error is already negligible.
5 Conclusions: Final test performance is generally similar between cross-validation and holdout protocols.
5 Conclusions: Cross-validation-based HPO is recommended for small-sample deep image classification when computational resources permit.The trade-off is additional computation for a more reliable estimate of subsequent test performance.
5 Conclusions: Fixed holdout tends to provide more accurate performance estimates than reshuffled holdout, although this secondary finding was less uniformly supported.

A Appendix

The appendix reports complete paired comparisons of absolute estimation error and test AUROC across the study’s experimental conditions, datasets, protocols, and architectures.

Complete paired comparisons: Table 5 compares absolute estimation error and test AUROC across all RSNA and BHAM conditions using ResNet-18.Differences are computed as the first protocol minus the second, with pointwise 95% confidence intervals from crossed random-effects models.
Complete paired comparisons: Table 6 extends the paired comparisons to RSNA with ViT and to Tiny ImageNet.As in Table 5, negative ΔAEE favors the first protocol, while positive Δtest AUROC favors it.