Source-linked AI summary
Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimation
Tristan Kirscher, Markus Bujotzek, Yannick Kirchhoff, Maximilian Rokuss, Fabian Isensee, Kim-Celine Kahl, Balint Kovacs, Klaus Maier-Hein
TL;DR
CV ensembles are often called deep ensembles despite differing in data exposure, complicating uncertainty interpretation. This paper audits the distinction and compares both constructions, finding DE improve calibration and failure detection while CV better reflect ambiguity on some datasets.
Problem
CV disagreement conflates seed-driven parameter variability with variability from differing data exposure, making ensemble construction relevant to uncertainty interpretation and downstream decisions.
Method
The paper audits ensemble terminology and compares 5-fold CV with 5-member DE under matched configurations across multi-rater datasets, modalities, and uncertainty tasks.
Results
DE retain comparable segmentation accuracy while consistently improving calibration and failure detection; CV ensembles better align with inter-rater ambiguity on some datasets.
Takeaways & Limitations
Use DE for reliability-sensitive decisions and CV ensembles when modeling annotation ambiguity, because ensemble construction affects uncertainty behavior.
Takeaways & Limitations
One case was excluded because of annotation spacing inconsistencies from one rater.
Abstract
from arXiv · showhide
Ensemble disagreement is widely used as a proxy for epistemic uncertainty in medical image segmentation. In practice, many studies form ensembles via K-fold cross-validation (CV), yet refer to them as ``deep ensembles'' (DE). Because CV members are trained on different data subsets, their disagreement mixes seed-driven variability with data-exposure effects, which can change how uncertainty should be interpreted. We audit recent segmentation uncertainty studies and find that terminology--implementation mismatches are common. We then compare a standard 5-fold CV ensemble to a 5-member DE (fixed training set, different random seeds) under otherwise identical configurations on three multi-rater segmentation datasets spanning three modalities. We evaluate uncertainty for calibration, failure detection, ambiguity modeling, and robustness under distribution shift. DE match segmentation accuracy while improving calibration and failure detection, whereas CV ensembles sometimes correlate more strongly with inter-rater variability on the studied datasets. Thus, ensemble construction should be chosen to match the research question: DE for reliability-oriented use (e.g., selective referral/failure detection) and CV ensembles as a proxy for ambiguity. We provide a lightweight nnU-Net modification enabling DE training within the default pipeline.
1 Introduction
The introduction distinguishes deep ensembles from cross-validation ensembles and motivates studying how their different data exposure affects uncertainty-based clinical tasks. It outlines a terminology audit, controlled multi-task comparison, and nnU-Net implementation enabling standard deep ensembles.
- Motivation: Deep ensembles train identical models on the same data with different random initializations, whereas CV members train on different data subsets.The distinction determines whether prediction variability primarily reflects parameter uncertainty or differences in data exposure.
- Motivation: Disagreement from CV models is increasingly used as an epistemic-uncertainty proxy for quality control, human review, and failure detection.These estimates can trigger selective referral, automated failure detection, and identification of reduced model performance.
- Contributions: A terminology audit documents that segmentation studies frequently call K-fold CV ensembles deep ensembles despite training members on different data subsets.The paper therefore distinguishes CV ensembles from DE throughout.
- Contributions: The study compares 5-fold CV ensembles with 5-member DE under unified configurations across multi-rater datasets, modalities, and downstream uncertainty tasks.Evaluated tasks include segmentation, calibration, ambiguity modeling, failure detection, and robustness under distribution shift.
- Contributions: A minor nnU-Net pipeline modification is provided to support deep-ensemble training as originally described.This addresses the framework’s default use of five-fold CV and prediction averaging at inference.
2 Related Work
Related work defines deep ensembles as independently initialized networks trained on the same dataset for epistemic uncertainty estimation, while medical segmentation uses ensembles for calibration and error awareness. In nnU-Net, however, 5-fold cross-validation models are often reused as ensembles and interpreted as deep ensembles despite differing data exposure.
- Deep ensembles and epistemic uncertainty: Deep ensembles train multiple networks on the same dataset with different random initializations and stochastic optimization trajectories to estimate epistemic uncertainty.They have often outperformed approximate Bayesian methods such as Monte Carlo dropout in calibration, robustness, and other reported criteria.
- Uncertainty estimation in medical image segmentation: Ensemble-based methods in medical image segmentation have consistently improved calibration and error awareness compared with single-model predictions.These methods support applications including quality control, error detection, and human-in-the-loop decision support.
- Ensembles in nnU-Net pipelines: nnU-Net trains models in 5-fold cross-validation for in-distribution performance estimation and reuses them as a CV ensemble to boost test-set performance.Recent studies have often interpreted disagreement among these CV models as epistemic uncertainty, sometimes referring to them as deep ensembles.
3 Problem Statement
The section distinguishes deep ensembles, whose models share the full training dataset but differ in stochastic optimization, from cross-validation ensembles, whose models train on incomplete data subsets. Consequently, CV disagreement combines epistemic uncertainty with variability caused by data subsampling, whereas DE disagreement targets uncertainty given the full dataset.
- Deep Ensembles: Deep ensembles train M models on the same dataset D with different random initializations and stochastic training.Initialization, data shuffling, and augmentation induce variation across model parameters.
- Deep Ensembles: Deep-ensemble prediction variability reflects epistemic uncertainty of the model given the full dataset D.The ensemble predictive distribution averages the models’ conditional predictions.
- Cross-Validation Ensembles: Cross-validation ensembles train each model on a different subset created by partitioning the dataset into disjoint folds.The resulting ensemble represents a mixture of models conditioned on different training sets.
- Key Distinction: CV disagreement reflects both posterior uncertainty and variability induced by training on incomplete data subsets.A model may appear uncertain because it had limited exposure to particular training examples rather than because of uncertainty under the full-data distribution.
4 Experimental Setup
The experiments isolate ensemble-construction effects by comparing 5-fold CV ensembles with five-member deep ensembles under identical nnU-Net training protocols. Evaluation spans heterogeneous multi-rater segmentation datasets and measures accuracy, calibration, ambiguity modeling, failure detection, and distribution-shift robustness.
- Datasets: Datasets span 2D and 3D MRI, CT, and retinal fundus segmentation, with expert, STAPLE, and majority-voting consensus masks and independent rater annotations.The design supports ambiguity analysis across heterogeneous annotation aggregation paradigms.
- Datasets: Distribution-shift evaluation holds out 95 RIGA images from the Magrabi Eye Center and four GoldAtlas patients from acquisition site 3 as OOD cases.These holdouts represent shifts in acquisition center and population for RIGA and an acquisition-site shift for GoldAtlas.
- Datasets: One case was excluded because of annotation-spacing inconsistencies from one rater.This exclusion is documented as a dataset limitation.
- Network and Training Configuration: Identical fixed-epoch, fixed-learning-rate training and final-checkpoint inference ensure differences reflect training-subset exposure rather than checkpointing or optimization artifacts.Validation is used only for reporting, not model selection, and no early stopping or validation-based learning-rate adaptation is applied.
- Ensemble Configurations: CV models train on 80% of cases per fold, whereas five DE members train independently on all available training data with different stochastic realizations.Both configurations use the same pool of training cases; DE variation comes from initialization, shuffling, augmentation, and optimization.
- Uncertainty Quantification and Evaluation: Evaluation combines mean-probability segmentation performance with voxel-level ACE and BA-ECE calibration, entropy–rater-variance NCC and GED ambiguity metrics, and disagreement-based failure detection.Failure detection ranks cases by inter-model disagreement and defines risk as 1−DSC between the mean prediction and consensus segmentation.
5 Results
Across datasets, deep ensembles (DE) preserve segmentation accuracy relative to cross-validation (CV) ensembles while offering better calibration and stronger sample-level failure detection. The results therefore show systematic differences in uncertainty properties despite comparable DSC.
- DE achieve DSC comparable to CV ensembles across all datasets, leaving segmentation accuracy essentially unchanged.
- DE consistently improve voxel-wise calibration across datasets by reducing ACE and yielding lower BA-ECE.
- At the sample level, DE provide stronger failure detection than CV ensembles.
6 Discussion and Conclusion
CV ensembles are not interchangeable with deep ensembles for uncertainty estimation because fold disagreement reflects both epistemic uncertainty and data-exposure sensitivity. The paper recommends matching ensemble construction to the application, reporting it explicitly, and provides a minimal nnU-Net modification supporting deep ensembles.
- Interpretation: CV disagreement reflects both epistemic uncertainty and sensitivity to data exposure because fold members train on different data subsets.This distinction matters when uncertainty is thresholded or ranked in reliability-oriented workflows.
- Task-dependent recommendations: Reliability-sensitive decision-making favors DE, whereas ambiguity modeling may benefit from CV ensembles.The recommendation is task-dependent rather than universally favoring one ensemble construction.
- Practical considerations: DE incur extra cost because their members must be trained in addition to the CV models typically produced during parameter optimization.This practical burden should be considered when selecting an ensemble construction.
- Recommendations and implementation: Future studies should explicitly report the ensemble construction used for uncertainty estimation and select DE or CV ensembles based on the application.The authors also release a minimal nnU-Net change enabling deep-ensemble support as described in.
Code and Data Availability
The study’s code is publicly available, while dataset access and reuse follow provider-specific terms and the repository does not redistribute derived artifacts.
- Code is available at https://github.com/Kirscher/LostInFolds.
- GoldAtlas, CURVAS, and RIGA access and reuse are subject to their providers’ terms, and preprocessed data, checkpoints, and predictions are not redistributed.