Source-linked AI summary
Assessing nnU-Net Generalization across Brain Tumor Populations in BraTS-GoAT 2026
Tristan Kirscher, Vivian Metzger, Philippe Meyer, Xavier Coubez
TL;DR
BraTS-GoAT evaluates tumor segmentation across heterogeneous populations, where robustness across populations and acquisition protocols remains challenging. This study evaluates a fixed nnU-Net protocol and related inference variants; validation DSC was lower than source OOF DSC, and the tested additions showed no clear improvement.
Problem
Robustness across populations and acquisition protocols remains challenging, and BraTS-GoAT evaluates segmentation across five heterogeneous tumor populations.
Source: §1 · p. 1Method
The study evaluated a fixed nnU-Net protocol, ensembling, mirroring, a residual encoder, and out-of-fold failures using pooled challenge data.
Source: §1 · p. 1, §2.1 · p. 2Results
Validation DSC was lower than source OOF DSC, and mirroring, ensembling, the residual configuration, and the ET sweep showed no clear improvement.
Source: §4 · p. 8Takeaways & Limitations
Conventional nnU-Net provides a transparent baseline, while the tested additions gave no clear improvement.
Source: §4 · p. 9
Abstract
from arXiv · showhide
BraTS-GoAT evaluates tumor segmentation across heterogeneous populations. We trained a conventional 3D nnU-Net on 1,351 labeled cases using five-fold cross-validation and 1,000 epochs per fold. The final predictor averaged all folds and applied test-time mirroring. On pooled official validation, global DSC values were 0.7805, 0.8288, and 0.8854 for enhancing tumor (ET), tumor core (TC), and whole tumor (WT). Under matched fold-0 inference, mean regional Dice decreased from 0.9058 on source out-of-fold (OOF) cases to 0.8310 on pooled validation (difference--0.0747). Mirroring gave small single-fold gains but no clear ensemble benefit; a residual-encoder alternative reached 0.8282 mean Dice. In labeled OOF predictions, failure cases had substantially smaller reference ET volumes; after adjustment for ET and WT volume, lower Dice remained associated with more disconnected ET components and a smaller fraction of ET contained in the largest component.
1 Introduction
Automatic brain-tumor segmentation supports clinical assessment, planning, and follow-up, but robustness across populations and imaging protocols remains challenging. BraTS-GoAT extends evaluation across five tumor populations; this study assesses a fixed nnU-Net protocol and analyzes out-of-fold failures, with target-domain analyses necessarily pooled because validation population labels were unavailable.
1 Introduction
- Population and acquisition-protocol variability makes robust brain-tumor segmentation challenging despite its role in assessment, treatment planning, and follow-up.BraTS established a common framework for multisequence MRI tumor segmentation [2].Source: §1 · p. 1
- BraTS-GoAT extends tumor-segmentation evaluation across adult glioma, sub-Saharan African glioma, meningioma, brain metastases, and pediatric tumors.The populations are listed as the task’s expanded evaluation scope [1] [10] [13] [7] [6].Source: §1 · p. 1
- The study evaluates fixed nnU-Net, ensembling, mirroring, and a residual encoder, motivated by population variability and prior adaptive post-processing work.A pediatric CT example found adult-derived configuration plans underperformed on pediatric anatomy, particularly for small structures.Source: §1 · p. 1
- Because validation population labels were unavailable, all target-domain analyses were necessarily pooled.Source: §1 · p. 2
2 Materials and Methods
The study used pooled challenge data to train on three source populations and evaluate on a validation cohort that also included two target populations. It evaluated three tumor regions from the segmentation labels.
2.1 Task, data, and target regions
- The pooled, anonymized training set contained 1,351 labeled cases from adult glioma, meningioma, and brain metastases; validation contained 451 cases and added SSA glioma and pediatric tumors.Validation labels and case-to-population mapping were hidden, and per-case cohort labels were unavailable.Source: §2.1 · p. 2
- Evaluation regions were enhancing tumor (ET), tumor core (TC), and whole tumor (WT), defined respectively as class 3, classes 1 and 3, and classes 1, 2, and 3.Source: §2.1 · p. 2
- Class 1 represented necrotic and non-enhancing core, class 2 edema or invaded tissue, and class 3 enhancing tumor; outputs were NIfTI label maps in input-image space.Source: §2.1 · p. 2
2.2 Preprocessing and network configuration
The protocol used nnU-Net’s selected full-resolution configuration and trained one model per deterministic fold for 1,000 epochs. Training followed the stock nnU-Net plan without hyperparameter search.
2.2 Preprocessing and network configuration
- The six-stage 3D PlainConvUNet used 1-mm isotropic inputs, z-score normalization, 128 × 160 × 112 patches, batch size 2, and overlapping WT, TC, and ET outputs.Each encoder and decoder stage used two 3 × 3 × 3 convolutions, with instance normalization and leaky ReLU.Source: §2.2 · p. 2
2.3 Training protocol
- Each deterministic fold was trained from scratch for 1,000 epochs with the standard nnU-Net loss, deep supervision, foreground oversampling, augmentations, and polynomial learning-rate decay.The final epoch-1,000 checkpoint was retained; no hyperparameter search was performed, and training plus OOF validation averaged 12.3 hours per fold on one A100 GPU.Source: §2.3 · p. 3
2.4 Five-fold inference and test-time augmentation
The final inference configuration averaged predictions across five folds and mirrored spatial axes. It used probability averaging without connected-component post-processing or tuned region thresholds.
2.4 Five-fold inference and test-time augmentation
- The final predictor averaged folds 0–4 and mirrored predictions over spatial axes 0, 1, and 2.Probability maps were averaged before converting overlapping regions to atomic labels, and the input geometry was restored.Source: §2.4 · p. 3
- This cross-validation ensemble used models trained on different 80% subsets, rather than repeated training on the full dataset [8].The ensemble reused models required for OOF analysis.Source: §2.4 · p. 3
- Inference applied no connected-component post-processing or tuned region threshold.Source: §2.4 · p. 3
2.5 Alternative architecture and ablation design
The alternative architecture changed the planned encoder and patch geometry while preserving the training schedule, split, and inference procedure. Validation metrics and paired contrasts were defined for post hoc configuration comparisons.
2.5 Alternative architecture and ablation design
- The ResidualEncoderUNet-L used 160 × 192 × 160 patches and batch size 3 with the same trainer and 1,000-epoch schedule.It changed the automatically planned encoder and patch geometry while preserving the split and inference procedure.Source: §2.5 · p. 3
- Five configurations compared fold 0 with five-fold probability averaging, with and without mirroring, plus the residual alternative.Validation results were used only for post hoc comparison, not checkpoint selection or parameter tuning.Source: §2.5 · p. 3
2.6 Validation metrics
- DSC and NSD were the announced ranking metrics, while HD95 was retained as a complementary diagnostic.The server's NSD tolerance and unit were undisclosed, so reported NSD values were not assigned a millimeter tolerance.Source: §2.6 · pp. 3–4
- Marginal metric means and case-level standard deviations used finite cases; paired contrasts used jointly finite cases and 10,000-resample percentile bootstrap intervals.Because configurations were selected post hoc, the analysis reported effect estimates rather than hypothesis tests.Source: §2.6 · p. 4
2.7 Pooled source–validation comparison
The comparison applied the same mirrored fold-0 checkpoint to held-out source cases and the official validation cohort. Its pooled-validation minus source OOF difference is descriptive and combines evaluator, case-mixture, and possible distributional effects.
2.7 Pooled source–validation comparison
- The same mirrored fold-0 checkpoint was evaluated on 271 held-out source cases and the official validation cohort.The reported performance difference was pooled-validation DSC minus source OOF DSC.Source: §2.7 · p. 4
- Source and validation cases were resampled independently, with regional mean recomputed in each of 10,000 bootstrap replicates.This procedure was used for the model-matched comparison.Source: §2.7 · p. 4
- The descriptive difference combines evaluator implementation, case mixture, and possible distributional effects, rather than estimating pure population shift.Source: §2.7 · p. 4
2.8 Out-of-fold failure analysis
The failure analysis defined cases using strict mean regional Dice and characterized reference-mask morphology. Its uncertainty estimates condition on fixed trained models, and the analyses remain exploratory.
2.8 Out-of-fold failure analysis
- Failures were cases in the bottom decile of strict mean ET/TC/WT Dice among 1,343 cases with three finite values.Threshold robustness was assessed with alternative bottom-5% and bottom-20% definitions.Source: §2.8 · p. 4
- Reference morphology included volume, 26-connected components of at least 10 voxels, largest-component fraction, and ET/WT ratio.At 1-mm isotropic spacing, voxel counts were divided by 1,000 to report volumes in mL.Source: §2.8 · p. 4
- Bootstrap intervals quantify case-sampling uncertainty with trained models fixed, omitting retraining and hidden-population-mixture uncertainty.All failure analyses were exploratory because hypotheses and estimates used the same cases and folds had overlapping training sets.Source: §2.8 · p. 4
2.9 Container and reproducibility controls
The offline container pins the inference software stack and validates input modalities, cases, and geometry. Code and checkpoints are available, but challenge data and outputs are not redistributed.
2.9 Container and reproducibility controls
- The offline Linux/AMD64 container pins nnU-Net 2.6.2, PyTorch 2.6.0, CUDA 12.4, and cuDNN 9.It reads /input and writes one flat NIfTI map per case to /output.Source: §2.9 · p. 4
- Input modalities, cases, and geometry are validated against T1n.Code, nnU-Net plans, container files, inference configuration, validation checks, and analysis scripts are available in the repository.Source: §2.9 · pp. 4–5
- Challenge data, predictions, and official score exports are not redistributed, so analyses relying on them support inspection but not clean-clone regeneration.The versioned bundle of five epoch-1,000 checkpoints is linked from the repository.Source: §2.9 · p. 5
3 Results
Five-fold TTA had the highest pooled-validation mean DSC and NSD, while matched fold-0 performance was lower on pooled validation than on source OOF cases. In labeled OOF predictions, lower Dice was associated with smaller ET volume and less favorable ET component morphology.
3.1 Internal five-fold cross-validation
- The unweighted mean of the five final nnU-Net fold-summary foreground Dice values was 0.9022 (SD 0.0073).Pooled per-case OOF regional Dice was 0.8674 for ET, 0.9125 for TC, and 0.9267 for WT; the strict complete-case mean was 0.9034.Source: §3.1 · p. 5
3.2 Official validation results
- The five-fold TTA model had the highest pooled-validation mean DSC (0.8315) and NSD (0.5164).Fold-0 TTA reached 0.8310 DSC and 0.5140 NSD; the residual configuration had lower DSC point estimates in every region, with paired intervals against PlainConv including zero.Source: §3.2 · p. 5, Table 1 · p. 6
3.3 Quantitative ablation effects
- Mirroring gave small single-fold DSC gains, but five-fold ensembling showed no clear paired DSC improvement over fold 0.Fold-0 TTA increased ET and WT DSC by 0.0021 and 0.0011; after ensembling, every DSC and NSD change was within 0.0008 of zero, and residual-minus-conventional DSC intervals included zero.Source: §3.3 · p. 5
- The exploratory threshold-and-component-size sweep worsened ET DSC by −0.00072 and ET NSD by −0.00225.Because calibration-fold models had seen some evaluated cases during training, the sweep was not fully nested and did not inform the final pipeline.Source: §3.3 · p. 5
3.4 Descriptive pooled source–validation gap
- Matched fold-0 TTA mean regional DSC decreased from 0.9058 on source OOF cases to 0.8310 on pooled validation (−0.0747).The 95% interval was −0.0976 to −0.0523; hidden labels precluded population-specific attribution.Source: §3.4 · p. 6, Table 2 · p. 7
3.5 Failure conditions in labeled out-of-fold predictions
- Failure cases had a median reference ET volume of 1.04 mL, versus 18.66 mL for non-failure cases.The negative rank-biserial effect persisted under bottom-5%, bottom-10%, and bottom-20% failure definitions.Source: §3.5 · p. 6
- After adjustment for ET and WT volume, more ET components were associated with lower Dice and a larger dominant component with higher Dice.The ET/WT volume ratio added no clear association after size adjustment (ρ = −0.039, q = 0.151); all four targeted analyses remained exploratory.Source: §3.5 · pp. 6, 8
- Illustrative OOF ET errors included small missed ET, incomplete fragmented ET, and local boundary undersegmentation.The selected examples illustrate only these error modes, not their frequencies or population effects.Source: Figure 3 · p. 8, §3.5 · p. 8
4 Discussion and Conclusion
Conventional nnU-Net provided a transparent baseline, while tested additions showed no clear improvement and validation DSC was lower than source OOF DSC. Interpretation is limited by pooled validation and source-only failure associations.
4 Discussion and Conclusion
- Validation DSC was lower than source OOF DSC, and tested additions gave no clear improvement; mirroring helped fold 0 but not the ensemble.Five-fold TTA was selected post hoc and does not establish general superiority.Source: §4 · pp. 8–9
- OOF failures concentrated in small, fragmented ET, but these source-only mask associations are noncausal, not deployable, and may not characterize SSA or PED.Source: §4 · p. 8
- The source–validation gap combines evaluator, case-mixture, and possible distributional effects rather than isolating pure distribution shift.The discussion also notes hidden validation labels and population mapping among the limitations.Source: §4 · p. 8