Source-linked AI summary
nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation
Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier-Hein, Paul F. Jaeger
TL;DR
The paper addresses weak validation behind claims that novel architectures outperform U-Net-based segmentation methods. It conducts a controlled benchmark and finds that CNN-based U-Nets, nnU-Net, and appropriate model scaling provide the strongest supported recipe for performance.
Problem
Many reported architectural superiority claims rely on inadequate baselines, insufficient datasets, inconsistent reporting, or unequal computational settings.
Method
The study conducts a comprehensive benchmark using standardized configurations, shared VRAM limits, consistent cross-validation splits, and a dataset-suitability analysis.
Results
CNN-based U-Nets consistently perform strongly, nnU-Net outperforms Auto3DSeg in five of six identical SwinUNETR comparisons, and scaling improves challenging tasks.
Takeaways & Limitations
Meaningful progress requires rigorous validation against strong standardized baselines, suitable datasets, and appropriately scaled models.
Abstract
from arXiv · showhide
The release of nnU-Net marked a paradigm shift in 3D medical image segmentation, demonstrating that a properly configured U-Net architecture could still achieve state-of-the-art results. Despite this, the pursuit of novel architectures, and the respective claims of superior performance over the U-Net baseline, continued. In this study, we demonstrate that many of these recent claims fail to hold up when scrutinized for common validation shortcomings, such as the use of inadequate baselines, insufficient datasets, and neglected computational resources. By meticulously avoiding these pitfalls, we conduct a thorough and comprehensive benchmarking of current segmentation methods including CNN-based, Transformer-based, and Mamba-based approaches. In contrast to current beliefs, we find that the recipe for state-of-the-art performance is 1) employing CNN-based U-Net models, including ResNet and ConvNeXt variants, 2) using the nnU-Net framework, and 3) scaling models to modern hardware resources. These results indicate an ongoing innovation bias towards novel architectures in the field and underscore the need for more stringent validation standards in the quest for scientific progress.
1 Introduction
The paper argues that rapid adoption of novel architectures in medical image segmentation has outpaced stringent validation. It proposes rigorous benchmarking to distinguish genuine methodological progress from unsupported superiority claims.
- Recent architectural innovations, including Transformers and Mamba, continue to attract attention despite nnU-Net showing that configuration can matter more than architectural modification.
- The study identifies inadequate validation as a source of unsupported claims of methodological superiority.
- The authors contribute a systematic analysis of validation pitfalls, a large-scale benchmark, suitable benchmarking datasets, and standardized nnU-Net residual-encoder baselines.
2 Validation Pitfalls
The paper identifies recurring validation practices that can distort comparisons and weaken claims of methodological superiority. These include confounded innovations, poorly configured baselines, insufficient or unsuitable datasets, and inconsistent reporting.
- Validation recommendations accompany identified pitfalls because empirical benchmarking shows that these shortcomings can produce unsupported superiority claims.
- Coupling a claimed innovation with residual connections, additional data, or self-supervised pretraining can artificially boost performance relative to weaker baselines.
- Poorly configured or nonstandardized baselines make architectural superiority claims misleading because configuration can affect performance more than architecture.
- Recent studies claiming superior segmentation performance use a median of three datasets, despite biomedical datasets varying widely in benchmarking suitability.
- Nonstandardized ensembling, test-time augmentation, post-processing, splits, and selective reporting can undermine meaningful comparisons and make minor gains unstable.
3 Systematic 3D Medical Segmentation Benchmark
The study performs a systematic benchmark of segmentation methods using standardized configurations, shared hardware limits, multiple datasets, and a dataset-suitability measure. Its protocol is designed to make methodological differences more interpretable.
- The benchmark revisits recently proposed methods using a systematic and comprehensive evaluation.
- Methods are categorized into CNN-based, Transformer-based, and Mamba-based groups, including nnU-Net configurations and residual-encoder variants.
- Configurations use available self-configuration, dataset-matched settings, defaults, or reduced learning rates, while trainings share a 40GB NVIDIA A100 VRAM budget.
- The benchmark uses six popular datasets selected to assess their suitability for measuring methodological differences.
- Dataset suitability combines low intra-method DSC variability with high inter-method variability, using their ratio as the final suitability score.
- Results use five-fold cross-validation with consistent nnU-Net splits, using average DSC as the primary metric and NSD as the secondary metric.
4 Results and Discussion
The benchmark finds that CNN-based U-Nets in nnU-Net generally outperform Transformer- and Mamba-based alternatives, while dataset suitability and model scaling substantially affect comparisons. Larger compute budgets particularly improve performance on challenging tasks, and benchmark conclusions depend on rigorous, size-aware evaluation.
- Dataset suitability: KiTS, AMOS, and ACDC are the most suitable datasets for benchmarking 3D segmentation because they combine low statistical noise with strong method differentiation.KiTS has the highest inter-method variation, indicating the lowest performance saturation; BraTS21 is saturated, while BTCV has more noise than signal.
- Benchmark overview: Table 1 reports DSC scores and runtime across prevalent 3D segmentation methods, with higher normalized DSC values shown in greener colors.Runtime is measured in GPU hours on an A100 40GB PCIe.
- Architecture comparison: CNN-based U-Nets consistently deliver strong performance across all six datasets, with MedNeXt leading except on KiTS.MedNeXt’s gains require substantially more training time, and some advantages are attributable to target-spacing selection rather than architecture alone.
- Architecture comparison: Transformer architectures fail to match CNN performance, including the original nnU-Net, while U-Mamba’s apparent gains disappear against the No-Mamba Base baseline.The reported U-Mamba gains instead reflect coupling with a residual U-Net.
- Framework comparison: nnU-Net outperforms Auto3DSeg in the reported comparisons, winning with identical SwinUNETR implementations on five of six datasets.Auto3DSeg uses less VRAM and training time, but manual configuration and additional compute still did not produce competitive performance in this benchmark.
- Model scaling: Increasing model compute significantly improves performance on AMOS and KiTS, whereas BTCV and BraTS offer less potential for scaling gains.Meaningful comparisons therefore require awareness of both dataset difficulty and model size.
5 Conclusion
The study concludes that most recent methods do not surpass the original nnU-Net baseline, linking this pattern to insufficiently rigorous validation. It calls for standardized baselines and a field-wide shift that values validation quality alongside architectural novelty.
- Most methods introduced in recent years fail to surpass the original nnU-Net baseline.
- The study links unsupported superiority claims to inadequate validation and introduces systematic pitfalls, standardized baselines, and dataset-suitability analysis.
- Sustained progress requires developers, users, and reviewers to value validation quality as much as network-architecture novelty.