Source-linked AI summary
The Medical Segmentation Decathlon
Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, AnnetteKopp-Schneider, Bennett A. Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M. Summers, Bram van Ginneken, Michel Bilello, Patrick Bilic, Patrick F. Christ, Richard K. G. Do, Marc J. Gollub, Stephan H. Heckers, Henkjan Huisman, William R. Jarnagin, Maureen K. McHugo, Sandy Napel, Jennifer S. Goli Pernicka, Kawal Rhode, Catalina Tobon-Gomez, Eugene Vorontsov, Henkjan Huisman, James A. Meakin, Sebastien Ourselin, Manuel Wiesenfarth, Pablo Arbelaez, Byeonguk Bae, Sihong Chen, Laura Daza, Jianjiang Feng, Baochun He, Fabian Isensee, Yuanfeng Ji, Fucang Jia, Namkug Kim, Ildoo Kim, Dorit Merhof, Akshay Pai, Beomhee Park, Mathias Perslev, Ramin Rezaiifar, Oliver Rippel, Ignacio Sarasua, Wei Shen, Jaemin Son, Christian Wachinger, Liansheng Wang, Yan Wang, Yingda Xia, Daguang Xu, Zhanwei Xu, Yefeng Zheng, Amber L. Simpson, Lena Maier-Hein, M. Jorge Cardoso
TL;DR
Medical segmentation challenges have usually targeted isolated clinical problems, leaving uncertainty about whether strong methods generalize to unseen tasks. The paper introduces the Medical Segmentation Decathlon, evaluates algorithms across diverse tasks and modalities, and finds that consistent multi-task performance predicts continued generalization.
Problem
Segmentation challenges have typically been organized around single clinical problems, leaving the generalizability of methods to unseen tasks uncertain.
Method
The study organizes the Medical Segmentation Decathlon across ten tasks and modalities, designs data around common medical-imaging difficulties, and evaluates algorithms with standardized rankings.
Results
Consistently strong algorithms preserved good average performance on unseen tasks, and the monitored winning method continued achieving state-of-the-art performance across many clinical problems.
Takeaways & Limitations
Consistent performance across multiple tasks is a strong surrogate for algorithmic generalizability, and accurate segmentation models can be trained without task-specific manual optimization.
Takeaways & Limitations
The data set uses retrospectively acquired radiological data with heterogeneous protocols, single-rater annotations, and limited scope beyond radiology.
Abstract
from arXiv · showhide
International challenges have become the de facto standard for comparative assessment of image analysis algorithms given a specific task. Segmentation is so far the most widely investigated medical image processing task, but the various segmentation challenges have typically been organized in isolation, such that algorithm development was driven by the need to tackle a single specific clinical problem. We hypothesized that a method capable of performing well on multiple tasks will generalize well to a previously unseen task and potentially outperform a custom-designed solution. To investigate the hypothesis, we organized the Medical Segmentation Decathlon (MSD) - a biomedical image analysis challenge, in which algorithms compete in a multitude of both tasks and modalities. The underlying data set was designed to explore the axis of difficulties typically encountered when dealing with medical images, such as small data sets, unbalanced labels, multi-site data and small objects. The MSD challenge confirmed that algorithms with a consistent good performance on a set of tasks preserved their good average performance on a different set of previously unseen tasks. Moreover, by monitoring the MSD winner for two years, we found that this algorithm continued generalizing well to a wide range of other clinical problems, further confirming our hypothesis. Three main conclusions can be drawn from this study: (1) state-of-the-art image segmentation algorithms are mature, accurate, and generalize well when retrained on unseen tasks; (2) consistent algorithmic performance across multiple tasks is a strong surrogate of algorithmic generalizability; (3) the training of accurate AI segmentation models is now commoditized to non AI experts.
1. Introduction
Medical image segmentation is widely studied and clinically important, yet methods developed for one task may not generalize to unseen problems. The MSD addresses this gap by benchmarking algorithms across multiple tasks and modalities.
- Semantic segmentation converts medical images into spatially structured information needed for applications such as radiotherapy planning and treatment monitoring.
- About 70% of biomedical image analysis challenges focus on segmentation, making it difficult to choose a baseline architecture for a new clinical problem.
- A method suited to one clinical problem may not generalize to unseen tasks, motivating the search for a fully automated generalizable learner.
- The MSD organizes competition across multiple tasks and modalities while explicitly testing difficulties including small data sets, unbalanced labels, multi-site data, and small objects.
- The paper contributes a multi-task benchmark framework and reports that one framework can generalize across varied clinical applications.
2. Methods
The MSD evaluates general-purpose segmentation algorithms across ten radiological data sets spanning diverse anatomies, modalities, and task complexities. Its two-phase design tests performance on known tasks and transfer to previously unseen tasks using standardized assessment procedures.
- 2.1. Challenge organization: The challenge covered ten disciplines, with seven known development tasks and three mystery tasks reserved for testing generalization.
- 2.1. Challenge organization: In the development phase, teams trained one general-purpose algorithm independently on seven task-specific training sets without human interaction or manual parameter settings.
- 2.1. Challenge organization: The mystery phase required teams to retrain their existing method on hepatic vessels, colon, and spleen data without modifying the method itself.
- 2.3. Challenge data sets: The data sets were selected to vary input modalities, target-region number, shape and size, and surrounding-tissue complexity.
- 2.3. Challenge data sets: The ten data sets contained 17 target regions of interest and were released online under a permissive CC-BY-SA 4.0 license.
- 2.4. Challenge assessment: Algorithm rankings used Wilcoxon signed-rank tests, significance scores, and mean significance ranks across target regions, with task-level averaging preventing multi-ROI tasks from being overweighted.
3. Results
The challenge attracted diverse CNN-based methods, with nnU-Net achieving the most robust rankings across tasks and target regions. Performance varied substantially by task, while nnU-Net’s strong results extended to many later segmentation challenges.
- Challenge submissions: 180 teams registered, 31 submitted complete development results, and 19 submitted valid mystery-phase results.
- Challenge submissions: 64% of mystery-phase methods used U-Net as their base architecture, while DSC loss and cross entropy were the most common loss functions.
- Challenge submissions: nnU-Net automatically adapted preprocessing, network topology, and post-processing independently for each task using the training data.
- Individual performances and rankings: Median mean DSC across participants ranged from 0.16 for mystery-phase colon cancer segmentation to 0.94 for liver and spleen segmentation.
- Individual performances and rankings: nnU-Net was the most robust method, with development-phase ranks ranging only from 1 to 4 across target regions.
- Impact of the challenge winner: Across 53 subsequent segmentation tasks, nnU-Net won 33 and had a median rank of 1 (IQR 1–2).
4. Discussion
The MSD showed that a single algorithm can generalize across ten heterogeneous segmentation tasks, while performance varies substantially with task difficulty. The dataset and benchmark also exposed limitations in annotation reliability and challenge execution, while later results showed continued progress and increasing computational cost.
- A single algorithm generalized across ten different biomedical segmentation disciplines, including unseen tasks, without human-based adjustments.The winning method’s generalizability was further supported by monitoring it for two years on other segmentation tasks.
- The challenge could not initially use containerized organizer-run inference because computational resources were insufficient, leaving possible overfitting or manual labeling concerns.The authors identified containerization as a potentially better control for these issues.
- The MSD dataset combined ten heterogeneous tasks, multiple modalities, and challenging characteristics, and was released under a permissive license for broad reuse.Its heterogeneity supports benchmarking and research on generalizability and domain adaptation.
- The dataset’s retrospective, multi-source annotations used a single human rater, limiting inter-rater reliability assessment and annotation reliability.The study also covered only radiological data, so its conclusions are limited to that application area.
- The dataset used two common metrics, DSC and NSD, to maintain comparability across ten tasks despite differences in clinical use cases and sample sizes.The authors noted that DSC is less suitable for very small objects, but retained both metrics for cross-task comparability.
- Algorithmic performance varied dramatically across tasks: median mean DSC ranged from 0.21 for pancreas tumor mass segmentation to 0.94 for liver segmentation.For nnU-Net, the corresponding median DSC values were 0.52 and 0.93, indicating greater resilience across task difficulties.
- nnU-Net ranked first in both development and mystery phases, while stable rankings among top teams suggested limited methodological overfitting to the development phase.The ranking approach was therefore presented as a surrogate for expected performance on unseen tasks.
- Later rolling challenges produced three methods surpassing the 2018 winner, with NAS improving performance relative to heuristic methods at increased computational cost.The follow-up years also showed gradual improvement among well-performing algorithms and the rise of task-specific Neural Architecture Search.
5. Conclusion
The MSD showed that fully automated segmentation methods can perform accurately across tasks of varying complexity, including previously unseen tasks. Monitoring the winning method further supported generalizability, while classic challenges such as domain shift and label accuracy remain.
- Fully automated methods achieved state-of-the-art performance on previously unseen segmentation tasks without manual parameter optimization.
- The MSD hypothesis that methods performing well across several tasks also generalize to unseen tasks was validated across challenge participants and tasks.
- nnU-Net achieved state-of-the-art performance on many tasks, including comparisons against task-optimized networks, during two years of follow-up monitoring.
- Domain shift and label accuracy remain classic semantic segmentation problems, while NAS and improved heuristics may continue advancing performance and generalizability.
- The challenge concluded that training accurate semantic segmentation networks can now be fully automated.
Author information
The author information identifies shared first authors and shared senior authors for the paper.
- Michela Antonelli and Annika Reinke contributed equally as shared first authors.
- Lena Maier-Hein and M. Jorge Cardoso contributed equally as shared senior authors.
- The author information distinguishes equal contributions among both first-author and senior-author groups.
Appendix A. Challenge organization
The MSD was organized as an open, publicly reported challenge with automated validation, controlled submission policies, standardized metrics, and declared sponsorship and access arrangements.
- Challenge organization: The challenge was organized in the scope of MICCAI 2018 and accepted evaluated submissions after the event through an open-call format.
- Participation policies: Participation required fully automatic methods without task-specific manual parameter settings, while allowing unchanged external pre-training data and limiting each laboratory to one team.
- Submission and validation: Submissions underwent immediate automated validation, teams had one development submission per day, and final development validation used each team’s last submission.
- Evaluation: The organizers supplied Python Notebook implementations of DSC and NSD and provided the statistical validation algorithms before challenge deadlines.
- Participation conditions: Code release was encouraged but not mandatory because some teams faced intellectual property restrictions.
- Funding and sponsorship: The challenge relied mainly on organizers’ in-kind contributions, with NVIDIA, Google DeepMind, and RSIP Vision providing specified forms of support.
- Test-set governance: Only two organizers had access to all test cases, while the KCL organizers committed not to participate in the challenge.
Appendix B. Method details
The appendix notes that the top three teams’ methods are described in the main paper rather than in this appendix.
- The appendix provides method details for remaining teams whose submissions were described separately from the top three teams.
- Methods from the top three teams are presented in Section 3.2 of the main paper.
- The appendix therefore directs readers to the main paper for the leading methods’ descriptions.
Appendix B.0.1. AI-MED
AI-MED used QuickNAT with added Conditional Random Fields for medical image segmentation. Its training combined DSC and cross-entropy losses with SGD, without augmentation or ensembling.
- AI-MED used QuickNAT with added Conditional Random Fields.
- The team combined DSC loss with cross entropy loss during training.
- SGD was used as the optimizer, without data augmentation or ensembling.
Appendix B.0.2. BCVuniandes
BCVuniandes and related teams used varied U-Net, V-Net, CNN, and DeepMedic designs across segmentation tasks. Methods differed in dimensionality, supervision, augmentation, optimization, and ensembling strategies.
- DeepMedic used two identical parallel pathways with multi-scale analysis and concatenated intermediate outputs.The outputs were processed by two fully connected layers, with softmax cross-entropy and Adam.
- Several teams modified U-Net with batch normalization, residual connections, deep supervision, dilation, or dropout.These architectural changes were paired with different augmentation, loss, and optimization choices.
- One U-Net variant was designed to operate on volumes regardless of slice count, using test-time augmentation and depth-wise cross-entropy.The model used concatenated lower-resolution features before upscaling and transpose convolution for upscaling.
- A two-level modified V-Net implemented coarse-to-fine segmentation with automatic region-of-interest extraction.The first level produced an initial coarse segmentation that determined the second level’s input region and size.
- Other approaches used task-specific 2.5D, 3D CNN, and 3D U-Net models, or ensembles combining 2D and 3D U-Nets.These methods incorporated adjacent slices, voxel classification, class weighting, augmentation, and architecture averaging.
Appendix C. DSC values and rankings for all target regions
Appendix C reports mean DSC values and ranking-robustness visualizations across target regions and tasks. The figures cover brain, heart, hippocampus, liver, lung, pancreas, prostate, colon, hepatic vessel, and spleen datasets.
- Tables report mean Dice Similarity Coefficient values for participating teams across target regions and tasks.The appendix includes brain, heart, hippocampus, liver, lung, pancreas, prostate, colon, hepatic vessel, and spleen datasets.
- Ranking-robustness line plots show each algorithm as a colored line across ranking methods.Line height represents rank, while horizontal lines indicate identical ranks across methods.
- The appendix captions identify development-phase tables for prostate and pancreas and mystery-phase tables for colon, hepatic vessel, and spleen.
- Additional ranking-robustness plots cover prostate, colon, hepatic vessel, and spleen tasks.These plots use the same algorithm-line and rank-height encoding described for the other tasks.
Appendix D. Bootstrap ranking analysis
Bootstrap analysis visualizes the distribution of participating algorithms’ achieved ranks across tasks and assesses ranking variability using repeated bootstrap datasets.
- The plot shows achieved algorithm ranks over 1,000 bootstrap datasets for all tasks, with colors identifying tasks.Vertical lines mark algorithms that achieved the same rank across the whole dataset.
- Bootstrap ranking analysis uses a stacked frequency plot to display rank frequencies across algorithms and tasks.
Appendix E. Mean DSC values for the 2018 MSD challenge and the live-decathlon challenge
Figure E.17 summarizes mean DSC values for each task and target ROI across algorithms in the 2018 MSD and live-decathlon challenges.
- The figure uses dot- and box-plots to display mean DSC values for each task and target ROI across all algorithms.
- Each box-plot summarizes the mean DSC values of all participants for a task and target ROI.
- The black line marks the median, box borders mark the first and third quartiles, and vertical lines show 1.5 interquartile ranges.