Source-linked AI summary
The state of the art in kidney and kidney tumor segmentation in contrast-enhanced CT imaging: Results of the KiTS19 Challenge
Nicholas Heller, Fabian Isensee, Klaus H. Maier-Hein, Xiaoshuai Hou, Chunmei Xie, Fengyi Li, Yang Nan, Guangrui Mu, Zhiyong Lin, Miofei Han, Guang Yao, Yaozong Gao, Yao Zhang, Yixin Wang, Feng Hou, Jiawei Yang, Guangwei Xiong, Jiang Tian, Cheng Zhong, Jun Ma, Jack Rickman, Joshua Dean, Bethany Stai, Resha Tejpaul, Makinna Oestreich, Paul Blake, Heather Kaluzniak, Shaneabbas Raza, Joel Rosenberg, Keenan Moore, Edward Walczak, Zachary Rengel, Zach Edgerton, Ranveer Vasdev, Matthew Peterson, Sean McSweeney, Sarah Peterson, Arveen Kalapara, Niranjan Sathianathen, Nikolaos Papanikolopoulos, Christopher Weight
TL;DR
Manual 3D kidney and tumor segmentation limits quantitative characterization, while the best-performing methods remain uncertain. KiTS19 released labeled CT data and evaluated automated predictions on a private test set, with the winning system achieving kidney Dice 0.974 and tumor Dice 0.851. The challenge established an open leaderboard, although dataset sampling and challenge-design limitations remain.
Problem
Manual production of high-quality 3D kidney and tumor segmentations limits adoption of quantitative characterization methods, while consensus on the best automatic approaches remains limited.
Method
KiTS19 released 210 labeled CT images for training and evaluated teams on predictions for 90 private test cases using average Sørensen-Dice across kidneys and tumors.
Results
0.974 kidney Dice and 0.851 tumor Dice were achieved by the winning team on the private 90-case test set.
Takeaways & Limitations
The challenge now provides an open leaderboard and a challenging benchmark for 3D semantic segmentation.
Takeaways & Limitations
The dataset is not fully representative because it excludes misdiagnosed cysts and cases with tumor thrombus, introducing a sampling-bias trade-off.
Abstract
from arXiv · showhide
There is a large body of literature linking anatomic and geometric characteristics of kidney tumors to perioperative and oncologic outcomes. Semantic segmentation of these tumors and their host kidneys is a promising tool for quantitatively characterizing these lesions, but its adoption is limited due to the manual effort required to produce high-quality 3D segmentations of these structures. Recently, methods based on deep learning have shown excellent results in automatic 3D segmentation, but they require large datasets for training, and there remains little consensus on which methods perform best. The 2019 Kidney and Kidney Tumor Segmentation challenge (KiTS19) was a competition held in conjunction with the 2019 International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) which sought to address these issues and stimulate progress on this automatic segmentation problem. A training set of 210 cross sectional CT images with kidney tumors was publicly released with corresponding semantic segmentation masks. 106 teams from five continents used this data to develop automated systems to predict the true segmentation masks on a test set of 90 CT images for which the corresponding ground truth segmentations were kept private. These predictions were scored and ranked according to their average So rensen-Dice coefficient between the kidney and tumor across all 90 cases. The winning team achieved a Dice of 0.974 for kidney and 0.851 for tumor, approaching the inter-annotator performance on kidney (0.983) but falling short on tumor (0.923). This challenge has now entered an "open leaderboard" phase where it serves as a challenging benchmark in 3D semantic segmentation.
1. Introduction
Kidney tumor segmentation could support objective characterization and treatment decisions, but manual annotation remains burdensome. KiTS19 addressed this by releasing labeled CT data and evaluating automated methods on private cases.
- Clinical motivation: Kidney tumors increasingly require treatment decisions distinguishing radical nephrectomy, partial nephrectomy, and active surveillance.The motivation is to quantify tumor complexity and aggression more objectively using imaging.
- Clinical motivation: Manual nephrometry scores have limited adoption because they require substantial effort, show interobserver variability, and have limited predictive power.These limitations motivate more expressive and reproducible computational characterization.
- Clinical motivation: Reliable automatic semantic segmentation would automate several nephrometry scores and enable kidney tumor morphology studies at unprecedented scales.Segmentation delineates kidneys and tumors in three-dimensional imaging, but producing masks manually imposes an even larger burden than most nephrometry scores.
- Challenge design: KiTS19 released 210 CT images with high-quality kidney and tumor segmentations for training learned models.The challenge also reserved 90 segmentation masks for participants to predict and evaluation.
- Challenge design: 106 teams submitted valid predictions, while 100 complete submissions appeared on the official leaderboard.The challenge was held with MICCAI 2019 and later entered an open-leaderboard phase allowing continued submissions.
2. Related Work
Biomedical segmentation challenges provide shared benchmarks and public data, addressing inconsistent comparisons and limited reproducibility in 3D medical-image research. KiTS19 also illustrates that a well-tuned U-Net baseline can outperform more elaborate architectural modifications.
- Challenges in biomedical imaging: Biomedical imaging challenges enable objective method comparisons on centrally maintained benchmarks and contribute public data for further research.Shared evaluation reduces inconsistencies caused by separate private datasets and sampling variation.
- Challenges in biomedical imaging: Challenges can encourage overfitting to quirks of a particular test set, so organizers emphasize continued improvement through experience and future iterations.The authors identify this as a general limitation of biomedical imaging challenges.
- 3D segmentation: 3D segmentation is especially demanding because annotation costs and training memory requirements increase, while transfer learning from major 2D vision benchmarks is limited.These constraints make standardized datasets and evaluations particularly valuable for 3D biomedical imaging.
- 3D segmentation: Private-dataset reporting remains common, and more than half of MICCAI papers still report results on private datasets.This limits comprehensive benchmarking against the state of the art.
- 3D segmentation: nnU-Net achieved state-of-the-art performance across several 3D segmentation challenges using a U-Net with systematic hyperparameter and preprocessing search.Its methodology searches a small configuration space rather than relying primarily on architectural novelty.
- KiTS19 and nnU-Net: KiTS19 found that vanilla nnU-Net achieved top performance despite several teams attempting to augment the baseline with architectural enhancements.The result challenges the expectation that architectural additions would determine the winning system.
3. Materials and Methods
KiTS19 assembled and annotated a 300-patient CT dataset, then used private test masks, standardized infrastructure, and a Dice-based evaluation to benchmark segmentation systems.
- 3.1. The KiTS19 Dataset: The dataset comprised 300 patients selected retrospectively after exclusions for unavailable late-arterial imaging, cysts, and tumor thrombus.Cases underwent manual kidney and tumor delineation, with additional annotations on 30 training cases used to estimate interobserver agreement.
- 3.1. The KiTS19 Dataset: Postoperative exclusion of misdiagnosed cysts introduced sampling bias because the cohort is not fully representative of patients undergoing nephrectomy for suspected renal cancer.The organizers accepted this limitation to avoid semantic conflicts in class labels.
- 3.1. The KiTS19 Dataset: Semantic segmentation was chosen over instance segmentation because boundaries between abutting tumors and joined horseshoe kidneys can be inherently ambiguous.The authors considered semantic segmentation to provide nearly the same clinical utility without adding label ambiguity.
- 3.1. The KiTS19 Dataset: More than half of scans came from over 50 referring institutions despite treatment at one tertiary center, producing diversity in acquisition protocols and voxel dimensions.The authors note that a prospectively collected multi-institution test set would be preferable in future work.
4. Results
The challenge attracted 100 official leaderboard teams using deep neural networks, with kidney segmentation substantially outperforming tumor segmentation. Ranking was generally stable near β=1, although comparisons around the median were less certain.
- 100 teams appeared on the official MICCAI leaderboard after duplicate-team and manuscript checks.126 users submitted; 20 belonged to previously submitting teams, and six submissions were disqualified for insufficient manuscripts.
- Deep neural networks formed the basis of every submission, despite substantial variation in preprocessing, architecture, and training procedures.
- 0.915 mean kidney Dice exceeded 0.580 mean tumor Dice across teams.The corresponding standard deviations were 0.0469 for kidney and 0.212 for tumor; kidneys were described as more predictable anatomically.
- Sensitivity to Choice of Metric: Dice was selected as the Fβ-Score special case with β=1 because precision and recall were treated as equally important.
- Statistical Analysis of Ranking Order: 15-20% was the maximum leaderboard traversal observed under extreme β values, indicating strong ranking stability around Dice.The top and bottom of the leaderboard showed the highest stability, while median submissions were more concentrated and unstable.
- Statistical Analysis of Ranking Order: The significance map identifies team pairs whose apparent ordering cannot be declared statistically significant while controlling a 5% false-positive chance.
5.1. First Place: An Attempt at Beating the 3D U-Net
The first-place method tested several 3D U-Net variants with cross-validation and selected an ensemble of residual 3D U-Nets for final prediction. It achieved the highest reported test performance in the challenge.
- Architecture: Three 3D U-Net architectures used 3D convolutions, leaky ReLU activations, instance normalization, and convolutional upsampling and downsampling.The variants were plain, residual, and pre-activation residual 3D U-Nets.
- Training: Training used randomly sampled 80 × 160 × 160 patches, combined cross-entropy and Dice loss with deep supervision, stochastic gradient descent, and extensive augmentation.
- Training: Known mislabeled cases and four cases consistently disagreeing with predictions were excluded from training.
- Results: 0.974 kidney Dice and 0.857 tumor Dice were obtained in five-fold cross-validation by the Residual 3D U-Net.This was marginally higher than the other tested approaches, including an ensemble of all three architectures.
- Architecture: The final system ensembled Residual 3D U-Net models from all five cross-validation folds.
- Results: 0.974 kidney Dice, 0.851 tumor Dice, and 0.912 composite score produced first place.
5.2. Second Place: Cascaded Semantic Segmentation for Kidney and Tumor
The second-place system used a three-stage cascade that progressively localized kidneys, segmented kidney regions, and separated tumors from kidney tissue. It achieved a 0.906 composite score.
- Architecture: The method used low-resolution images for coarse localization and high-resolution cropped regions for fine-grained kidney and lesion delineation.The reported resolutions were 1.72 × 1.72 × 3.41 mm and 0.781 × 0.781 × 0.781 mm, respectively.
- Architecture: The cascade first coarsely localized kidneys, then finely segmented each kidney against background, and finally segmented tumor from kidney.
- Training: Models combined cross-entropy and Dice loss with elastic deformation, rotation, random cropping, and Adam optimization.Training used learning-rate reduction after insufficient improvement and early termination after prolonged validation stagnation.
- Postprocessing: Postprocessing filled holes in tumor predictions and removed some regions judged to be false positives.
- Results: 0.967 kidney Dice, 0.845 tumor Dice, and 0.906 composite score produced second place.
5.3. Third Place: Segmentation of kidney tumor by multi-resolution VB-nets
The third-place method used multi-resolution cascaded V-Nets to localize kidneys at low resolution and predict final segmentations at high resolution. Its final composite score was 0.903.
- Architecture: The architecture extended V-Net with bottleneck blocks containing three convolutional layers.The first and last 1 × 1 × 1 convolutions reduced and restored feature-map counts around a spatial convolution.
- Architecture: The cascade generated low-resolution kidney Volumes of Interest and passed them to a high-resolution model for final segmentation.The reported voxel sizes were 6 × 6 × 6 mm for localization and 1 × 1 × 1 mm for final prediction.
- Training: Training sampled random 96 × 96 × 96 patches and used generalized Dice loss with Adam at a constant learning rate of 1e-4.
- Training: Generalized Dice loss compares predicted class probabilities with binary voxel labels across the C class labels.The prose defines p_c(i) as the predicted probability for class c at voxel i and g_c(i) as its binary label.
- Postprocessing: Connected-component filtering removed small kidney regions, while spatial relationships and average HU values helped classify tumors versus cysts.
- Results: 0.973 kidney Dice, 0.832 tumor Dice, and 0.903 composite score produced third place.
5.4. Fourth Place: Cascaded Volumetric Convolutional Network for Kidney Tumor Segmentation from CT volumes
The fourth-place submission used a cascaded 3D U-Net pipeline that localized kidneys coarsely before producing finer kidney and tumor segmentations. Its final predictions achieved a 0.902 composite score, with kidney Dice substantially higher than tumor Dice.
- Method: A 3D U-Net first localized each kidney at lower resolution, after which cropped full-resolution patches were processed by a finer segmentation network.The coarse-to-fine pipeline used voxel spacing 3×1.56×1.56 mm for localization before fine prediction.
- Method: The architecture used nnU-Net-based 3D U-Nets with instance normalization for both coarse localization and fine prediction.The baseline extracted 30 feature maps at the original resolution.
- Method: Deep supervision, residual blocks, and a spatial prior jointly produced the best validation performance and were selected for test-time prediction.The spatial prior fed coarse predictions from the first network into the second as an input channel.
- Method: Decoder-stage predictions were ensembled by majority voting to produce the final predictions.This deep-prediction step complemented deep supervision.
- Results: 0.974 kidney Dice, 0.831 tumor Dice, and 0.902 composite score placed the submission fourth.The team used only the official training set, with random parameter initialization and no transfer learning.
5.5. Fifth Place: Cascaded U-Net Ensembles
The fifth-place submission combined nnU-Net-based 3D U-Net models, cascaded prediction, test-time augmentation, and heuristic post-processing. It achieved a 0.899 composite score, again performing better on kidney than tumor segmentation.
- Method: Five-fold cross-validation was used, with CT intensities clipped at the 0.5 and 99.5 percentiles before standardization.The data came from the interpolated repository branch.
- Method: The submission used a main nnU-Net-based 3D U-Net architecture with three models ensembled for kidney and tumor prediction.Kidney outputs were combined by majority voting, whereas tumor outputs used an OR operation.
- Method: Mirrored test-time augmentation averaged predictions to increase robustness.
- Method: A TopK loss was used for cascaded models to prevent tumor predictions in the upstream model.The baseline training objective combined cross-entropy and Dice losses, optimized with Adam.
- Results: 0.973 kidney Dice, 0.825 tumor Dice, and 0.899 composite score resulted in fifth place.Each model required about four days of training on one Titan-Xp GPU and two Intel Xeon CPUs.
6. Discussion
The discussion highlights that sophisticated segmentation innovations were uncommon among the highest-performing submissions, while also stressing that benchmark performance has limited clinical scope. KiTS19 primarily supports segmentation-based characterization of patients already diagnosed with renal tumors.
- Discussion: Few sophisticated methods appeared among the highest-performing submissions, despite the large number of proposed techniques in medical image segmentation.This pattern was described as consistent with results from many concurrent challenges.
- Clinical scope: All KiTS19 cases contained at least one renal tumor, limiting confidence that high-performing systems would perform similarly on healthy controls or the general population.
- Clinical scope: The intended use case is segmenting and characterizing kidneys and tumors in patients already diagnosed with a renal tumor.Examples include computing volume or surface area and analyzing enhancement patterns.
- Clinical scope: Most imaging came from Minnesota, North Dakota, and western Wisconsin, while the random train-test split constituted internal validation.A temporally and geographically separate test dataset would better estimate generalization to future data and other centers.
7. Limitations
KiTS19 was a successful but imperfect benchmark whose conclusions are constrained by cohort composition, label imperfections, challenge design, and limited reproducibility. Planned revisions target broader data, temporal validation, improved review, clearer error communication, and richer segmentation tasks.
- Dataset: The cohort was geographically concentrated, retrospectively assembled, and randomly split, raising concerns about generalization across regions and to prospective data.The authors specifically note possible distributional shift over time.
- Dataset: The sampled patients were only a subset of those evaluated for renal malignancy, with tumor thrombus and cyst cases excluded.The cohort’s benign-lesion proportion was consequently lower than reported elsewhere, 8% versus 30%.
- Dataset: Semantic segmentation labels remained imperfect, ranging from boundary noise to cases in which an entire structure was mislabeled.
- Challenge design: The 20-day public label-review period was too short to prevent some errors from remaining in the official training set.The authors suggest freezing the data about one month before test release.
- Challenge design: Publicly released test images allowed the possibility of manual prediction refinement or segmentation, which could not be excluded entirely.A private-access prediction kernel would have reduced this risk.
- Reproducibility: Few submissions were open source, reflecting a tradeoff between demanding detailed reporting and attracting more high-performing teams.Some submitted manuscripts also fell short of supporting full replication.
- Future directions: Planned improvements include multi-institutional and pseudo-prospective cohorts, longer data review, clearer label-error communication, rare-subtype representation, and additional segmentation classes.Proposed new structures include renal cyst, renal artery and vein, and ureter.
- Future directions: Implementing broader cohorts and more segmentation classes will require significant effort and may be phased into future challenges.
8. Conclusion
KiTS19 accelerated and measured the state of the art in automatic kidney and kidney tumor segmentation on contrast-enhanced CT. The winning system achieved strong kidney segmentation, while the challenge continues as an open benchmark for 3D semantic segmentation.
- Over 100 teams submitted systems, and the highest-scoring team achieved kidney and tumor Dice scores of 0.974 and 0.851, respectively.These scores were obtained on the private 90-case test set.
- The winning experiments found no meaningful benefit from several reported enhancements over the 3D U-Net baseline.The winning team instead achieved its margin by submitting predictions from the baseline+residual connections model alone.
- The challenge has entered an indefinite open-leaderboard phase as a high-quality, challenging benchmark in 3D semantic segmentation.
- A planned second KiTS iteration will target greater clinical significance and external validity while increasing difficulty with additional anatomical structures.Proposed structures include ureters, renal arteries, and renal veins.