Source-linked AI summary
Mitosis domain generalization in histopathology images -- The MIDOG challenge
Marc Aubreville, Nikolas Stathonikos, Christof A. Bertram, Robert Klopleisch, Natalie ter Hoeve, Francesco Ciompi, Frauke Wilm, Christian Marzahl, Taryn A. Donovan, Andreas Maier, Jack Breen, Nishant Ravikumar, Youjin Chung, Jinah Park, Ramin Nateghi, Fattaneh Pourakpour, Rutger H. J. Fick, Saima Ben Hadj, Mostafa Jahanifar, Nasir Rajpoot, Jakob Dexl, Thomas Wittenberg, Satoshi Kondo, Maxime W. Lafarge, Viktor H. Koelzer, Jingtang Liang, Yubo Wang, Xi Long, Jingxin Liu, Salar Razavi, April Khademi, Sen Yang, Xiyue Wang, Mitko Veta, Katharina Breininger
TL;DR
Mitotic-figure assessment is clinically relevant but suffers from observer variability and scanner-related domain shift that limits cross-laboratory deployment. The paper evaluates MIDOG challenge approaches designed to improve scanner generalization, finding that suitable architectures and augmentation can compensate for scanner shifts to a high degree, although performance remains weaker on some scanners and broader clinical settings remain untested.
Problem
Mitotic-figure assessment is important for tumor grading, but observer variability and scanner-related domain shifts limit reliable deployment across laboratories.
Method
The paper compares MIDOG challenge approaches using deep-learning architectures, augmentation strategies, and scanner-specific F1 evaluation with bootstrap confidence intervals.
Results
F1 score 0.748 was the overall best test-set performance, achieved by Yang et al. using segmentation and Fourier-domain mixing augmentation.
Takeaways & Limitations
Proper augmentation strategies and deep-learning architectures can compensate for whole-slide-scanner domain shifts to a high degree.
Takeaways & Limitations
Performance was considerably weaker on Scanners D and F, and generalization to whole-slide images and other tissue or cancer types remains challenging.
Abstract
from arXiv · showhide
The density of mitotic figures within tumor tissue is known to be highly correlated with tumor proliferation and thus is an important marker in tumor grading. Recognition of mitotic figures by pathologists is known to be subject to a strong inter-rater bias, which limits the prognostic value. State-of-the-art deep learning methods can support the expert in this assessment but are known to strongly deteriorate when applied in a different clinical environment than was used for training. One decisive component in the underlying domain shift has been identified as the variability caused by using different whole slide scanners. The goal of the MICCAI MIDOG 2021 challenge has been to propose and evaluate methods that counter this domain shift and derive scanner-agnostic mitosis detection algorithms. The challenge used a training set of 200 cases, split across four scanning systems. As a test set, an additional 100 cases split across four scanning systems, including two previously unseen scanners, were given. The best approaches performed on an expert level, with the winning algorithm yielding an F_1 score of 0.748 (CI95: 0.704-0.781). In this paper, we evaluate and compare the approaches that were submitted to the challenge and identify methodological factors contributing to better performance.
1. Introduction
Mitotic-figure detection supports tumor grading but is affected by inter-rater variability and domain shifts across clinical environments. MIDOG focuses on scanner-induced domain shift, which changes image appearance and can undermine cross-laboratory use.
- Mitotic-figure density correlates with tumor proliferation and supports tumor grading, but pathologists show substantial inter-rater variability.
- Deep-learning methods can improve reproducibility and accuracy, yet laboratory-specific factors such as tissue preparation, staining, and scanners limit cross-laboratory deployment.
- Different scanners introduce color shifts and altered depth of field through differences in objective optics.
- MIDOG targets scanner-induced domain shift in mitotic-figure identification because whole-slide scanners can strongly alter the digital image domain.
Challenge format and task
The challenge evaluated fully automatic mitotic-figure detectors on standardized breast-cancer regions of interest acquired across multiple scanners. Its design included scanner variation in training and testing, including unseen scanners, with containerized submissions and platform-based evaluation.
- Challenge format and task: Participants received training data and documentation before submitting fully automatic algorithm containers for platform-based evaluation.
- Challenge format and task: The task was automatic detection of mitotic figures in predefined-size breast-cancer regions of interest.
- Challenge format and task: The training set contained 200 breast-cancer cases scanned by four systems, while the test set contained 80 cases across four scanners, including two unseen scanners.
- Challenge format and task: Participants were restricted from using additional image sources for training, except general-purpose datasets such as ImageNet.
- Challenge format and task: Forty-six users submitted at least one container, and 17 teams submitted to the final test set; 12 teams exceeded the F1 threshold for workshop comparison.
2. Material and methods
The MIDOG challenge assembled a rigorously controlled, multi-scanner breast-cancer dataset and evaluated scanner-robust mitosis detection using standardized labeling and F1-based assessment.
- The challenge dataset was designed to represent a relevant diagnostic task, preserve an independent holdout set, and ensure high label quality.
- The cohort comprised 300 breast cancer cases from a retrospective diagnostic archive, with training and test cases distributed across multiple scanners.The supplied methods passages specify 200 training cases across four scanners and a test set containing known and unseen scanners.
- Image acquisition: Slides were digitized using four scanners, including Hamamatsu NanoZoomer and Leica Aperio systems, with Scanner A serving as the clinical reference scanner.Scanner A was used routinely at UMC Utrecht and provided a reference for region-of-interest selection.
- Image acquisition: Regions of interest were selected on reference scans, registered to other scanner images, manually quality-checked, and extracted across scanners.
- Labeling: Three expert pathologists and a machine-learning-aided annotation strategy were used to address object-level disagreement and improve dataset labeling quality.The labeling process considered both mitotic figures and morphologically similar hard negatives.
- Reference approach: The reference approach used adversarial domain classification with gradient reversal, while the CNN baseline used standard image augmentation without domain-specific techniques.The reference model jointly addressed domain discrimination, object classification, and localization-related objectives; the supplied passage truncates the final loss description.
- Evaluation methods: Overall F1 was selected because precision and recall errors are equally consequential, and bootstrap resampling produced confidence intervals for overall and scanner-specific performance.The bootstrap procedure repeated resampling 10,000 times.
3. Overview of the submitted methods
Submitted methods shared CNN-based modeling but varied in task formulation, multi-stage design, domain adaptation, augmentation, segmentation supervision, and ensembling. No single strategy clearly won overall, although Fourier-domain mixing and segmentation were associated with especially strong approaches.
- All submitted methods used convolutional neural networks, with differences in architectures, augmentation, and normalization strategies.
- Multi-stage approaches: Five of twelve teams used multi-stage pipelines that first generated high-recall candidates and then classified extracted patches.Candidate generation used image features or object-detection networks, followed by ResNet or EfficientNet refinement.
- Domain robustness: Participants used standard augmentation, stain normalization, GAN-based synthesis, or domain-adversarial training to improve robustness to scanner shifts.GANs simulated scanner and style variation, while domain-adversarial methods encouraged domain-independent feature extraction.
- Segmentation supervision: Three teams enhanced approximate bounding-box labels with pixel-level instance masks for segmentation-based training.Mask generation used interactive segmentation, Hover-Net filtering, or approximately 100 manually annotated mitoses followed by Mask R-CNN inference.
- Data augmentation: Yang et al. combined standard transformations with Fourier-domain adaptation that transferred low-frequency stain information while preserving high-frequency structural content.The method acted as stain normalization without relying on a specific stain-matrix transformation.
- Ensembling: Five teams used moderate ensembling, including cross-validation fusion, averaged predictions, detection thresholds, and weighted boxes fusion.Ensembling can improve robustness and performance but increases computation time and carbon footprint when multiple large models run in parallel.
4. Results
Most approaches exceeded the CNN baseline, while Yang et al. achieved the strongest and most consistent scanner-wide performance. Errors varied by scanner and object difficulty, and the best methods commonly combined segmentation-related supervision with ensembling or test-time augmentation.
- Overall performance: F1 0.748 was the best overall score, achieved by Yang et al. with segmentation and Fourier-domain mixing.The approach was consistently strong across scanners and performed well on Scanner D, where many other methods struggled.
- Overall performance: The majority of approaches outperformed the CNN baseline, while the domain-adversarial reference method achieved F1 0.718.The reference method was outperformed by only four approaches on the test set.
- Post-challenge ensembling: The top-five ensemble considerably exceeded the leading individual approaches in overall F1, mainly through a Scanner D improvement.It was approximately on par with the best scanner-specific approach elsewhere and achieved the highest precision on all scanners.
- Object-level agreement: Most unique false detections were identified by only a small number of approaches, with little variation across scanners.This pattern was assessed using a histogram of false positives across scanners.
- Object-level agreement: Low contrast on Scanner D, atypical mitotic appearances, and unclear cell boundaries were difficult for many approaches.Examples included atypical or late-telophase cells and cells with unclear boundaries.
- Methodological factors: The three best-performing approaches included auxiliary mitosis segmentation, while five of the seven top approaches used ensembling or test-time augmentation.The authors could not determine whether segmentation specifically improved domain generalization.
5. Discussion
MIDOG results suggest scanner-induced domain shift can be substantially compensated, but performance and clinical relevance remain bounded by selected regions, scanner variability, and unevaluated clinical settings.
- Proper augmentation strategies and deep learning architectures compensated for scanner-induced domain shift to a high degree.
- The best approaches performed in the range of well-performing human experts on the same mitosis-detection task.
- The evaluation was limited to selected regions of interest, whereas whole-slide images contain greater tissue-quality variability and harder negative examples.
- Generalization to whole-slide images, additional tissue and cancer types, and clinical outcomes remains outside the challenge scope.
- The best results varied by scanner, and weaker performance on Scanners D and F may reflect incompletely covered domain shift, image quality, or unfamiliar visual patterns.
- Although many high-ranking approaches used ensembling or test-time augmentation, their causal contribution to performance remains unclear.
- The top5 ensemble achieved superior performance, supporting ensembling as a likely success factor for this task.
Data usage statement
The training data were released under a Creative Commons license permitting attribution and noncommercial, non-derivative use.
- The training set was released under the Creative Commons 4.0 BY-NC-ND license.
Appendix A. Author contributions
The appendix lists the challenge organizers, reference-approach contributors, writing group, expert pathologists, participating teams, and manuscript reviewers.
- Katharina Breininger, Natalie ter Hoeve, Christof A. Bertram, Francesco Ciompi, Robert Klopfleisch, Andreas Maier, Nikolas Stathonikos, Mitko Veta, and Marc Aubreville organized the challenge.
- Frauke Wilm, Christian Marzahl, Katharina Breining, and Marc Aubreville provided the algorithmic reference approach.
- Marc Aubreville, Nikolas Stathonikos, Christof A. Bertram, Katharina Breininger, and Mitko Veta formed the core writing group.
- Taryn A. Donovan, Robert Klopfleisch, and Christof A. Bertram served as expert pathologists for annotating the complete challenge dataset.
- The appendix acknowledges the participating teams, including Team Leeds, Team CGV, Team PixelPath-AI, Team Tribun Healthcare, Team jdex, Team SK, Team ML, Team XidianUOUC, and Team No. 0.
- All authors reviewed the manuscript.