Source-linked AI summary
Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge
Mitko Veta, Yujing J. Heng, Nikolas Stathonikos, Babak Ehteshami Bejnordi, Francisco Beca, Thomas Wollmann, Karl Rohr, Manan A. Shah, Dayong Wang, Mikael Rousson, Martin Hedlund, David Tellez, Francesco Ciompi, Erwan Zerhouni, David Lanyi, Matheus Viana, Vassili Kovalev, Vitali Liauchuk, Hady Ahmady Phoulady, Talha Qaiser, Simon Graham, Nasir Rajpoot, Erik Sjöblom, Jesper Molin, Kyunghyun Paeng, Sangheum Hwang, Sunggyun Park, Zhipeng Jia, Eric I-Chao Chang, Yan Xu, Andrew H. Beck, Paul J. van Diest, Josien P. W. Pluim
TL;DR
Tumor proliferation is an important prognostic biomarker, motivating assessment from whole-slide images. TUPAC16 presented an overview of submitted methods, with moderate agreement for manual ground truth and support for the molecular PAM50 proliferation-score hypothesis, although the results were not sufficient for second-opinion use.
Problem
Tumor proliferation is an important prognostic biomarker for breast cancer, motivating assessment from whole-slide images.
Method
TUPAC16 presented an overview of submitted methods for predicting tumor proliferation from whole-slide images.
Results
κ = 0.567 indicated moderate agreement with the manual ground truth, while the second task accepted the hypothesis that the molecular PAM50 proliferation score could be predicted.
Takeaways & Limitations
TUPAC16 was the first challenge to predict tumor proliferation scores from whole-slide images, providing valuable insight into the problem.
Takeaways & Limitations
The results did not reach a level sufficient for use as a “second opinion” score, and further research is needed.
Abstract
from arXiv · showhide
Tumor proliferation is an important biomarker indicative of the prognosis of breast cancer patients. Assessment of tumor proliferation in a clinical setting is highly subjective and labor-intensive task. Previous efforts to automate tumor proliferation assessment by image analysis only focused on mitosis detection in predefined tumor regions. However, in a real-world scenario, automatic mitosis detection should be performed in whole-slide images (WSIs) and an automatic method should be able to produce a tumor proliferation score given a WSI as input. To address this, we organized the TUmor Proliferation Assessment Challenge 2016 (TUPAC16) on prediction of tumor proliferation scores from WSIs. The challenge dataset consisted of 500 training and 321 testing breast cancer histopathology WSIs. In order to ensure fair and independent evaluation, only the ground truth for the training dataset was provided to the challenge participants. The first task of the challenge was to predict mitotic scores, i.e., to reproduce the manual method of assessing tumor proliferation by a pathologist. The second task was to predict the gene expression based PAM50 proliferation scores from the WSI. The best performing automatic method for the first task achieved a quadratic-weighted Cohen's kappa score of $κ$ = 0.567, 95% CI [0.464, 0.671] between the predicted scores and the ground truth. For the second task, the predictions of the top method had a Spearman's correlation coefficient of r = 0.617, 95% CI [0.581 0.651] with the ground truth. This was the first study that investigated tumor proliferation assessment from WSIs. The achieved results are promising given the difficulty of the tasks and weakly-labelled nature of the ground truth. However, further research is needed to improve the practical utility of image analysis methods for this task.
1. Introduction
Tumor proliferation is clinically important but its assessment is subjective, labor-intensive, and affected by reproducibility and standardization problems. TUPAC16 addressed these limitations by evaluating methods that assess proliferation directly from whole-slide images.
- Tumor proliferation is an important biomarker indicative of breast cancer prognosis.
- Pathologists commonly assess proliferation by counting mitotic figures on H&E slides and assigning mitotic scores.
- Clinical translation of Ki67 and PAM50 proliferation scores is impeded by nonstandardized procedures, interpretation issues, and complex gene-expression workflows.
- Automated image analysis could improve the efficiency, accuracy, and reliability of mitotic-figure detection, which is subjective, labor-intensive, and prone to reproducibility problems.
- Previous mitosis-detection challenges focused on predetermined tumor regions rather than whole-slide images and tumor proliferation scoring.
- TUPAC16 evaluated semi-automatic methods for assessing tumor proliferation from whole-slide images and presented an overview of submitted methods and results.
2. Materials and Methods
The challenge used TCGA breast cancer WSIs with manual mitotic scores and PAM50 proliferation scores, split into training and testing sets. Auxiliary ROI and mitosis datasets supported development of WSI-based proliferation methods, which generally used preprocessing, ROI detection, and deep convolutional networks.
- Main dataset: Mitotic scores were based on summed mitosis counts across 10 consecutive high-powered fields selected from areas of highest mitotic activity.Fields with scant tumor cells or necrosis were excluded; scores 1, 2, and 3 represented 0–5, 6–10, and >10 mitoses per 10 fields.
- Main dataset: Pathologist agreement for mitotic scoring was 78%, with Krippendorff’s alpha of 0.488 among 311 cases scored by multiple pathologists.Consensus scores used the most common score, with the highest score selected in ties.
- Main dataset: The PAM50 proliferation score was the average expression of 11 proliferation-associated genes in the PAM50 signature.The genes included BIRC5, CCNB1, CDC20, CEP55, MKI67, NDC80, NUF2, PTTG1, RRM2, TYMS, and UBE2C.
- Main dataset: The main dataset comprised 821 TCGA breast cancer WSIs, randomly split into 500 training and 321 testing cases, with mitotic and PAM50 proliferation labels.Mitotic scores came from manual pathologist counting, while PAM50 scores were derived from gene expression.
- Auxiliary datasets: The auxiliary ROI dataset contained 148 training cases with three pathology-resident-annotated regions per case, while the mitosis dataset contained 73 cases with consensus mitotic annotations.The auxiliary datasets were provided to facilitate method development.
- Submitted methods: Submitted methods generally addressed tissue-appearance variability through staining normalization and background removal, then used ROI detection followed by mitosis-based or appearance-based prediction.Most teams used deep convolutional neural networks; approaches differed in whether mitosis detection was explicitly performed.
RADBOUD
The RADBOUD submission used image-derived features and classifiers for the challenge tasks, with reported ranks differing between task 1 and task 2.
- Method: The submission used staining normalization and a random forest classifier based on the number of detected mitoses in selected ROIs.The listed preprocessing included Macenko staining normalization.
- Results: RADBOUD achieved rank 3 for task 2 and rank 9 for task 1.The task 2 result is listed for the same method configuration used for task 1.
- Method: RADBOUD’s task 2 approach used a RankSVM with a linear kernel and ResNet features combined with Pnorm pooling.The method used manually selected ROIs and did not participate in some listed components.
3. Results
Across the two challenge tasks, automatic methods achieved moderate agreement with manual mitotic scores and PAM50 scores. Ensembling the top automatic methods improved performance over the best individual automatic submission.
- Task 1: κ = 0.567 was achieved by LUNIT for task 1, measured with quadratic-weighted Cohen’s kappa against the ground-truth mitotic scores.The 95% CI was [0.464, 0.671].
- Task 1: Mitotic score 2 was the most commonly misclassified, with per-class accuracies of 17%, 50%, and 31% for LUNIT, MICROSOFT, and CONTEXTVISION.The values correspond to the three named methods in that order.
- Task 2: The semi-automatic MICROSOFT method achieved r = 0.710 for task 2, with 95% CI [0.681, 0.737].This was the best reported task 2 performance, while LUNIT was the best fully automatic method.
- Task 2: r = 0.617 was the best score among automatic methods for task 2, achieved by LUNIT using Spearman correlation with PAM50 ground truth.RADBOUD followed with r = 0.516; the LUNIT 95% CI was [0.581, 0.651].
- Method ensembling: Averaging the top three automatic task 2 predictions produced r = 0.682, improving by 0.065 over LUNIT’s r = 0.617.Predictions were scaled to zero mean and unit variance before averaging; the ensemble 95% CI was [0.651, 0.711].
- Method ensembling: Averaging the top three automatic task 1 methods produced κ = 0.613, improving by 0.046 over LUNIT’s κ = 0.567.The ensemble comprised LUNIT, CONTEXTVISION, and SECTRA, with 95% CI [0.504, 0.722].
4. Discussion
The discussion finds that TUPAC16 advanced whole-slide tumor-proliferation assessment, but performance and evaluation limitations still constrain clinical utility. It also identifies domain shift, modular submissions, and feature-based prediction as priorities for future work.
- Clinical motivation: Manual mitosis counting is subjective and time consuming, while reproducing it from global slide-level annotations is challenging because tissue assessment is multi-scale.The challenge addressed a clinically important assessment that traditionally requires manual analysis.
- Task 1 findings: κ = 0.567 represented moderate agreement for the best automatic mitosis-score method, below the κ = 0.792–0.893 inter-observer agreement reported for pathologists.The comparison is affected by differences in evaluation setup: prior pathologists counted mitoses in predefined areas, whereas challenge teams could use different ROIs.
- Task 1 findings: The top first-task methods made substantial errors, including score differences of two, and such discordance may lead to more severe clinical management of cancer patients.The authors note that similar errors can also occur with manual scoring, although to a lesser extent.
- Future evaluation: Auxiliary mitosis data from three Dutch medical centers differed from the main TCGA dataset from U.S. institutions, creating domain shift and increasing task difficulty.Future evaluations should include independent cases from pathology laboratories or scanners absent from training.
- Method diversity: Performance gains from ensembling likely reflected the heterogeneous designs of the top methods, which varied in preprocessing, ROI detection, mitosis detection, and slide-level scoring.The top methods used different neural-network depths, staining normalization, resolutions, ROI detectors, and slide-level classifiers or heuristics.
- Task 2 findings: The second task accepted the hypothesis that molecular PAM50 proliferation scores could be predicted from WSIs, with strong-performing methods using region-level features without intermediate mitosis detection.The best automatic first-task method still relied on mitosis counting, whereas several methods performed particularly well on the second task without it.
- Challenge contribution: TUPAC16 was the first challenge to predict tumor proliferation scores from WSIs and aimed to advance automatic assessment beyond mitosis-detection-focused challenges.The challenge evaluated automatic and semi-automatic methods for tumor proliferation assessment from whole-slide images.
- Evaluation and reproducibility: The challenge could not quantitatively isolate how individual pipeline components affected performance because submitted methods were heterogeneous.The authors recommend modular Docker submissions to improve reproducibility and enable recombination and evaluation of individual building blocks.