Source-linked AI summary
Assessment of algorithms for mitosis detection in breast cancer histopathology images
Mitko Veta, Paul J. van Diest, Stefan M. Willems, Haibo Wang, Anant Madabhushi, Angel Cruz-Roa, Fabio Gonzalez, Anders B. L. Larsen, Jacob S. Vestergaard, Anders B. Dahl, Dan C. Cireşan, Jürgen Schmidhuber, Alessandro Giusti, Luca M. Gambardella, F. Boray Tek, Thomas Walter, Ching-Wei Wang, Satoshi Kondo, Bogdan J. Matuszewski, Frederic Precioso, Violet Snell, Josef Kittler, Teofilo E. de Campos, Adnan M. Khan, Nasir M. Rajpoot, Evdokia Arkoumani, Miangela M. Lacle, Max A. Viergever, Josien P. W. Pluim
TL;DR
Manual mitosis counting is an important but subjective and laborious way to assess breast-tumor proliferation. The paper evaluates automatic detection methods through the AMIDA13 challenge using shared whole-slide-image data. The best-performing method achieved accuracy on the order of inter-observer variability.
Problem
Mitosis counting is an important prognostic assessment of breast-tumor proliferation, but it is laborious, subjective, and affected by observer variation.
Method
The AMIDA13 challenge evaluated automatic mitosis detection methods on regions extracted from whole slide images using shared training and testing data.
Results
The best-performing method achieved accuracy on the order of inter-observer variability among pathologists.
Takeaways & Limitations
The challenge establishes a comparative record of mitosis-detection methods and is intended to track the field through ongoing data and submissions.
Takeaways & Limitations
Whole slide images captured with a single focal plane lack fine focusing, which is needed to distinguish mitoses from similar-looking objects.
Abstract
from arXiv · showhide
The proliferative activity of breast tumors, which is routinely estimated by counting of mitotic figures in hematoxylin and eosin stained histology sections, is considered to be one of the most important prognostic markers. However, mitosis counting is laborious, subjective and may suffer from low inter-observer agreement. With the wider acceptance of whole slide images in pathology labs, automatic image analysis has been proposed as a potential solution for these issues. In this paper, the results from the Assessment of Mitosis Detection Algorithms 2013 (AMIDA13) challenge are described. The challenge was based on a data set consisting of 12 training and 11 testing subjects, with more than one thousand annotated mitotic figures by multiple observers. Short descriptions and results from the evaluation of eleven methods are presented. The top performing method has an error rate that is comparable to the inter-observer agreement among pathologists.
1 Introduction
Mitotic figures are a widely used marker of breast-tumor proliferation, but manual counting is subjective and laborious. AMIDA13 evaluates automatic detection methods on shared whole-slide-image data to address these challenges.
- Prognostic factors related to tumor proliferation are among the most powerful factors for assessing breast-cancer patient risk.
- Mitoses are visually identifiable in H&E sections by characteristic hyperchromatic morphology and remain the most widely used assessment of tumor proliferation.
- The mitotic activity index counts mitoses in a 2 mm^2 area, corresponding to 8–10 microscope high-power fields, but assessment is subjective and vulnerable to observer variation.
- Typical manual mitosis counting takes 5–10 minutes and may need repetition across areas or slides in borderline cases.
- Whole slide imaging enables image-analysis methods that could reduce the subjectivity and tediousness of manual mitosis counting.
- AMIDA13 compares (semi-)automatic mitosis detection methods on regions extracted from a common whole-slide-image data set.
2 Materials
The AMIDA13 dataset was assembled from routinely prepared breast-cancer slides, manually selected tumor regions, and consensus annotations by multiple pathologists. Images were distributed as high-power-field tiles, with training and independent testing subsets, while observer agreement was quantified using Dice overlap.
- Dataset construction: The dataset comprised slides from 23 consecutive invasive breast carcinoma patients selected from a pathology department archive.The slides were produced at the University Medical Center Utrecht.
- Dataset construction: The challenge manually selected slides and regions rather than automating these selection steps, focusing evaluation on mitosis detection.The authors identify automated slide and region selection as necessary for a fully automatic workflow.
- Dataset construction: An expert pathologist selected one representative slide and a tumor region per patient, emphasizing invasive, peripheral, highly cellular areas in larger tumors.Outlined regions ranged from 7 mm2 to 58 mm2, with a median of 26 mm2.
- Ground truth annotation: Ground truth combined two independent pathologists’ annotations, direct agreement, and panel review of discordant objects using a 7.5 μm matching criterion.Objects accepted as ground truth had agreement from at least two experts.
- Ground truth annotation: The initial observers produced 1088 and 1599 annotations, while consensus annotation retained 1157 objects.A panel resolved 1389 discordant annotations after 649 locations were agreed upon directly.
- Image data distribution: Whole-slide images were divided into 2000×2000-pixel TIFF tiles representing approximately 0.25 mm2, or roughly one microscope high-power field.Tiles outside the marked region or intersecting marker lines were excluded; selected non-empty tiles and additional empty tiles formed the dataset.
- Observer agreement: Observer Dice overlap was 0.527 for training HPFs and 0.566 for testing HPFs, compared with 0.483 before HPF selection.The higher selected-HPF values reflect selection of non-empty HPFs based on consensus annotation.
- Observer agreement: Individual-to-consensus Dice overlaps ranged from 0.749 to 0.796 across observers and training or testing HPFs.The authors caution that this measure is biased because consensus partly incorporates the individual annotations.
3 Methods
The described method combines color-based candidate detection with two complementary classification strategies: learned CNN features and handcrafted features classified by random forests. Their outputs are fused into a final prediction score.
- Preprocessing: Seven image channels were used, combining RGB channels with LAB, LUV, and blue-ratio representations for segmentation and feature extraction.The additional channels were L from LAB, V and L from LUV, and the blue ratio image.
- Candidate detection: Candidate regions were generated for each HPF by thresholding the blue ratio image.The blue ratio divides the blue channel by the sum of the other two RGB channels.
- Classification: The method fused CNN-based feature learning with handcrafted features classified by random forests, processing both representations independently for each candidate.The CNN and random-forest classifiers generated separate outputs for fusion.
- Classification: CNN processing used 80×80-pixel YUV patches centered on candidates and a four-layer architecture trained with stochastic gradient descent.The architecture included convolution-pooling layers, a fully connected layer, and a softmax layer.
- Classification: Handcrafted morphology, statistics, and texture features from seven channels were reduced by PCA retaining 98.5% of principal components, then classified with a two-stage random-forest cascade.Each cascade used 100 random trees, with non-mitosis nuclei reduced to balance classes.
- Classification: The final prediction score was computed as a weighted average of the CNN and random-forest classifier outputs.This combines the two independently classified feature representations.
- Candidate detection: A second candidate-detection variant applied a Laplacian-of-Gaussian blob detector to the blue ratio image before connected-component labeling.This procedure was also based on thresholding the detector response.
3.3 IDSIA5
The IDSIA method uses multi-column max-pooling convolutional neural networks for supervised pixel classification of mitosis locations.
- 3.3 IDSIA: MCMPCNNs classified 63×63-pixel raw RGB patches by predicting whether the central pixel lay within 20 pixels of a mitosis centroid.Three identical 10-layer networks were trained on 20 million patches, including randomly sampled mitosis, non-mitosis, and difficult non-mitosis examples.
3.4 ISIK6
The ISIK method extends earlier mitosis-detection work with contrast enhancement, morphological candidate segmentation, staged feature-based filtering, and three-class SVM classification.
- Preprocessing: The method first performed image contrast stretching before candidate detection.
- Candidate detection and segmentation: Candidate objects were segmented using a morphological double threshold, area opening with a 50-pixel minimum, and hole filling.The area opening removed isolated spurious regions.
- Feature extraction and classification: A first classification stage used simple geometric and intensity features to reduce false objects while minimizing loss of true mitotic figures.Features included area, axis lengths, perimeter, equivalent diameter, eccentricity, extent, and color intensities.
- Feature extraction and classification: A second stage extracted multiscale color, shape, Laplacian, morphological, and texture features from windows around surviving candidates.The feature vector combined five feature groups across windows of increasing width.
- Preprocessing and segmentation: Hematoxylin and eosin channels were separated by color deconvolution, and candidate segmentation was performed entirely on the hematoxylin channel.A diameter-closing operation used an 80-pixel diameter to detect potential nuclei.
- Classification: The classifier used three classes—non-mitosis, early mitosis, and late mitosis—because the two mitosis classes were morphologically different.An RBF-kernel SVM was trained to distinguish the three classes.
3.6 NTUST8
NTUST8 combines multiscale color preprocessing, candidate segmentation, morphological and texture features, and random-forest classification for mitosis detection.
- A ten-layer cwBoost cascade uses diverse mitosis detectors to filter negative areas during supervised pixel classification.
- Preprocessing: RGB images were transformed into L*a*b, HSV, blue-ratio, red-ratio, and blue-red-ratio color spaces for later processing.
- Candidate detection and segmentation: Candidates were obtained by thresholding the blue-ratio image at three standard deviations and removing regions smaller than 80 pixels.
- Feature extraction and classification: Each candidate used morphological measurements plus local-binary-pattern histograms and Haralick features across multiple color channels.
- Feature extraction and classification: A random forest classified candidate objects as mitoses or non-mitoses.
3.8 POLYTECH/UCLAN10
POLYTECH/UCLAN10 applies color correction and threshold-based candidate extraction, then classifies candidates using directional texture and shape features.
- Preprocessing: Blue-corrected images were intensity-adjusted to compensate partly for differences in tissue appearance.
- Candidate detection and segmentation: Thresholding and binary morphological operations detected candidate objects, followed by extraction of 128×128-pixel patches around their centroids.
- Feature extraction and classification: The classifier used 10 run-length-matrix features and eight co-occurrence-matrix features averaged across four directions.
- Preprocessing: Histogram matching aligned each color channel to the mean histogram of the whole training data set, excluding white non-tissue regions.
- Candidate detection and segmentation: Color-based likelihood lookup tables identified candidate locations from quantized RGB values and neighborhoods around ground-truth locations.
- Feature extraction and classification: Candidate patches were converted to grayscale, segmented objects were described with rotation-invariant features, and an RBF SVM performed classification.
3.10 SHEFFIELD/SURREY12
SHEFFIELD/SURREY12 combines stain normalization and statistical candidate extraction with object, contextual, and imbalance-aware classification features; an alternative models mitosis patches in a low-dimensional manifold.
- The method models mitosis images with a low-dimensional manifold, enabling visualization of their range of appearances.
- BGPLVM represents normalized grayscale candidate patches as vectors and learns a nonlinear mapping between observations and a low-dimensional latent space.
- Preprocessing: Staining normalization used nonlinear color mapping to neutralize variation in staining color.
- Candidate detection and segmentation: Candidate objects were extracted by modeling Lab b-channel intensities, using a Gamma distribution for mitoses and a Gaussian distribution for non-mitoses.
- Feature extraction and classification: Object-level color, shape, and texture features were combined with contextual features from a surrounding window for classification.
- Feature extraction and classification: RUSBoost addressed the disproportionate number of non-mitosis candidates through under-sampling and boosting.
4 Evaluation
Evaluation used distance-based detection matching and F1-based accuracy measures, supplemented by subject-level, precision-recall, observer-review, and density-correlation analyses.
- A detection counted as a true positive when its Euclidean distance from a ground-truth location was less than 7.5 μm, or 30 pixels.
- The primary overall F1-score pooled all ground-truth objects across patients and combined precision with recall through their harmonic mean.
- Subject-level F1-scores complemented the pooled measure, while precision-recall curves were computed for submissions providing probability estimates.
- Visual inspection found that many false positives from top-performing methods closely resembled mitotic figures.
- F1-score is equivalent to the Dice coefficient, allowing direct comparison with inter-observer agreement.
- False positives from the two leading methods were re-evaluated by two observers alongside ground-truth mitoses as a control.
- Detection performance for mitotic density was assessed by correlating detections and ground-truth mitoses per high-power field across testing subjects.
5 Results
IDSIA ranked highest on both overall and average F1-score, while re-annotation showed that many apparent false positives from the top methods resembled true mitoses.
- Overall detection performance: 0.611 overall F1-score was achieved by IDSIA, the top-ranking method.
- Per-subject performance: 0.445 average F1-score again ranked IDSIA first across subjects.Subject #8 had no annotated mitoses, so its F1-score was undefined and excluded from the average.
- Overall detection performance: Precision-recall curves compared IDSIA and DTU with the remaining methods.The paper reports that IDSIA and DTU performed notably better than the other methods, not merely at a different operating point.
- False-positive re-annotation: 29% of IDSIA false positives and 16% of DTU false positives were re-annotated as mitotic figures.These proportions corresponded to 61 of 208 IDSIA detections and 62 of 397 DTU detections.
- False-positive re-annotation: 61% of IDSIA detections were re-annotated as mitotic figures, compared with 42% for DTU and 71% for the ground truth dataset.The proportions were computed over the entire detection sets: 326/534 for IDSIA, 293/693 for DTU, and 379/533 for ground truth.
- Mitoses per HPF: IDSIA achieved the best correlation between estimated and ground-truth mitoses per HPF, with r = 0.90, 95% CI [0.62, 0.96].The scatter plots covered the five methods with highest overall F1-score.
6 Discussion
The challenge showed that IDSIA and DTU substantially outperformed the other methods, while staining variability, difficult cases, annotation variability, and computational cost remained important practical issues.
- Performance of the proposed methods: IDSIA achieved the best performance across all evaluation measures, with an overall F1-score comparable to inter-observer agreement among pathologists.DTU also performed solidly, and both methods were notably better than the remaining methods.
- Performance of the proposed methods: Many missed mitoses had less intensive staining, whereas lower-performing methods often produced false positives on dark non-mitotic objects lacking characteristic edge protrusions.The authors attribute these errors either to insufficiently discriminative texture and shape features or to underrepresentation of lightly stained mitoses in training data.
- Performance of the proposed methods: Re-annotation indicated that a large portion of IDSIA's false positives were true mitotic figures missed in the original ground truth because of task intricacy and observer variability.The label distributions for the original ground truth and IDSIA detections were also very similar.
- Performance of the proposed methods: Lower object-level F1-scores did not preclude good correlation between ground-truth and estimated mitoses per HPF, suggesting that MAI or mitotic activity grade estimation may be possible from lower-resolution detection or global image features.The challenge dataset removed empty HPFs, so mitoses per HPF did not correspond directly to the MAI of the outlined regions.
- Performance of the proposed methods: Combining methods by majority voting or intersecting better-performing methods did not improve results over the best individual method.Workshop discussion identified staining variation as a major obstacle, while preliminary staining-normalization experiments showed promising results but were not included.
- Feasibility of mitosis counting on whole slide images: The challenge did not evaluate running time, although IDSIA required approximately 9 days of total training and under 13 minutes to test all 24 variations for one HPF.DTU training took 1.5 hours and testing one HPF took approximately 30 seconds; thousands of HPFs per slide make further speed improvements necessary.
- Performance on individual cases: Performance was worst for low-activity cases with atypical, weakly stained mitoses and for cases with many dark non-mitotic nuclei, while it was best when most hyperchromatic objects were mitoses.These case differences expose the limitations of relying heavily on staining intensity.
- Feasibility of mitosis counting on whole slide images: Whole slide images lack fine focusing in single-focal-plane scans, potentially causing discrepancies between mitotic activity estimated by light microscopy and from unifocal digital images.Multifocal scanning is increasingly available but remains limited by increased scanning time and storage requirements.
7 Conclusions
The paper summarizes the AMIDA13 breast-cancer mitosis-detection challenge and its evaluation of 11 methods. The challenge established performance near inter-observer variability and was intended to continue through expanded datasets and submissions.
- Conclusions: AMIDA13 included 12 training and 11 testing subjects, each with more than 500 annotated mitotic figures by multiple observers, and described 11 of 14 submitted methods.The paper summarizes the proposed methods and results from the challenge workshop.
- Conclusions: The best-performing method achieved accuracy in the order of inter-observer variability.The challenge was intended to remain open for incremental extensions of its training and testing datasets.
- Conclusions: The authors intended to keep the challenge website open for new submissions while expanding the training and testing datasets.This was meant to maintain a record of the state of the art in breast-cancer histopathology mitosis detection.