PaperlayerView exact revision on arXiv

Source-linked AI summary

Whole-Slide Mitosis Detection in H&E Breast Histology Using PHH3 as a Reference to Train Distilled Stain-Invariant Convolutional Networks

David Tellez, Maschenka Balkenhol, Irene Otte-Holler, Rob van de Loo, Rob Vogels, Peter Bult, Carla Wauters, Willem Vreuls, Suzanne Mol, Nico Karssemeijer, Geert Litjens, Jeroen van der Laak, Francesco Ciompi

arXiv:1808.05896v1cs.CV

TL;DR

Automated mitosis detection in breast cancer H&E whole-slide images is limited by noisy annotations, stain variation, and the computational cost of CNN ensembles. The paper combines PHH3-based reference generation, H&E-specific stain augmentation, ensembling, and knowledge distillation, achieving competitive multicenter performance while reducing inference requirements. Its reference standard still contains noise from PHH3 sensitivity and specificity limitations and CNN performance.

  • Problem

    Automated mitosis detection is hindered by limited and noisy reference standards, staining variation across laboratories, and the computational cost of high-capacity CNN ensembles for whole-slide images.

  • Method

    The method combines PHH3 restaining and registration for scalable H&E annotation, direct hematoxylin/eosin-channel stain augmentation, network ensembling, and knowledge distillation.

  • Results

    The distilled CNNs achieved F1-scores of 0.623, 0.628 and 0.636, respectively, with similar performance to the ensemble and fewer trainable parameters.

  • Takeaways & Limitations

    H&E-specific augmentation with ensembling transferred performance from single-center training data to multicenter slides, while distillation enabled efficient dense prediction on gigapixel images.

  • Takeaways & Limitations

    The PHH3-derived reference standard contains noise from limited antibody sensitivity, limited specificity, and the CNN used in its generation.

Abstract

from arXiv · show

Manual counting of mitotic tumor cells in tissue sections constitutes one of the strongest prognostic markers for breast cancer. This procedure, however, is time-consuming and error-prone. We developed a method to automatically detect mitotic figures in breast cancer tissue sections based on convolutional neural networks (CNNs). Application of CNNs to hematoxylin and eosin (H&E) stained histological tissue sections is hampered by: (1) noisy and expensive reference standards established by pathologists, (2) lack of generalization due to staining variation across laboratories, and (3) high computational requirements needed to process gigapixel whole-slide images (WSIs). In this paper, we present a method to train and evaluate CNNs to specifically solve these issues in the context of mitosis detection in breast cancer WSIs. First, by combining image analysis of mitotic activity in phosphohistone-H3 (PHH3) restained slides and registration, we built a reference standard for mitosis detection in entire H&E WSIs requiring minimal manual annotation effort. Second, we designed a data augmentation strategy that creates diverse and realistic H&E stain variations by modifying the hematoxylin and eosin color channels directly. Using it during training combined with network ensembling resulted in a stain invariant mitosis detector. Third, we applied knowledge distillation to reduce the computational requirements of the mitosis detection ensemble with a negligible loss of performance. The system was trained in a single-center cohort and evaluated in an independent multicenter cohort from The Cancer Genome Atlas on the three tasks of the Tumor Proliferation Assessment Challenge (TUPAC). We obtained a performance within the top-3 best methods for most of the tasks of the challenge.

I. INTRODUCTION

The paper addresses noisy reference standards, stain variation, and computational costs in automated mitosis detection for breast cancer H&E whole-slide images. It combines PHH3-based annotation, H&E-specific stain augmentation, ensembling, and knowledge distillation to improve robustness and efficiency.

  • Motivation: Public mitosis-detection datasets are limited in size, tissue representativity, and agreement between reference annotations.Existing datasets contain only about 1,500 annotated mitotic figures, often from selected tumor regions, while manual annotations can have disagreement and limited recall.
  • Computational efficiency: High-capacity CNN ensembles improve detection but are computationally expensive for dense prediction on gigapixel whole-slide images.The paper therefore proposes knowledge distillation to transfer ensemble performance to a single smaller network.
  • Reference standards: PHH3 restaining and registration provide a scalable way to exhaustively annotate mitotic figures in H&E whole-slide images with minimal human labeling.PHH3 highlights mitotic cells as high-contrast objects that are easier to detect than in H&E.
  • Stain invariance: Stain augmentation modifies hematoxylin and eosin channels directly to generate broad, realistic H&E stain variations from single-center images.This approach targets staining differences across laboratories without requiring a multicenter training cohort or preprocessing all slides.
  • Generalization: The method transfers performance from single-center training data to a multicenter cohort by enforcing stain invariance through augmentation and ensembling.The training cohort has homogeneous staining, whereas the validation cohort contains variable staining across centers.
  • Computational efficiency: Knowledge distillation reduces the ensemble to a single network while aiming to preserve similar performance and substantially increase processing speed.The distilled model is intended for mitosis detection in gigapixel whole-slide images.

II. MATERIALS

The study uses separate cohorts for algorithm development and independent evaluation, including paired TNBC H&E–PHH3 slides and multicenter TUPAC data.

  • Cohorts: Two cohorts support algorithm development and independent system evaluation.The datasets are described in Table I.
  • TNBC cohort: The TNBC cohort contains 18 patients from three Dutch hospitals, with tissue sections processed at Radboudumc.Each case has paired H&E and PHH3 whole-slide images representing the same tissue section.
  • TUPAC cohort: The TUPAC cohort contains 814 multicenter H&E whole-slide images from The Cancer Genome Atlas.The cases include mitotic-count grading and a molecular proliferation score; 493 cases have public reference standards and 321 are reserved for organizers.
  • Auxiliary annotations: TUPAC auxiliary data comprise 84 proliferative tumor regions exhaustively annotated by two observers, with disagreements reviewed by a third pathologist.Fifty cases have public reference standards and 34 are available only to TUPAC organizers.
  • Preprocessing: All whole-slide images were tissue-background segmented to exclude non-tissue areas from analysis.

III. PHH3 STAIN: REFERENCE STANDARD FOR MITOTIC ACTIVITY

PHH3 slides are analyzed automatically to generate exhaustive mitosis reference standards for paired H&E slides, reducing the need for manual annotation.

  • Reference-standard pipeline: A CNN identifies mitotic figures throughout PHH3 whole-slide images, and detections are registered to paired H&E slides.This process builds the reference standard used for H&E mitosis detection.
  • Training stage: PHH3 candidates are extracted from the brown DAB channel, artifact-pruned, manually annotated in a subset, and used to train CNN2.CNN2 distinguishes mitotic from non-mitotic patches.
  • Inference stage: F1-score 0.9 was obtained by CNN2 when classifying PHH3 candidates as mitotic or non-mitotic.The trained network classified all PHH3 candidates to generate exhaustive whole-slide reference data.

B. Registering mitotic figures from PHH3 to H&E slides

The H&E detector uses candidate mining, augmentation, ensemble training, and distillation to support exhaustive whole-slide mitosis detection.

  • Registering mitotic figures from PHH3 to H&E slides: Paired PHH3 and H&E slides are registered so PHH3-derived mitotic detections can provide H&E training labels.The exact same tissue section appears in each pair, simplifying alignment.
  • Whole-slide detection: The detector is trained on TNBC-H&E slides and later applied exhaustively throughout H&E whole-slide images.CNN5 performs fully convolutional inference across each whole-slide image.
  • Whole-slide detection: Stain standardization was applied despite limited training-cohort variation, making generalization to unseen stains more challenging.
  • Candidate generation: Candidate detection assumes mitotic figures are non-overlapping circular objects with dark inner cores.Candidates are found iteratively around local RGB-intensity minima until pixels above threshold t are exhausted.
  • Training dataset construction: Hard-negative mining uses CNN3 predictions to sample informative negatives, producing bootstrapped datasets for CNN4 ensemble training.Most uniformly sampled negatives were easy to classify and contributed little to decision-boundary improvement.

B. H&E stain augmentation

The H&E augmentation strategy varies morphology, stain channels, and scanner-like artifacts to improve detector robustness to image variation.

  • Augmentation framework: CNN4 training applies on-the-fly augmentation to reduce overfitting and improve generalization.The augmentation routines are illustrated using multiple versions of the same mitotic patch.
  • Morphology invariance: Morphology augmentation varies mitotic patch geometry through rotation, mirroring, scaling, elastic deformation, and translation.
  • H&E stain augmentation: Stain augmentation decomposes RGB patches into hematoxylin, eosin, and residual channels, independently modifies each channel, and reconstructs RGB images.Random factors α_i and biases β_i control channel modifications; brightness, contrast, and color intensity are also varied.
  • Artifact invariance: Artifact augmentation simulates scanner blur with Gaussian filtering and compression artifacts with Gaussian noise.

C. Ensemble & network distillation

The method uses an ensemble of independently trained CNNs to improve mitosis detection, then distills their averaged probabilities into a smaller CNN for efficient whole-slide inference.

  • Ensemble: The ensemble trains k CNNs on different bootstrapped datasets and averages their predicted probabilities.Negative candidates are sampled with replacement to increase model independence.
  • Network distillation: Knowledge distillation transfers the ensemble’s performance to a smaller CNN by training on continuous ensemble probabilities rather than dataset labels.The distilled model is named CNN5, and γ controls its trainable capacity.
  • Whole-slide inference: Whole-slide detection applies CNN5 densely at 0.25 µm/pixel to produce mitosis probability maps.Post-processing thresholds, labels connected components, and suppresses detections within a specified distance.
  • Whole-slide inference: A 2 mm2 virtual hotspot scans each WSI, and the 95th percentile of non-empty mitotic counts identifies high mitotic activity while limiting outlier influence.The hotspot counts mitoses at each unique spatial position.
  • Whole-slide inference: Tumor grades are assigned using two thresholds on hotspot mitotic counts, producing grades 1, 2, or 3.A continuous tumor proliferation score is also derived from the hotspot count.

A. Mitosis detection in PHH3

The PHH3-based detection pipeline supplies training data for H&E mitosis detection by combining specialized CNNs with candidate selection and extensive augmentation.

  • PHH3-based detection: CNN1 classifies artifactual samples, while CNN2 classifies mitotic and non-mitotic patches from TNBC-PHH3 slides.Training and validation splits were performed at case level.
  • PHH3-based detection: The candidate detector uses d = 100 pixels and t = 0.6, recalling 99% of validation-reference mitoses at a 1:1000 positive-to-negative sampling rate.The H&E dataset was split into 11 training, 3 validation, and 4 test slides.
  • Experimental setup: Experiments evaluate CNNs trained on TNBC-H&E and tested independently on TUPAC-aux-train using highest F1-score across detection thresholds.The analysis measures data augmentation, ensembling, and distillation effects.
  • Candidate refinement: CNN3 is trained on all positive candidates and uniformly sampled negatives, after which difficult negatives are selected for CNN4 training.This procedure progressively focuses training on candidates likely to be confused with mitoses.
  • Data augmentation: Augmentation samples color, scaling, elastic deformation, intensity, and blur variations from specified parameter ranges.The color routine directly varies stain-related channels through α and β.

VI. EXPERIMENTS AND RESULTS

Experiments show progressively higher independent-test F1-scores with color and broader augmentation, ensembling, and distilled models, while Table IV frames challenge evaluation across three tasks.

  • Data augmentation: 0.018 F1-score was obtained with basic augmentation, compared with 0.412 after adding color augmentation and 0.613 using all listed augmentations.These experiments tested the proposed augmentation strategy on the independent TUPAC-aux-train dataset.
  • Challenge evaluation: Table IV reports independent evaluation of the proposed method on the three TUPAC Challenge tasks against the best leaderboard solutions.Top-1, Top-2, and Top-3 denote the best performing public-leaderboard solutions.
  • Distillation: 0.623, 0.628, and 0.636 F1-scores were obtained by distilled CNNs with γ values of 1.0, 0.8, and 0.6, respectively.These models performed similarly to the ensemble with drastically fewer trainable parameters.

B. Comparison with the state of the art

The system was evaluated independently on TUPAC challenge data and achieved performance comparable to leading methods across tumor grading, proliferation scoring, and mitotic figure detection.

  • Evaluation setting: The authors lacked access to TUPAC-test and TUPAC-aux-test ground truth, so organizers independently evaluated the predictions.The independent evaluation was intended to ensure fair comparison with state-of-the-art teams.
  • Challenge comparison: 0.471 Kappa agreement was obtained for tumor grading on TUPAC-test, comparable to the leaderboard’s top-3 entry.The 95% confidence interval was [0.340, 0.603].
  • Challenge comparison: 0.519 Spearman correlation was obtained for the genetic-based proliferation score on TUPAC-test, comparable to the top-2 entry.The 95% confidence interval was [0.477, 0.559].
  • Challenge comparison: 0.480 F1-score was obtained for individual mitotic figure detection on TUPAC-aux-test, with precision 0.467 and recall 0.494.This performance was comparable to the top-7 leaderboard entry.
  • Evaluation setting: The lower auxiliary-test F1-score than auxiliary-train F1-score prompted investigation of interobserver variability and limited test-sample size.A blinded resident-pathologist review was used to examine the detections.

VII. DISCUSSION AND CONCLUSION

The method addresses noisy references, stain variation, and inference cost in whole-slide mitosis detection through PHH3-based annotation, stain augmentation, ensembling, and knowledge distillation. It achieved competitive performance while enabling efficient gigapixel processing, although PHH3-derived references and evaluation annotations contain noise.

  • Reference standards: PHH3 restaining, automatic image analysis, and registration exhaustively annotated mitotic figures across entire H&E WSIs with only 2 hours of manual annotations per observer.The resulting dataset was at least an order of magnitude larger than the publicly available mitosis-detection dataset.
  • Stain invariance: H&E-specific stain augmentation and network ensembling reduced generalization error to unseen stain variations across single-center and multicenter WSIs.The results suggest robust detectors can be trained without multicenter training cohorts or stain-standardization algorithms.
  • Computational efficiency: Knowledge distillation shrank the model 28 times with negligible performance loss while preserving efficient dense prediction at gigapixel scale.The distilled network processed entire TUPAC WSIs at 0.25 µm/pixel resolution in 30–45 minutes.
  • Evaluation caveats: TUPACaux-test precision differed between the organizers’ annotations (0.467) and an observer’s assessment (0.707), partly because missed mitoses and the small test set distorted evaluation.The test set contained only 34 tiles.
  • Limitations: PHH3-derived reference standards contain noise from limited antibody sensitivity, limited specificity, and CNN2 performance, restricting detection of small training differences.CNN2 had an F1-score of 0.9; improved restaining and training protocols were proposed to reduce this effect.
  • Future work: Future work includes using whole-slide mitotic density to locate tumor hotspots and extending restaining-based datasets and H&E-specific augmentation beyond breast-tissue mitosis detection.Hotspot localization could potentially produce significant speedups in daily practice.

APPENDIX

The appendix describes optical-density color representation for histology images. RGB measurements are related to stain amounts through the Lambert-Beer law and converted into HED stain space using a normalized optical-density matrix.

  • Color representation: The Lambert-Beer law relates specimen light absorption to stain amount for each RGB channel.Ii is emitted radiant flux, I0,i is received radiant flux, A is stain amount, and ci is the absorption factor.
  • Optical density: Optical density is defined per channel because RGB relative absorption does not provide a linear relationship with stain amount.This conversion supports stain-aware processing in optical-density color space.
  • Optical density: Each optical-density vector represents a stain in the converted RGB color space, with hematoxylin producing channel values 0.18, 0.20, and 0.08.The values correspond to the RGB channels, respectively.
  • HED transformation: Hematoxylin, eosin, and DAB optical-density vectors form a normalized 3 × 3 matrix M linking OD-converted pixels to HED stain space.Each vector is divided by its total length to balance absorption factors across stains.
  • HED transformation: The stain-amount vector x is recovered from an OD-converted pixel vector y by inverting the normalized matrix M.x represents the per-pixel amounts of hematoxylin, eosin, and DAB.

B. Color stain augmentation algorithm

The color stain augmentation algorithm converts RGB patches into HED space, perturbs stain components, and converts the augmented representation back to RGB. This creates alternative stain intensities while preserving the patch format.

  • HED conversion: An RGB image patch is transformed into HED color space using the normalized hematoxylin–eosin–DAB matrix M.The patch is reshaped into N RGB pixels before transformation, and ε prevents numerical errors.
  • Stain perturbation: Stain augmentation stochastically modifies each HED stain component to simulate alternative stain intensities.The transformed patch S is converted into an augmented patch S′.
  • Stain perturbation: Each stain component is scaled by αi and shifted by βi, with αi sampled from U(1 − σ, 1 + σ), βi from U(−σ, σ), and typically σ = 0.05.The subscript i indexes the stain channels.
  • RGB reconstruction: The perturbed HED patch is converted back into RGB space by inverting the transformation, then reshaped to its original patch dimensions.The resulting P′ has N RGB pixels before reshaping.
Loading 1808.05896v1…