Source-linked AI summary
Detecting Cancer Metastases on Gigapixel Pathology Images
Yun Liu, Krishna Gadepalli, Mohammad Norouzi, George E. Dahl, Timo Kohlberger, Aleksey Boyko, Subhashini Venugopalan, Aleksei Timofeev, Philip Q. Nelson, Greg S. Corrado, Jason D. Hipp, Lily Peng, Martin C. Stumpe
TL;DR
Breast cancer staging depends on detecting lymph-node metastases, a time-consuming and error-prone task. This paper develops a CNN framework for tumor detection and localization, reporting state-of-the-art sensitivity and pathologist-level slide-level AUCs across two independent test sets.
Problem
Lymph-node metastasis detection is time-consuming and error-prone, despite its central role in breast cancer staging.
Method
A CNN framework uses Inception architecture, image patch sampling, data augmentation, and center-region tumor labeling for lymph-node metastasis detection.
Results
The method achieves state-of-the-art sensitivity for detecting small tumors and pathologist-level slide-level AUCs in two independent test sets.
Takeaways & Limitations
The approach could improve the accuracy and consistency of breast cancer case evaluation and potentially improve patient outcomes.
Takeaways & Limitations
Errors involve out-of-focus tissues and tissue-preparation artifacts, while hyperparameter tuning was limited by near-perfect validation performance.
Abstract
from arXiv · showhide
Each year, the treatment decisions for more than 230,000 breast cancer patients in the U.S. hinge on whether the cancer has metastasized away from the breast. Metastasis detection is currently performed by pathologists reviewing large expanses of biological tissues. This process is labor intensive and error-prone. We present a framework to automatically detect and localize tumors as small as 100 x 100 pixels in gigapixel microscopy images sized 100,000 x 100,000 pixels. Our method leverages a convolutional neural network (CNN) architecture and obtains state-of-the-art results on the Camelyon16 dataset in the challenging lesion-level tumor detection task. At 8 false positives per image, we detect 92.4% of the tumors, relative to 82.7% by the previous best automated approach. For comparison, a human pathologist attempting exhaustive search achieved 73.2% sensitivity. We achieve image-level AUC scores above 97% on both the Camelyon16 test set and an independent set of 110 slides. In addition, we discover that two slides in the Camelyon16 training set were erroneously labeled normal. Our approach could considerably reduce false negative rates in metastasis detection.
1 Introduction
Breast cancer staging depends on examining nearby lymph nodes for metastasis, but manual review is time-consuming and error-prone. The paper presents a CNN framework intended to improve automated detection and localization.
- Lymph-node metastasis evidence is a central component of breast cancer staging and treatment management.
- Manual metastasis review requires highly skilled pathologists and is particularly difficult for lymph nodes with no or small tumors.
- Computer-assisted detection could increase the sensitivity, speed, and consistency of metastasis detection.
- Deep CNNs have improved accuracy across image recognition, object detection, semantic segmentation, and healthcare applications.
- The proposed framework uses Inception architecture, patch sampling, and augmentation, while reporting state-of-the-art performance at 8 false positives per slide.
2 Methods
The method divides gigapixel slides into patches, predicts tumor probability over a sliding window, and aggregates the resulting heatmap for slide-level classification. It uses Inception V3 with center-region labels, balanced sampling, augmentation, and optional multi-scale inputs.
- The task prioritizes tumor detection and localization over pixel-level segmentation because slides are enormous and pixel-accurate annotations are difficult.
- Inference scans patches across each slide to generate a tumor probability heatmap, whose maximum value becomes the slide-level tumor prediction.
- The model uses Inception V3 with 299×299 inputs and predicts whether the central 128×128 region contains tumor.
- Multi-scale inputs combine patches at multiple magnifications, but preliminary experiments found no benefit from using up to four magnifications.
- Balanced training samples normal and tumor classes equally, then selects slides uniformly within the chosen class to reduce slide-size bias.
- Augmentation includes all eight rotations and flips, color perturbations, and jitter during patch extraction.
3 Evaluation and Datasets
Evaluation uses slide-level AUC and lesion-level FROC on Camelyon16, with bootstrap confidence intervals and a separate 110-slide set. The protocol emphasizes tumor localization at controlled false-positive rates.
- AUC evaluates slide-level classification, while FROC evaluates tumor detection and localization from heatmap-derived predictions.
- FROC reports sensitivity at 0.25, 0.5, 1, 2, 4, and 8 average false positives per tumor-negative slide.
- The evaluation reports 95% confidence intervals using 2,000 bootstrap samples of predicted points or slides.
- The study focuses on FROC because Camelyon16 contains approximately twice as many tumors as slides, improving metric reliability.
- Camelyon16 contains 400 slides, including 270 pixel-annotated slides and 130 unlabeled test slides.
- An additional evaluation set contains 110 H&E-stained lymph-node slides from 20 patients, including 57 tumor-containing slides.
4 Experiments & Results
Experiments show strong slide-level performance, robustness across evaluation settings, and several design findings about post-processing, pre-training, model size, multi-scale inputs, and color normalization. The authors also identify labeling errors and characterize remaining limitations.
- Slide-level classification: AUCs above 97% were achieved using the maximum heatmap value for slide-level classification, statistically indistinguishable from the current best results.The approach avoided training a slide-level classifier because validation AUC reached 100%.
- Tumor-level detection: A 1−5% FROC gain occurred with connected components when FROC was below 80%, but the approach was threshold-sensitive and could group nearby tumors.The authors instead used non-maxima suppression, which was relatively insensitive to radius values between 4 and 6.
- Model design: Pre-training improved convergence speed but did not improve FROC, potentially because pathology images differ substantially from ImageNet scenes.The authors attribute the result also to the large dataset and extensive augmentation enabling accurate training without pre-training.
- Model design: Slimmed-down Inception models with 3% of the parameters achieved similar performance to the full version, so subsequent experiments used the smaller model.The smaller model was motivated initially by faster experiment turnaround.
- Multi-scale analysis: Combining 40X inputs with lower magnifications provided no performance benefit, although it produced smoother heatmaps whose visual improvements could be deceptive.Some 40X speckles represented small non-tumor regions surrounded by tumor, while other heatmap regions reflected unannotated tumor.
- Color variation: Color normalization revealed no benefit, possibly because extensive color augmentations encouraged color-invariant features.The experiments addressed substantial color variation arising from tissue, preparation, and scanner differences.
- Ensembling: Ensembling eight rotations and flips improved metrics by a few percent, while independently trained-model ensembling produced smaller gains with diminishing returns after three models.The reported heatmaps included a three-model ensemble configuration.
- Additional validation: An independent set of 110 slides yielded an AUC of 97.6 (93.6, 100), despite different scanners, patients, and tissue-preparation protocols.The result was described as on-par with Camelyon16 test-set performance.
5 Conclusion
The method achieves state-of-the-art sensitivity for detecting small tumors in gigapixel pathology slides and reaches pathologist-level slide-level AUCs across two independent test sets.
- The method yields state-of-the-art sensitivity for detecting small tumors in gigapixel pathology slides.It reduces the false negative rate to a quarter of a pathologist and less than half of the previous best result.
- The method achieves pathologist-level slide-level AUCs in two independent test sets.
- The approach could improve the accuracy and consistency of evaluating breast cancer cases and potentially improve patient outcomes.
A.1 Dataset Details
The dataset details document exclusions caused by mislabeled or duplicated slides and define tumor-size categories and dataset annotations.
- Normal slide 144 was excluded after preliminary experiments uncovered tumors in the slide.
- Normal slide 086 was later found to contain tumors but remained in training for the reported results.
- Test slide 049 was excluded because it duplicated Tumor 036.
- Macrometastases are larger than 2000µm, while micrometastases are larger than 200µm and at most 2000µm.
A.2 Soft Labels
The experiments compared binary thresholded patch labels with soft labels based on tumor-pixel fractions, and thresholded labels performed substantially better.
- A patch was labeled positive when at least one pixel in its center 128 x 128 region was annotated as tumor.
- Soft labels assigned each patch the fraction of tumor pixels in the center region.
- Thresholded labels yielded substantially better performance than soft labels in preliminary experiments.
- The authors suggest this result might reflect training the model to assign lower values to small tumors, because FROC rewards detecting tumors of all sizes equally.
A.3 Image Color Normalization
The paper tested HSD-based color normalization to address substantial H&E staining variation, but neither inference-time nor training-time normalization improved performance.
- H&E-stained tissue varies substantially because of biological tissue, slide preparation, and scanner adjustments.
- The method maps RGB channels to optical density and then applies a Hue-Saturation-Intensity transformation, using HSD coordinates for hue and saturation.
- Color coordinates are normalized by fitting a Gaussian to tissue-containing patches and transforming their covariance toward a reference covariance.
- Intensity is normalized by transforming empirical patch-intensity mean and variance to reference values, whose color and intensity statistics are chosen as training-slide medians.
- The normalized coordinates are mapped back to RGB by inverse HSI transformation and inversion of the optical-density mapping.
- Inference-only color normalization reduced FROC by a few percent, while training and testing normalized-slide models did not improve performance.
A.4 Sample Results
Sample results illustrate that the framework identifies tumor cells amid visually similar normal tissue and challenging slide conditions. Experiments also uncovered additional tumor slides with incomplete annotations, verified by a pathologist.
- Annotation quality: Seven additional tumor slides were verified by a pathologist to have incomplete pixel-level annotations.The slides were 010, 025, 034, 056, 067, 085, and 110.
- Tumor identification: In H&E tissue, the predicted heatmap assigns higher probabilities to darker tumor regions than to lighter pink non-tumor regions.
- Tumor identification: The model distinguishes larger pink tumor cells from smaller pink macrophages in a slide labeled “Normal” 086.The heatmap ignores the macrophages while identifying the tumor cells.
- Tumor identification: The model identifies tumor clusters and interspersed tumor cells while ignoring lymphocytes and connective tissue in “Normal” 144.
- Challenging conditions: An additional validation patch contains artifacts, hemorrhage, necrosis, poor processing, and altered H&E coloration.Visible artifacts include an air bubble and white cutting streaks.