Source-linked AI summary
Deep Neural Networks Improve Radiologists' Performance in Breast Cancer Screening
Nan Wu, Jason Phang, Jungkyu Park, Yiqiu Shen, Zhe Huang, Masha Zorin, Stanisław Jastrzębski, Thibault Févry, Joe Katsnelson, Eric Kim, Stacey Wolfson, Ujas Parikh, Sushma Gaddam, Leng Leng Young Lin, Kara Ho, Joshua D. Weinstein, Beatriu Reig, Yiming Gao, Hildegard Toth, Kristine Pysarenko, Alana Lewin, Jiyon Lee, Krystal Airola, Eralda Mema, Stephanie Chung, Esther Hwang, Naziya Samreen, S. Gene Kim, Laura Heacock, Linda Moy, Kyunghyun Cho, Krzysztof J. Geras
TL;DR
The paper asks whether deep neural networks can classify breast cancer screening exams using local and global image information. It develops an image-and-heatmaps model and finds this approach strongest overall for cancer classification.
Problem
The paper examines whether neural networks can accurately classify breast cancer screening exams using both breast-level and pixel-level labels.
Method
The model combines a patch-level network producing heatmaps with a breast-level network that uses local and global visual information.
Results
The image-and-heatmaps model remained the strongest overall model for cancer classification.
Takeaways & Limitations
Effectively using both local and global visual information supports superior performance on the cancer classification problem.
Takeaways & Limitations
Comparisons with other methods may be misleading because datasets differ in population distributions, imaging quality, and label definitions.
Abstract
from arXiv · showhide
We present a deep convolutional neural network for breast cancer screening exam classification, trained and evaluated on over 200,000 exams (over 1,000,000 images). Our network achieves an AUC of 0.895 in predicting whether there is a cancer in the breast, when tested on the screening population. We attribute the high accuracy of our model to a two-stage training procedure, which allows us to use a very high-capacity patch-level network to learn from pixel-level labels alongside a network learning from macroscopic breast-level labels. To validate our model, we conducted a reader study with 14 readers, each reading 720 screening mammogram exams, and find our model to be as accurate as experienced radiologists when presented with the same data. Finally, we show that a hybrid model, averaging probability of malignancy predicted by a radiologist with a prediction of our neural network, is more accurate than either of the two separately. To better understand our results, we conduct a thorough analysis of our network's performance on different subpopulations of the screening population, model design, training procedure, errors, and properties of its internal representations.
Data
The retrospective dataset comprised 229,426 screening mammography exams from 141,473 patients, with breast-level labels derived from biopsy pathology and pixel-level lesion annotations from radiologists. Biopsied exams included malignant, benign, and mixed findings, while many biopsied lesions were mammographically occult.
- Dataset: 229,426 digital screening mammography exams comprising 1,001,093 images were collected from 141,473 patients.Each exam contained at least four images corresponding to the standard screening views.
- Labels: 5,832 exams had at least one biopsy within 120 days, yielding 985 breasts with malignant findings and 5,556 with benign findings.These corresponded to 8.4% and 47.6% of breasts, respectively.
- Labels: 234 breasts had both malignant and benign findings, while unmatched screening exams were labeled as having neither finding in either breast.The mixed finding category represented 2.0% of breasts.
- Pixel-level annotations: Radiologists retrospectively marked biopsied lesion locations at pixel level using the corresponding pathology reports.Approximately 32.8% of exams were mammographically occult, meaning the biopsied lesions were not visible on mammography even retrospectively.
Discussion
The authors attribute accurate screening-exam classification to large-scale breast-level and pixel-level supervision combined with a computationally intensive patch-level model. They note that the relatively simple design and basic visible-cancer task leave substantial opportunities for more sophisticated models and future-cancer prediction.
- Contributions: Large-scale breast-level and pixel-level labels enabled accurate breast cancer screening-exam classification.The patch-level model was densely applied to images to produce heatmaps that served as additional input channels for a breast-level model.
- Limitations: The model design is relatively simple, leaving room for more sophisticated and accurate models.The authors also characterize predicting visible cancer at screening as the simplest task among tasks of interest.
- Future directions: Future work should test real-time screening-mammogram reading and predict breast cancer before it becomes visible to trained human observers.The authors identify prediction of future cancer before visual detectability as a clear next step.
Additional analysis of results
Additional analyses show that combining original mammograms with patch-classification heatmaps provides the strongest overall performance, while model predictions vary by breast view and label. Error analyses further reveal ambiguity in mammographic findings and limitations in integrating information across images and views.
- Heatmaps-only model: The image-and-heatmaps model remains the strongest overall model, indicating that combining local and global visual information improves cancer classification.The heatmaps-only model performs comparably with the image-only model for malignant/not malignant classification but underperforms for benign/not benign classification.
- Correlation between model predictions: Predictions for benign and malignant findings are highly correlated within the same breast but show low correlations between breasts.The image-and-heatmaps ensemble has lower benign–malignant correlation than the image-only ensemble.
- Comparison of CC and MLO model branches: The full model averages symmetric CC and MLO branch predictions, generally outperforming either branch individually except for benign prediction in the biopsied population.The CC branch performs better overall, consistent with radiologists’ view that findings may be more conspicuous in the CC view.
- Classifying malignant vs. benign: 0.738 was the AUC attained by the image-only ensemble for distinguishing malignant from benign findings in a one-class biopsied subpopulation.This analysis used 384 breasts labeled only malignant or only benign and normalized the two class probabilities to sum to one.
- Error analysis and limitations: False positives and false negatives illustrate inherent ambiguity in imaging, while the model also lacks the ability to summarize changes across multiple images and views.Radiologists sometimes use global mammographic context and architectural distortion across views when assessing malignancy.
Network architecture and training
The breast-level classifier processes four standard mammographic views through shared-weight, view-specific ResNet columns and produces four binary label probabilities. Training uses pretrained view-column weights, Adam optimization, cross-entropy across labels, sampling strategies for scarce positive cases, and validation-AUC early stopping.
- Architecture: Four view-specific ResNet columns process the standard mammographic views, with weights shared across corresponding CC and MLO sides and four binary probability outputs.The model predicts left-benign, left-malignant, right-benign, and right-malignant labels.
- Training procedure: Training initializes view-specific columns from BI-RADS pretraining and optimizes the full model with Adam, learning rate 10^-5, minibatch size 4, and four-label cross-entropy.L2 regularization uses coefficient 10^-4.5.
- Model selection: 20 epochs without improvement in average validation AUC trigger early stopping, after which the model version with the best validation AUC is selected as the final candidate.The average validation AUC is computed across the four labels on the validation set.
- Evaluation during training: Training and validation AUC curves use different data distributions: subsampled balanced biopsied and non-biopsied examples for training, and the full validation set.Therefore, the training and validation AUC curves are not directly comparable.
- Model variants: 6,132,592 trainable parameters comprise the image-only model, versus 6,135,728 for the image-and-heatmaps model.On an Nvidia V100 GPU, training takes about 12 hours for the image-only model and about 24 hours for the image-and-heatmaps model.
Data augmentation for model training
Training used size and location augmentation to increase sample diversity and robustness, while test predictions were averaged across 10 random augmentations. Validation used no data augmentation.
- Training augmentation: Size augmentation varied crop-window size and resized crops with bicubic interpolation to the model’s desired input size.Location augmentation added noise around the chosen optimal crop-window center.
- Test-time augmentation: 10 random augmentations were applied at test time, with predictions averaged to compute each sample’s prediction.
- Validation: No data augmentation was used during validation.
Details of the auxiliary patch-level classifier
The auxiliary patch-level classifier was trained on pixel-level radiologist annotations to distinguish four patch classes, using class-balanced sampling to address the rarity of visible findings. A DenseNet-121 generated malignant and benign heatmaps that were evaluated as additional channels for the breast-level classifier.
- Patch labels: The classifier distinguished malignant, benign, outside, and negative patches using radiologist-provided pixel-level annotations linked to pathology results.Mammographically occult lesions were excluded from training-set generation.
- Training and architecture: DenseNet-121 provided the patch-level architecture, with approximately seven million parameters and ImageNet initialization.Its four dense blocks contained 6, 12, 24, and 16 dense layers, respectively.
- Training and architecture: 10,000 patches were sampled per training epoch: 20 malignant, 35 benign, 5,000 outside, and 4,945 negative.Visible biopsied findings comprised 0.85% of images and 0.87% of image area, motivating the sampling ratio.
- Patch classification heatmap generation: The classifier was applied in sliding windows to full-resolution images to produce malignant and benign class-specific heatmaps.The heatmaps were intended as extra channels for the breast-level classifier.
- Model evaluation and selection: 2,000 epochs were trained, with parameters saved every 200 epochs for validation-based model selection using generated heatmaps.Ten saved models were used to generate malignant and benign heatmaps for validation images.