Source-linked AI summary
An interpretable classifier for high-resolution breast cancer screening images utilizing weakly supervised localization
Yiqiu Shen, Nan Wu, Jason Phang, Jungkyu Park, Kangning Liu, Sudarshini Tyagi, Laura Heacock, S. Gene Kim, Linda Moy, Kyunghyun Cho, Krzysztof J. Geras
TL;DR
High-resolution mammograms contain small, sparse lesions, making direct application of natural-image classifiers challenging. GMIC combines memory-efficient global context and high-capacity local details with fusion, achieving radiologist-level performance while improving efficiency and interpretability.
Problem
Small, sparse mammographic lesions and essential global and local diagnostic information challenge classifiers designed for natural images.
Method
GMIC uses a memory-efficient global module to generate saliency maps and select regions, a local module to analyze details, and fusion to combine global and local features.
Results
The model surpasses average radiologist AUC by 0.11 and outperforms ResNet-34 while being 4.3x faster and using 76.1% less GPU memory.
Takeaways & Limitations
GMIC processes original-resolution medical images with image-level training while producing pixel-level saliency maps for interpretability.
Takeaways & Limitations
Different learning speeds can reduce patch diversity, causing the local module to overfit and its validation AUC to decrease.
Abstract
from arXiv · showhide
Medical images differ from natural images in significantly higher resolutions and smaller regions of interest. Because of these differences, neural network architectures that work well for natural images might not be applicable to medical image analysis. In this work, we extend the globally-aware multiple instance classifier, a framework we proposed to address these unique properties of medical images. This model first uses a low-capacity, yet memory-efficient, network on the whole image to identify the most informative regions. It then applies another higher-capacity network to collect details from chosen regions. Finally, it employs a fusion module that aggregates global and local information to make a final prediction. While existing methods often require lesion segmentation during training, our model is trained with only image-level labels and can generate pixel-level saliency maps indicating possible malignant findings. We apply the model to screening mammography interpretation: predicting the presence or absence of benign and malignant lesions. On the NYU Breast Cancer Screening Dataset, consisting of more than one million images, our model achieves an AUC of 0.93 in classifying breasts with malignant findings, outperforming ResNet-34 and Faster R-CNN. Compared to ResNet-34, our model is 4.1x faster for inference while using 78.4% less GPU memory. Furthermore, we demonstrate, in a reader study, that our model surpasses radiologist-level AUC by a margin of 0.11. The proposed model is available online: https://github.com/nyukat/GMIC.
1. Introduction
Screening mammography is difficult because lesions are small, sparse, subtle, and embedded in very high-resolution images where both local details and global breast structure matter. GMIC addresses these properties by combining global localization with detailed analysis of selected regions, achieving strong classification, interpretability, and computational efficiency using image-level labels.
- Motivation: Screening mammography is challenging because small, sparse, subtle lesions appear in very high-resolution images and can trigger false-positive recalls and biopsies.Only 20% to 40% of biopsies following abnormal screening mammograms yield cancer diagnoses.
- Motivation: Mammography differs from natural images because accurate diagnosis requires both local lesion details and global breast structure.Microcalcifications are common in benign and malignant findings, but a ductal distribution is a strong indicator of malignancy.
- Approach: GMIC uses a memory-efficient global module to generate saliency maps, selects informative regions, and combines local details with global context for prediction.This design supports processing screening mammograms at their original resolutions while keeping GPU memory manageable.
- Approach: GMIC extends the original architecture with a fusion module that combines global and local features for more accurate predictions.The paper reports that the fusion module outperforms the original design in an ablation study.
- Results: 0.93 AUC was achieved for identifying breasts with malignant findings on a held-out test set, outperforming baseline approaches including ResNet-34.The evaluation used the NYU Breast Cancer Screening Dataset and 14,148 held-out exams.
- Results: 0.11 higher AUC than average radiologists was reported in a reader study, while weakly supervised saliency maps correlated with objects of interest.GMIC also used 78.4% less GPU memory and was 4.1x faster during inference than ResNet-34 while being more accurate.
2. Methods
GMIC processes full-resolution mammograms by using a compact global network to produce saliency maps, selecting informative ROI patches for detailed local analysis, and fusing global and local information for prediction.
- GMIC frames screening mammography as multi-label classification of benign and malignant lesion presence from a grayscale image.
- The global network extracts whole-image features and converts them into benign and malignant saliency maps that indicate approximate lesion locations.
- A compact global network preserves original image resolution while reducing memory use, then retrieves K informative patches for higher-capacity local processing.The saliency maps have resolution h = 46, w = 30, while the input has H = 2944, W = 1920; the local network processes fine-grained features from selected patches.
- Top t% pooling aggregates the strongest saliency-map locations, balancing global average pooling’s diluted signal against global max pooling’s unstable single-location gradient.Global max pooling corresponds to t = 1/(h×w), whereas global average pooling corresponds to t = 100%.
- Gated attention assigns learnable relevance scores to local patches before aggregating their fine-grained representations into a local vector.The attention mechanism uses a sigmoid-based nonlinearity and produces an attention-weighted representation for the selected patches.
- The fusion network concatenates global max-pooled features with local patch features and applies a sigmoid classifier to produce the final prediction.The model also uses L1 regularization to encourage sparser saliency maps and trains end-to-end with a binary cross-entropy-based loss.
3. Experiments and Results
Experiments evaluate GMIC’s breast-level classification, efficiency, reader comparison, localization, and ablated components. GMIC combines global and local signals, achieves strong malignant-lesion performance, and provides localization without pixel-level training labels.
- Evaluation setup: GMIC defines breast-level predictions by averaging the two image-level predictions from each breast’s CC and MLO views.Classification is reported with breast-level AUC; reader-study comparisons also use PRAUC.
- Classification performance: 0.930 AUC was achieved by the best GMIC ensemble for identifying breasts with malignant findings.The ensemble averages predictions from the top five GMIC-ResNet-18, GMIC-ResNet-34, and GMIC-ResNet-50 models.
- Efficiency: 28.8% fewer parameters, 78.43% less GPU memory, 4.1x faster inference, and 5.6x faster training were achieved by GMIC-ResNet-18 versus ResNet-34.The efficiency gains are attributed to avoiding excessive whole-image computation and selectively focusing on informative regions.
- Reader study: GMIC achieved AUC 0.891 and PRAUC 0.39, exceeding DMV-CNN’s AUC 0.876 and PRAUC 0.318 and the average reader’s AUC 0.778.Individual-reader AUCs ranged from 0.705 to 0.860, while reader PRAUCs ranged from 0.244 to 0.453.
- Reader study: 0.114 mean-AUC improvement was obtained by hybrid models combining GMIC with individual readers at λ = 0.5.These hybrids averaged AUC 0.892 and PRAUC 0.449, improving readers’ mean PRAUC by 0.085.
- Localization: GMIC produced meaningful lesion localization without pixel-level supervision, with saliency maps and attention concentrated on annotated lesions in examples.The best localization model achieved mean test DSC 0.325 for malignant lesions and 0.240 for benign lesions.
- Ablation study: Fusion consistently outperformed global-only, local-only, and averaged global-local predictions, indicating synergy beyond simple averaging.The ablation also found that disabling patch-wise attention reduced malignant-lesion AUC from 0.898 to 0.874.
4. Related Work
Medical-image classification has evolved from whole-image CNNs and patch-based aggregation toward methods that select task-relevant regions, but breast screening remains challenging because lesions are small and sparse. Weakly supervised localization reduces reliance on costly pixel-level annotations, while ensemble weighting can combine complementary models.
- Whole-image and patch-based methods: Whole-image CNNs can struggle with medical images because regions of interest are typically small and sparsely distributed.This limitation motivates approaches that focus computation on informative regions rather than processing the entire image indiscriminately.
- Whole-image and patch-based methods: Patch-based methods divide images into tiles, classify each tile, and aggregate patch-level predictions into an image-level prediction.This strategy has been widely applied to pathology-image segmentation and classification.
- Whole-image and patch-based methods: Task-relevant patch selection methods use coarse attention or localization maps to identify important regions before applying more detailed analysis.Examples include UNet-generated coarse attention maps and related selection strategies for intravascular optical coherence tomography.
- Ensembling: The ensemble’s optimal weights assign the largest coefficient to GMIC while retaining non-negligible weights for two other model types.The reader-study and test-set optima achieve AUCs of 0.905 and 0.939, respectively.
- Breast cancer screening: Breast-screening systems have used multi-view CNNs and pixel-level labels to improve small-lesion detection and localization.Prior approaches include four-view classification and patch-level classifiers trained with segmentation annotations.
- Weakly supervised localization: Weakly supervised object detection avoids reliance on expensive segmentation labels and has been applied to medical disease classification, cell segmentation, and lesion detection.Attention gates and spatial-smoothing approaches are among the medical-imaging applications described.
5. Discussion and Conclusion
GMIC is designed for high-resolution mammograms by combining memory-efficient global context, selected local detail, and fusion-based prediction. It produces interpretable saliency maps from image-level labels, while its training remains more complex because global-module stabilization can reduce local-patch diversity and promote overfitting.
- Discussion and Conclusion: GMIC combines a low-capacity global module, a higher-capacity local module, and a fusion module for high-resolution mammogram classification.The global module generates coarse localization, the local module extracts fine-grained details from selected regions, and fusion produces the final prediction.
- Discussion and Conclusion: GMIC processes original-resolution medical images memory-efficiently while retaining capacity to capture fine visual details.This design addresses the combination of high image resolution and small regions of interest in medical imaging.
- Discussion and Conclusion: GMIC generates pixel-level saliency maps despite training with only image-level labels, providing additional interpretability.The saliency maps are intended to indicate possible benign or malignant findings.
- Discussion and Conclusion: GMIC outperforms ResNet-34 while being 4.3x faster and using 76.1% fewer GPU memory in the reported mammography evaluation.The conclusion also reports predictions as accurate as radiologists given equivalent input information.
- Discussion and Conclusion: Training GMIC is slightly more complex than training standard ResNet models because the global and local modules learn at different speeds.As global learning stabilizes, reduced patch diversity can cause local-module overfitting and declining validation AUC.