Source-linked AI summary
High-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networks
Krzysztof J. Geras, Stacey Wolfson, Yiqiu Shen, Nan Wu, S. Gene Kim, Eric Kim, Laura Heacock, Ujas Parikh, Linda Moy, Kyunghyun Cho
TL;DR
Medical-image screening requires approaches that preserve fine details and combine multiple views, unlike conventional natural-image architectures. The paper develops a multi-view high-resolution network for mammography and finds that larger training sets and original-resolution images improve performance.
Problem
Natural-image convolutional architectures downscale inputs despite medical-image detection depending on fine details, while mammography exams require multiple views to be fused.
Method
The paper develops a multi-view deep convolutional network that processes separate mammography views, concatenates their representations, and predicts BI-RADS assessments from high-resolution images.
Results
Performance improves as the training set grows, and experiments show that keeping mammography images at high resolution is essential.
Takeaways & Limitations
Large-scale training and preservation of high resolution are important for applying multi-view deep convolutional networks to breast cancer screening.
Takeaways & Limitations
The study did not systematically search learning hyperparameters because computational resources were limited.
Abstract
from arXiv · showhide
Advances in deep learning for natural images have prompted a surge of interest in applying similar techniques to medical images. The majority of the initial attempts focused on replacing the input of a deep convolutional neural network with a medical image, which does not take into consideration the fundamental differences between these two types of images. Specifically, fine details are necessary for detection in medical images, unlike in natural images where coarse structures matter most. This difference makes it inadequate to use the existing network architectures developed for natural images, because they work on heavily downscaled images to reduce the memory requirements. This hides details necessary to make accurate predictions. Additionally, a single exam in medical imaging often comes with a set of views which must be fused in order to reach a correct conclusion. In our work, we propose to use a multi-view deep convolutional neural network that handles a set of high-resolution medical images. We evaluate it on large-scale mammography-based breast cancer screening (BI-RADS prediction) using 886,000 images. We focus on investigating the impact of the training set size and image size on the prediction accuracy. Our results highlight that performance increases with the size of training set, and that the best performance can only be achieved using the original resolution. In the reader study, performed on a random subset of the test set, we confirmed the efficacy of our model, which achieved performance comparable to a committee of radiologists when presented with the same data.
I. INTRODUCTION
Breast screening reduces mortality but generates substantial false-positive recalls, follow-ups, biopsies, anxiety, and costs. The paper addresses limitations in prior medical-imaging deep learning by developing and evaluating a large-scale, multi-view, high-resolution approach for BI-RADS prediction.
- 30% reduction in mortality was shown for asymptomatic women undergoing screening mammography in multiple randomized clinical trials.
- 10-15% of screening mammograms lead to recalls, corresponding to approximately 3.3 to 4.5 million callback exams for additional testing.
- Only 20-40% of biopsies following abnormal screening mammograms yield a cancer diagnosis, creating substantial avoidable anxiety and follow-up costs.
- Prior medical-imaging deep learning studies often did not investigate how medical images differ fundamentally from natural images or how those differences affect model design.
- The study used 201,698 screening exams containing 886,437 images to train a multi-view network that predicts BI-RADS 0, 1, or 2 without downscaling.
- A reader study found the model was almost as accurate as a committee of radiologists on a random subset of the test set.
II. HIGH-RESOLUTION MULTI-VIEW DEEP CONVOLUTIONAL NEURAL NETWORKS
The proposed MV-DCN combines information from the four standard mammography views while training view-specific feature extractors and a shared classifier jointly. This design directly accommodates the multi-view structure of screening exams.
- Screening mammography provides four standard images per exam: L-CC, R-CC, L-MLO, and R-MLO.
- The approach builds on prior multi-view deep convolutional classification work rather than using multi-view representations only as an unsupervised feature extractor.
- The MV-DCN applies convolutional and pooling layers separately to each view before fusing their learned representations.
- Concatenated view-specific representations are passed to a fully connected layer and softmax layer that produce p(y|x).
- The complete network is trained jointly using stochastic gradient descent with backpropagation.
C. High-Resolution Convolutional Neural Network
Mammography requires preserving subtle diagnostic cues that can disappear during the aggressive downscaling common in natural-image networks. The model therefore processes full-resolution images while reducing feature-map size early through aggressive convolution and pooling.
- Subtle diagnostic findings in screening mammography may be identifiable only at the original image resolution.
- 224 × 224 inputs are typical of high-performing natural-image systems, but this downscaling is undesirable for mammography.
- Aggressive convolution and pooling reduce feature-map dimensions early to make full-resolution processing computationally feasible.
- The first convolutional layers use strides larger than one, and the first pooling layer uses a larger stride than later pooling layers.
- Figure 1 illustrates the four standard views for BI-RADS 0, BI-RADS 1, and BI-RADS 2 exams, including malignant, normal, and postsurgical examples.
III. RELATED WORK
Earlier mammography systems commonly targeted individual pipeline stages, used small datasets or patches, and often relied on controlled outcome distributions. The paper contrasts these practices with end-to-end, large-scale screening under a natural population distribution.
- Multi-Stage vs. End-to-End Approaches: Traditional screening systems segment the breast, identify regions of interest, classify lesions, and combine results across multiple views.
- Multi-Stage vs. End-to-End Approaches: Most recent deep-learning studies replaced one or more pipeline stages, such as mass detection or region-of-interest classification.
- Data Size: Earlier work generally used fewer than 1,000 images for training and testing, encouraging patch-based or region-of-interest training instead of end-to-end learning.
- Natural vs. Controlled Distribution: The paper's training distribution contained approximately 46% BI-RADS 1, 41% BI-RADS 2, and 13% BI-RADS 0 cases.
- Natural vs. Controlled Distribution: Artificially balancing benign and malignant cases can bias models toward malignant predictions and more frequent recalls than necessary.
- Table I summarizes prior breast-mammography deep-learning studies and their task, end-to-end status, and dataset characteristics.
IV. DATA
The study used a large retrospective mammography dataset from five affiliated imaging sites, partitioned into patient-disjoint training, validation, and test sets. Images were retained under defined technical and view-quality criteria, with BI-RADS distributions reported across splits.
- A. Collection: 129,208 patients contributed consecutive screening mammograms collected from 2010–2016 at five New York University–affiliated imaging sites.The cohort included patients aged 19–99, with mean age 57.2 years and standard deviation 11.6.
- B. Data Statistics: Patients were ordered by latest exam date, with the first 80% assigned to training, the next 10% to validation, and the final 10% to testing.For test patients, performance was evaluated only on the latest exam to estimate accuracy on future exams.
- A. Collection: Images were excluded only for non-screening views, technical errors, dimensions below 2600 × 2000 pixels, or magnification factors outside 1.0–1.1.Included exams required at least one correct image for each standard view.
- B. Data Statistics: Table II reports each BI-RADS cell as the number of exams followed by the number of images for training, validation, and test data.
C. Data preprocessing and augmentation
Preprocessing standardized mammography views while preserving their large spatial scale. The proposed pipeline represents each view with a convolutional-network column before combining multiple views.
- C. Data preprocessing and augmentation: Each image was normalized by subtracting its pixel mean and dividing by its pixel standard deviation.Right craniocaudal and mediolateral oblique views were horizontally flipped so the breast appeared on the same side.
- C. Data preprocessing and augmentation: Images were cropped to 2600 × 2000 pixels to unify batch sizes, maintain similar scale, and remove uninformative background.
- C. Data preprocessing and augmentation: A single-view network column transforms a grayscale input image into a 256-dimensional vector.
- C. Data preprocessing and augmentation: The multi-view network overview connects separate convolutional-network columns for the available mammographic views.The figure defines DCN as the convolutional-network column and uses arrows to indicate information flow.
V. SETTINGS
The model was evaluated with macro-averaged one-versus-rest AUC and analyzed using predictive confidence. Its multi-view architecture combines view-specific processing with regularization and validation-based model selection.
- A. Evaluation Metrics: macAUC averages three one-versus-rest AUC values, treating each BI-RADS class in turn as positive and the other two as negative.
- A. Evaluation Metrics: The network’s confidence was quantified from the entropy of its conditional prediction distribution p(y|x).Higher entropy indicates lower confidence, enabling AUC comparisons across confidence levels.
- B. Model Setup: Each mammographic view had a separate convolutional column, and corresponding left-right view columns shared weights.
- B. Model Setup: Figure 4 compares class-specific ROCs for all test cases with ROCs from cases on which the network was confident.BI-RADS 1 and BI-RADS 2 ROCs improve substantially for confident examples, whereas BI-RADS 0 remains similar.
- B. Model Setup: Gaussian input noise with standard deviation 0.01 and dropout at rate 0.2 regularized training, but both were disabled during validation and testing.
- B. Model Setup: The model used Adam with initial learning rate 10^-5, mini-batches of four, and up to 100 epochs on one NVIDIA Tesla V100 GPU.The reported test model was the epoch with the lowest validation macAUC; training took approximately four weeks.
VI. QUANTITATIVE RESULTS ANALYSIS
Experiments tested how training-set size affects classification performance. Performance improved as the number of training examples increased, supporting the use of large training sets.
- VI. QUANTITATIVE RESULTS ANALYSIS: 100%, 50%, 20%, 10%, 5%, 2%, and 1% training-set sizes were evaluated using separately trained networks.Subsets were created by random sampling without replacement.
- VI. QUANTITATIVE RESULTS ANALYSIS: Classification performance improved as the number of training examples increased.The authors identify this trend as evidence for the importance of a large training set.
B. Effect of Resolution
The study tested how input resolution and prediction confidence affect mammography classification performance. Lower-resolution inputs were compared with original-resolution images, and high-confidence predictions were evaluated separately.
- Confidence: Prediction confidence was measured using the entropy of the network’s output distribution.Exams were stratified by class, sorted by entropy, and evaluated after selecting test examples below class-specific validation thresholds.
- Training data: Increasing the amount of training data yielded better results.
- Confidence: High-confidence predictions were more accurate across the experiments.High-confidence macAUC was defined using examples whose prediction entropy fell below the threshold for k = 30% of validation examples.
VII. VISUALIZATION
The paper introduces a visualization method for examining which mammographic pixels influence the network’s predictive confidence. It addresses the interpretability challenge of deep convolutional networks without requiring additional training.
- VII. VISUALIZATION: The proposed visualization avoids the additional training required by some recent approaches.The paper motivates this simpler technique by noting that existing visualization methods are computationally inefficient and difficult to apply to medical images.
- VII. VISUALIZATION: The visualization measures the network output’s sensitivity to perturbations of individual input pixels.Backpropagation is used to compute how the entropy of the predictive distribution changes with respect to each pixel.
- VII. VISUALIZATION: Pixels that strongly influence prediction confidence receive high visualization values, while minimally contributing pixels approach zero.
- VII. VISUALIZATION: The method was demonstrated on two patients whose breast cancer was confirmed by follow-up examination.
IX. CONCLUSIONS
The paper presents an initial end-to-end study of multi-view deep convolutional networks for breast cancer screening. Its conclusions emphasize high-resolution inputs and large training sets, while identifying unresolved performance and computational limitations.
- IX. CONCLUSIONS: The study is a first step toward end-to-end large-scale training of multi-view deep convolutional networks for breast cancer screening.
- IX. CONCLUSIONS: High-resolution images were essential to the model’s performance.The authors expect this finding to apply to other medical-image tasks where fine details determine the outcome.
- IX. CONCLUSIONS: The model’s performance had not saturated despite using the largest breast cancer screening dataset reported in the literature.The authors therefore expect performance to improve with more data.
- IX. CONCLUSIONS: Performance was lowest when differentiating BI-RADS 0 from the other classes, similarly to the participating doctors.The authors suggest future labeling based on whether a person subsequently develops breast cancer.
- IX. CONCLUSIONS: Limited computational resources prevented a systematic search for optimal learning hyperparameters.The authors state that additional computational resources could improve results without substantially changing the methodology.