PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning in Breast Cancer Imaging: A Decade of Progress and Future Directions

Luyang Luo, Xi Wang, Yi Lin, Xiaoqi Ma, Andong Tan, Ronald Chan, Varut Vardhanabhuti, Winnie CW Chu, Kwang-Ting Cheng, Hao Chen

arXiv:2304.06662v4eess.IVcs.CV

TL;DR

Deep learning has advanced breast cancer imaging across screening, diagnosis, treatment response prediction, and prognosis, but evidence remains constrained by limited data, labels, and evaluation designs. This survey reviews a decade of work across major imaging modalities and discusses challenges and future directions.

  • Problem

    Breast cancer imaging requires methods that can interpret complex imaging information across screening, diagnosis, treatment response prediction, and prognosis.

  • Method

    The paper surveys a decade of deep learning-based breast cancer imaging research across mammography, ultrasound, MRI, and digital pathology.

  • Results

    The review finds growing applications and high reported performance across modalities, with screening and diagnosis receiving the most attention and outcome prediction increasingly studied.

  • Takeaways & Limitations

    Deep learning shows promise for clinical breast cancer imaging applications, including screening, diagnosis, outcome prediction, personalized treatment planning, and biomarker discovery.

  • Takeaways & Limitations

    Progress is constrained by small public datasets, scarce or variable annotations, under-representation of demographic groups, and reliance on retrospective limited-dataset evaluations.

Abstract

from arXiv · show

Breast cancer has reached the highest incidence rate worldwide among all malignancies since 2020. Breast imaging plays a significant role in early diagnosis and intervention to improve the outcome of breast cancer patients. In the past decade, deep learning has shown remarkable progress in breast cancer imaging analysis, holding great promise in interpreting the rich information and complex context of breast imaging modalities. Considering the rapid improvement in deep learning technology and the increasing severity of breast cancer, it is critical to summarize past progress and identify future challenges to be addressed. This paper provides an extensive review of deep learning-based breast cancer imaging research, covering studies on mammogram, ultrasound, magnetic resonance imaging, and digital pathology images over the past decade. The major deep learning methods and applications on imaging-based screening, diagnosis, treatment response prediction, and prognosis are elaborated and discussed. Drawn from the findings of this survey, we present a comprehensive discussion of the challenges and potential avenues for future research in deep learning-based breast cancer imaging.

I. INTRODUCTION

Breast cancer remains a major global health burden, making early detection and effective treatment important. This survey reviews deep learning across breast imaging modalities and clinical applications, while identifying challenges and future directions.

  • 2.3 million new breast cancer cases were estimated worldwide in 2020, with 685,000 deaths.
  • Five-year survival varies sharply by stage, from >99% for stage I to 29% for stage IV, underscoring the value of early detection.
  • Breast imaging spans mammography, ultrasound, MRI, and pathology, providing information from macroscopic to microscopic levels.
  • Deep learning is used for screening, diagnosis, treatment response prediction, and prognosis in breast cancer imaging.
  • The survey addresses a gap by reviewing 366 papers from 2012 to 2022 across four imaging modalities and multiple clinical applications.

II. DEEP LEARNING METHODS FOR BREAST CANCER ANALYSIS

The paper introduces classification as a core deep learning task for breast cancer image analysis. Classification models map images or features to disease predictions and are commonly optimized with cross entropy.

  • Classification produces discrete predictions, such as whether a breast image contains cancer.
  • A classification model maps image or feature inputs X to disease outputs Y, typically represented as one-hot labels.
  • Classification models are typically optimized by minimizing cross entropy between the target y and model output ŷ.
  • CNNs replaced hand-crafted feature engineering with end-to-end image classification, while AlexNet’s 2012 ImageNet success helped launch modern deep learning.

B. Detection

Breast cancer imaging methods combine detection, segmentation, and multiple learning paradigms to localize lesions, delineate their contours, and learn from varied amounts of labeled data.

  • B. Detection: Detection predicts region-wise classes and bounding boxes, using classification and localization losses as sibling objectives.
  • B. Detection: Detection commonly uses cross entropy for classification and smooth L1 or other losses for bounding-box regression.
  • B. Detection: Fast R-CNN extracts and pools candidate proposals from a pretrained CNN before jointly performing localization and classification.
  • C. Segmentation: Segmentation outputs pixel-wise classifications that delineate object contours, typically through encoder-decoder networks with downsampling and upsampling.
  • Weakly supervised, semi-supervised, unsupervised, transfer, and multimodal learning address limited labels, limited data, domain gaps, and complementary imaging information.

III. DEEP LEARNING APPLICATIONS IN BREAST CANCER

Deep learning applications span screening, diagnosis, lesion localization, segmentation, and multi-task systems across breast imaging. Reported performance can approach or exceed expert benchmarks, but external generalization remains a concern.

  • Screening and diagnosis: Screening and diagnosis overlap in deep learning studies, which often determine malignancy from normal, benign, or suspicious subjects.
  • Mammogram-based screening and diagnosis: Mammogram studies classify malignancy, lesion type, breast density, and cancer risk, using CNNs and transfer learning from ImageNet.
  • Mammogram-based screening and diagnosis: Multi-view inputs, multiple-instance learning, and multi-task learning combine information from several mammographic images or related tasks.
  • Mammogram-based screening and diagnosis: Deep learning models have shown mammogram classification performance comparable to the average of 101 radiologists and can improve radiologists’ performance.
  • Mammogram-based screening and diagnosis: Despite expert-comparable or expert-surpassed accuracy, mammogram models can show significant performance drops on external testing data.

2) Ultrasound-based Diagnosis:

Deep learning-based ultrasound diagnosis addresses classification, segmentation, and combined CAD tasks while leveraging multimodal views, synthetic data, and limited annotations. Despite progress toward expert-level performance, ultrasound segmentation remains difficult because of artifacts, low contrast, and ambiguous boundaries.

  • Classification: Ultrasound studies use feature-level or output-level fusion to combine complementary information from multiple views and modalities.Clinical interpretation commonly integrates B-mode, color Doppler, and elastography images.
  • Classification: Data augmentation, transfer learning, pretrained models, and GAN-generated samples are used to reduce overfitting from limited training data.Unsupervised and semi-supervised learning also exploit unlabeled data through autoencoders, Mean Teacher, and Virtual Adversarial Training.
  • Segmentation: FCN and U-Net models support ultrasound segmentation, but speckle artifacts, low contrast, shadows, blurry boundaries, and variable shapes remain major challenges.Boundary-aware modules and additional boundary maps were designed to improve predictions in ambiguous lesion margins.
  • Segmentation: Attention mechanisms and saliency maps direct models toward informative regions or feature channels to improve segmentation.These approaches emphasize spatially specific regions and semantically important channels.
  • Multiple Tasks: Deep learning CAD systems combine lesion detection, segmentation, and classification through sequential or multitask designs, including weakly supervised and whole-breast ultrasound approaches.Studies also examine contrast-enhanced ultrasound videos and automated whole-breast ultrasound images.
  • Clinical Validation: Observer studies report that some deep learning approaches achieve expert-level performance, supporting their potential for commercial applications.Robustness methods include noise filters and noisy-annotation-tolerant networks for classification and segmentation.

3) MRI-based Diagnosis:

MRI-based deep learning diagnosis spans screening, benign–malignant classification, fine-grained prediction, detection, and segmentation. Research increasingly exploits temporal and multiparametric MRI information through recurrent networks and multimodal fusion.

  • Classification: Breast MRI studies address screening, benign–malignant classification, molecular subtype and BI-RADS prediction, metastasis prediction, detection, and segmentation.The review organizes MRI applications into classification, segmentation, detection, and other tasks.
  • MRI Architectures: LSTM and convolutional LSTM networks exploit temporal information in dynamic contrast-enhanced MRI sequences.Maximum intensity projection is also used to reduce the computational cost of 3D MRI.
  • Multimodal Fusion: Feature fusion significantly outperformed other fusion strategies for benign–malignant lesion classification, while DCE + T2 inputs performed best for axillary lymph node metastasis prediction.Studies also combine deep features with handcrafted radiomics features.
  • Learning Strategies: Pretrained networks and ensemble frameworks are used to address data shortage, accelerate convergence, and reduce model uncertainty.Multiple gating networks can share inputs and fuse outputs at the final stage.
  • Learning Strategies: Weakly supervised and unsupervised learning methods address scarce MRI labels, including whole-slide-style classification and unsupervised pretraining for molecular subtype prediction.These approaches transfer information from related source domains or avoid requiring targeted regions of interest.
  • Detection and Segmentation: U-Net-based architectures are the most commonly used structures for breast tumor segmentation, while reinforcement-learning and 3D RetinaNet methods support lesion detection.Detection studies include accurate localization and reduced inference time.
  • Detection and Segmentation: Classification can be combined with weakly supervised detection or ROI localization to support comprehensive lesion diagnosis.Examples use Class Activation Maps, conditional random fields, Faster R-CNN, and custom CNNs.

4) Digital Pathology Images-based Diagnosis:

Deep learning-based digital pathology diagnosis covers classification, segmentation, detection, and related tasks, with whole-slide analysis increasingly using weak or slide-level supervision. Key constraints include annotation burden, scarce nuclei labels, and domain shift from staining and imaging variability.

  • Classification: Breast pathology analysis uses deep learning for cancer type and grade classification, segmentation, detection, and other diagnostic tasks.Pathology imaging remains the gold standard for cancer characterization.
  • Classification: Patch-wise classification can aggregate local features into whole-slide predictions for lymph node metastasis or negative findings.The CAMELYON16 challenge used patch-level models followed by whole-slide feature aggregation and Random Forest classification.
  • Classification: Pixel- or patch-level annotation is costly because gigapixel whole-slide images require extensive expert pathologist labeling.CAMELYON16 included 270 whole-slide images at 40× magnification, with approximately the pixel count of ImageNet.
  • Classification: Multiple instance learning reduces annotation demands by extracting instance features and aggregating them into bag-level whole-slide predictions from slide-level labels.Follow-up work improves instance representation, discriminative feature extraction, and aggregation strategies.
  • Segmentation: Gland and nuclei segmentation supports morphological assessment used in breast cancer grading, diagnosis, and prognosis.Nuclear morphology includes size, shape, and texture, while grading assesses tubule formation, mitotic count, and nuclear pleomorphism.
  • Segmentation: Weak annotations such as scribbles or points are explored because one whole-slide image can contain tens of thousands of nuclei.Self-training, co-training, and multitask learning help reduce bias from incomplete coarse annotations.
  • Other Tasks: Staining and imaging variability can create domain shift between source and target domains in pathology image analysis.Virtual staining methods include transformations designed to reproduce clinically useful immunohistochemical information.
  • Clinical Validation: Pathologist observer studies are important for clinical validation, and reported applications include lymph node metastasis detection and pathology-based diagnosis.The cited studies report significant effectiveness compared with doctors and pathologists in several applications.

C. Prognosis

Deep learning prognosis research uses imaging, clinical, genomic, and biomarker information to predict survival and recurrence and to identify prognostic factors. Models also investigate relationships among tumor, immune, morphological, and genomic features.

  • Scope: Prognosis applications include survival prediction, recurrence prediction, and discovery of prognostic biomarkers and interactions among prognostic factors.The review distinguishes classification and other prognosis-related tasks.
  • Survival Prediction: Neural networks can segment collagen fibers and extract morphometric, density, orientation, texture, and fractal features for Cox-based hazard prediction.The extracted features are analyzed with a Cox regression model under the proportional hazard framework.
  • Survival Prediction: Multimodal survival models combine clinical information, multiscale whole-slide images, segmentation results, and genomic features.MobileNetv2 and bilinear neural networks are among the architectures used for these combinations.
  • Recurrence Prediction: Recurrence and metastasis prediction studies combine image-derived features with clinical information and recurrent or convolutional neural networks.Some approaches use whole-slide images together with clinical data for HER2-positive breast cancer outcomes.
  • Biomarker Discovery: Neural networks identify tumor, immune, fibroblast, and other cellular features that can yield independent and robust prognostic indicators.One study derived five machine tumor-infiltrating lymphocyte variables from detected cellular features.
  • Biomarker Discovery: High density of positive tumor-infiltrating lymphocyte cells correlates with improved survival, while graph neural networks model interactions among biomarkers and contextual whole-slide features.These models aim to reveal hidden prognostic values and interpretable biomarkers.
  • Biomarker Discovery: Attention-based fusion of whole-slide and genomic data identifies survival-associated imaging and genomic biomarkers linked to interactions between tumor-infiltrating lymphocytes and tumors.The approach specifically models interactions between the two feature types.

D. Discussion

Deep learning is increasingly feasible across breast imaging modalities and clinical tasks, with reported performance sometimes comparable to radiologists and growing use in screening, diagnosis, treatment response prediction, and prognosis. Progress remains constrained by limited data, generalization challenges, emerging modalities, and scarce expert annotations.

  • Deep learning methods have reported performance comparable to radiologists across mammography, ultrasound, MRI, and pathology imaging under limited data.
  • Mammography has the most mature clinical translation, including FDA-cleared or approved products for multi-view cancer classification and lesion detection.
  • MRI is used mainly for preoperative treatment planning and outcome prediction, while studies also explore screening, false-positive reduction, and shorter examinations with less contrast agent.
  • Pathology-image deep learning supports molecular subtyping, mitosis and metastasis detection, fine-grained classification, virtual staining, grading, and outcome prediction.
  • A. Robust Learning with Limited Data: Limited public datasets, especially outside mammography, may hinder model robustness and generalizability; federated learning and synthetic data are proposed privacy-preserving ways to expand training data.
  • A. Robust Learning with Limited Data: Limited training data also restricts generalization to unseen domains, motivating domain adaptation, domain generalization, fine-grained annotations, and adversarial robustness research.

C. Multimodal Learning for Information Fusion

Multimodal learning is increasingly relevant to breast cancer imaging because different modalities and nonimaging data provide complementary information, yet most surveyed studies use only one imaging type. Future work must address incomplete multimodal datasets, fairness, evaluation strength, and regulatory accountability.

  • C. Multimodal Learning for Information Fusion: Most surveyed multimodal studies combine only one type of imaging data, leaving the broader clinical context underused.Examples include multiple mammogram views, ultrasound modes, or MRI sequences rather than distinct imaging modalities.
  • C. Multimodal Learning for Information Fusion: Mammography, ultrasound, MRI, and pathology provide complementary information from macroscopic to microscopic scales, while clinical, molecular, genomic, and report data can further enrich patient descriptions.The survey notes that mammography is more sensitive to calcifications, MRI provides more spatial detail, and pathology reveals microscopic and cellular status.
  • C. Multimodal Learning for Information Fusion: Constructing complete patient-level datasets across imaging, demographics, genes, therapy, and clinical records remains difficult, creating a need to handle missing modalities.The authors call for open-source multimodal breast cancer datasets to stimulate further development.
  • D. Reliable and Explainable Model Learning: Deep learning models can inherit unfair diagnosis and shortcut learning when minority groups are under-represented or data contain spurious correlations.The survey highlights missing race and ethnicity reporting in many FDA-supported datasets and links insufficient diversity to under-learning of minority groups.
  • E. Personalized Treatment Planning: Multimodal learning is particularly important for prognosis and individualized treatment planning, where combining imaging, medical history, and genetic profiles may support precise biomarkers.The survey links multimodal information with improved prognostic accuracy and identifying patients who may benefit from escalated or de-escalated treatment.
  • F. Accountable AI Evaluation and Regulation: Clinical adoption requires external validation, prospective assessment, robust generalizability testing, clearer regulation, and stronger evidence than retrospective studies on limited datasets.The survey also identifies gaps in validation approaches for FDA-approved or FDA-cleared breast cancer AI software.

VI. APPENDIX

The appendix catalogs public mammography and digital breast tomosynthesis datasets across countries, imaging formats, cohort sizes, labels, and lesion annotations. Dataset scale and available clinical or lesion-level information vary substantially across resources.

  • VI. APPENDIX: MIAS contains 322 mediolateral oblique mammograms from 161 cases, categorized as normal, benign, or malignant with circular lesion annotations.The mini-MIAS version resizes and clips or pads images to 1024×1024 resolution.
  • VI. APPENDIX: DDSM contains 2,620 scanned film mammography studies and 10,480 images with pathological labels and manually generated abnormality bounding boxes.Each study includes mediolateral oblique and craniocaudal views for each breast.
  • VI. APPENDIX: CBIS-DDSM provides 10,239 curated mammography images with lesion segmentation masks.The dataset is a subset of DDSM selected by a trained mammographer.
  • VI. APPENDIX: Public datasets include smaller specialized resources such as BancoWeb, INBreast, BCDR, CMMD, and larger databases such as OMI-DB and the Digital Mammography DREAM Challenge.Their metadata range from lesion categories and BI-RADS scores to biopsy confirmation and molecular subtypes.
  • VI. APPENDIX: The Digital Mammography DREAM Challenge provides over 640,000 de-identified mammograms from over 86,000 women categorized into cancer and non-cancer.More than 99% of examinations include both craniocaudal and mediolateral oblique views for each breast.
  • VI. APPENDIX: BCS-DBT contains 22,032 digital breast tomosynthesis volumes from 5,060 patients with four diagnostic categories and 435 lesion bounding boxes.Most cases provide four views comprising mediolateral oblique and craniocaudal views of both breasts.

C. Ultrasound Datasets

Public breast ultrasound datasets span diagnostic classification, lesion segmentation, radio-frequency echoes, and clinical reporting. They differ widely in size, acquisition systems, patient cohorts, labels, and annotation types.

  • C. Ultrasound Datasets: BUSI contains 780 breast ultrasound images from 600 women, categorized as normal, benign, or malignant with lesion masks.The images were collected at Baheya Hospital in Cairo in 2018.
  • C. Ultrasound Datasets: UDIAT contains 163 images from different women, including 53 cancerous masses and 110 benign lesions, with radiologist-delineated lesions.Images were acquired with a Siemens ACUSON Sequoia C512 system and have a mean size of 760×570 pixels.
  • C. Ultrasound Datasets: OABRFD provides ultrasonic radio-frequency echoes from 52 malignant and 48 benign lesions recorded in 78 women.Each lesion includes two orthogonal scans from the pathological region.
  • C. Ultrasound Datasets: BUSIS comprises 562 B-mode ultrasound images collected across multiple hospitals and ultrasound devices.The dataset includes women aged 26 to 78 and was collected using systems from GE, Hitachi, Philips, and Siemens.
  • C. Ultrasound Datasets: Additional resources include Radiopaedia’s 439 tumor images with reports, the 78,000-woman NCID record collection, and smaller labeled datasets from Shantou and Thammasat University hospitals.These resources vary from image-level cases and reports to expert contours and majority-vote ground truth.
  • C. Ultrasound Datasets: BUS-BRA contains 1,875 images from 1,064 patients with biopsy-proven tumors, BI-RADS categories 2–5, and ground-truth division into tumoral and normal regions.The dataset supports both diagnostic labeling and lesion delineation.

D. MRI Datasets

Public breast MRI datasets support imaging research on treatment response, recurrence, prognosis, radiogenomics, and multimodal diagnosis. They combine MRI sequences with clinical, pathological, molecular, or longitudinal treatment information to varying degrees.

  • D. MRI Datasets: The Duke Breast Cancer MRI dataset contains 922 biopsy-confirmed invasive breast cancer patients with multiple T1-weighted sequences and nonimaging clinical information.Available information includes demographics, treatments, tumor characteristics, and recurrence.
  • D. MRI Datasets: QIN-Breast contains longitudinal PET/CT and quantitative MRI for 68 patients at three treatment time points, with patient-level treatment-response labels.MRI includes diffusion-weighted, dynamic contrast-enhanced, and T1-mapping data.
  • D. MRI Datasets: ISPY1 contains 222 subjects with MRI examinations at four time points for evaluating treatment response and recurrence risk.The protocol includes localization, T2-weighted, and contrast-enhanced T1-weighted sequences.
  • D. MRI Datasets: ISPY2 includes 719 patients with locally advanced breast cancer from 22 clinical centers, each receiving four MRI exams before and after treatment.The dataset also includes histopathologic and demographic data such as race, ethnicity, age, treatment arm, menopausal status, and receptor subtype.
  • D. MRI Datasets: ACRIN-6698 enrolled 406 women and provides diffusion-weighted, T2-weighted, and dynamic contrast-enhanced MRI for assessing neoadjuvant chemotherapy response.The cohort includes randomized experimental-treatment and control arms.
  • D. MRI Datasets: TCGA-BRCA, TCGA-Breast-Radiogenomics, and BREAST-DIAGNOSIS connect MRI with mammography, pathology, clinical data, molecular assays, or multiple diagnostic modalities.BREAST-DIAGNOSIS includes MRI, mammogram, CT, and PET together with clinical and pathological information.

E. Pathology Image Datasets

The survey compiles pathology-image datasets spanning classification, segmentation, detection, scoring, biomarker assessment, and metastasis analysis. These resources vary in image type, annotation granularity, cohort size, staining, and clinical task.

  • Challenge datasets: MITOS-ATYPIA-14 provides frames from whole-slide images for mitosis detection and nuclear atypia grading.Two pathologists annotated mitosis categories, while three junior pathologists graded nuclear atypia.
  • Metastasis datasets: CAMELYON16 and CAMELYON17 address breast-cancer metastasis detection in lymph-node whole-slide images at slide and patient levels, respectively.CAMELYON17 additionally provides patient-level pN-stage annotations and lesion-level metastasis locations.
  • Challenge datasets: BACH supports four-class histology classification and segmentation using microscopy images and whole-slide images.The classes are normal, benign, in situ carcinoma, and invasive carcinoma.
  • Annotation and biomarker datasets: The dataset collection includes resources for mitosis, apoptosis, tumor, tubule, nuclear, cellularity, and receptor-status analysis across H&E, IHC, and other stained images.Examples include BreCaHAD, HER2C, UCSB Bio-Segmentation, BreastPathQ, and ABCTB-derived images.
  • Large and multi-source datasets: Large or multi-source resources include BreaKHis, TCGA breast whole-slide images, MoNuSeg, and HASHI, supporting heterogeneous histopathology and multimodal analyses.Their annotations and metadata include patient labels, nuclear boundaries, cancer-region masks, and molecular or clinical information.

F. Full Tables of Surveyed Papers

The survey tables organize deep learning studies by imaging modality, clinical task, and application stage. They cover screening, diagnosis, pathology analysis, treatment-response prediction, and prognosis.

  • Mammogram screening and diagnosis: The survey tables classify mammogram studies into classification, segmentation, detection, and other tasks.The overview is specifically focused on mammogram screening and diagnosis.
  • Ultrasound screening and diagnosis: Ultrasound studies are organized by classification, segmentation, and detection tasks.The corresponding table summarizes deep learning-based ultrasound screening and diagnosis.
  • MRI diagnosis: MRI diagnosis studies are grouped across classification, segmentation, detection, and other tasks, with multiple MRI acquisition types represented.The table includes modalities such as DCE, T1, T1+C, T2, DWI, and ADC.
  • Pathology diagnosis: Pathology diagnosis studies cover classification, segmentation, detection, stain normalization, and virtual staining.The table uses CL, SE, DE, SN, and VS to distinguish these task categories.
  • Treatment-response prediction: Treatment-response studies include classification and other tasks across NAC, pCR, ultrasound, MRI, CT, mammograms, whole-slide images, and pathology modalities.The table also distinguishes image-based from non-image modalities.
  • Prognosis: Prognosis studies include classification, survival prediction, recurrence prediction, and other tasks across imaging and non-image modalities.The overview includes H&E and Sirius Red pathology images, ultrasound, and mammograms.
Loading 2304.06662v4…