Source-linked AI summary
A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises
S. Kevin Zhou, Hayit Greenspan, Christos Davatzikos, James S. Duncan, Bram van Ginneken, Anant Madabhushi, Jerry L. Prince, Daniel Rueckert, Ronald M. Summers
TL;DR
Medical imaging offers important clinical evidence but poses challenges from distribution drift, sparse and noisy labels, and heterogeneous data. This survey links those traits and clinical needs to emerging deep-learning approaches, then reviews case studies and future directions, highlighting both successes and unresolved generalization, robustness, interpretability, and annotation concerns.
Problem
Medical imaging confronts deep learning with distribution drift, sparse and noisy labels, annotation demands, interpretability concerns, and generalization challenges across systems and institutions.
Method
The paper surveys medical-imaging traits, clinical needs, emerging deep-learning trends, and representative case studies rather than presenting an exhaustive literature survey.
Results
The reviewed studies report successful deep-learning applications, including pathology models with AUCs above 0.98 and survival prediction with hazard ratios of 3.84 and 3.04.
Takeaways & Limitations
Deep learning shows promise for image-based detection, localization, quantification, characterization, treatment assessment, and identifying complex imaging patterns in clinical applications.
Takeaways & Limitations
The field still faces robustness and generalization challenges across acquisition protocols, machines, hospitals, and countries, alongside concerns about interpretability and annotated data.
Abstract
from arXiv · showhide
Since its renaissance, deep learning has been widely used in various medical imaging tasks and has achieved remarkable success in many medical imaging applications, thereby propelling us into the so-called artificial intelligence (AI) era. It is known that the success of AI is mostly attributed to the availability of big data with annotations for a single task and the advances in high performance computing. However, medical imaging presents unique challenges that confront deep learning approaches. In this survey paper, we first present traits of medical imaging, highlight both clinical needs and technical challenges in medical imaging, and describe how emerging trends in deep learning are addressing these issues. We cover the topics of network architecture, sparse and noisy labels, federating learning, interpretability, uncertainty quantification, etc. Then, we present several case studies that are commonly found in clinical practice, including digital pathology and chest, brain, cardiovascular, and abdominal imaging. Rather than presenting an exhaustive literature survey, we instead describe some prominent research highlights related to these case study applications. We conclude with a discussion and presentation of promising future directions.
I. OVERVIEW
Medical imaging uses diverse physical phenomena to produce non-invasive or invasive visual representations of the body and supplies a major share of healthcare data. Its clinical importance motivates technological trends summarized in the review.
- Medical imaging uses light, electromagnetic radiation, radioactivity, nuclear magnetic resonance, and sound to generate representations of body tissues.
- X-ray radiography, CT, MRI, ultrasound, and digital pathology are among the most commonly used clinical imaging modalities.
- 90% of healthcare data are accounted for by imaging data, making medical imaging an important source of evidence for clinical analysis and intervention.
A. Traits of medical imaging
Medical imaging combines heterogeneous, high-resolution data with fragmented acquisition, diverse diseases, sparse and noisy labels, and severe sample imbalance. These traits create a broad set of complex analysis tasks for deep learning systems.
- Medical images span multiple modalities and dense pixel resolutions, with clinical CT and MRI reaching sub-millimeter spatial resolution.Ultrasound has even better spatial resolution and temporal resolution exceeding real-time.
- Non-standard acquisition equipment and settings create distribution drift, while privacy requirements scatter images across hospitals and imaging centers.Truly centralized open-source medical big data are rare.
- Disease patterns are numerous and long-tailed, leaving most diseases infrequently observed despite sufficient cases for large-scale analysis of some common diseases.
- Medical-image labels are sparse, expensive, time-consuming, inconsistent across and within users, and lacking established gold standards.
- Samples are heterogeneous and imbalanced, with tumor pixels usually one to many orders of magnitude fewer than normal-tissue pixels.
- Medical imaging combines reconstruction, enhancement, restoration, classification, detection, segmentation, and registration across modalities and diseases into highly complex tasks.
B. Clinical needs and applications
Medical imaging supports diagnosis, treatment planning, follow-up, and procedures, but human interpretation is constrained by workload, subjectivity, inter-reader variation, and fatigue. Deep learning can support physicians through automated detection, quantification, characterization, and decision-support tools.
- Medical imaging informs diagnosis and treatment plans, verifies treatment success during follow-up, and supports surgical planning and real-time procedural imaging.
- CT scanners increased from 50–100 slices per case in 1990 to 1000–2500 slices today, while a prostate-biopsy whole-slide image can occupy 10GB at 40x magnification.
- Human interpretation is limited by subjectivity, variation across interpreters, fatigue, increasing image volume, missed findings, long turnaround times, and limited quantification.
- Deep learning can support physicians by automating pathological-finding detection, disease-extent quantification, pathology characterization, and decision support.
C. Key technologies and deep learning
Medical-image technologies can be framed as function approximation tasks mapping images to task-specific outputs. The review emphasizes how deep learning connects emerging methods to medical-imaging needs rather than re-presenting established technical details.
- Medical-image reconstruction forms images from scanner signals, including high-quality reconstruction from low-dose or fast acquisitions.
- Enhancement adjusts image intensities through denoising, super-resolution, bias-field correction, harmonization, and modality translation or synthesis.
- Segmentation assigns pixel labels to form objects, while registration aligns images into a common coordinate system for analysis, fusion, and label transfer.
- CADe localizes lesions with bounding boxes, whereas CADx classifies localized lesions as benign, malignant, or among multiple lesion types.
- Across these technologies, deep learning approximates y = F(x), mapping one or more images to task-specific outputs such as masks, deformation fields, or enhanced images.
- The review omits established deep-learning technical details and instead focuses on connections between emerging approaches, medical-imaging needs, and representative case studies.
D. Historical perspective
Deep learning became prominent in medical imaging after ImageNet-era advances, but adoption was initially constrained by limited labeled medical data. Transfer learning and synthetic augmentation addressed this data challenge, while U-Net became a key segmentation architecture.
- The 2012 ImageNet challenge established CNN superiority and helped make deep learning a leading imaging and computer-vision method.
- Limited labeled medical data was identified as the main challenge to applying deep learning in medical imaging.
- Transfer learning enabled medical imaging models to reuse knowledge from source tasks, including natural-image pretraining, for related target problems.
- Synthetic data augmentation emerged as a second strategy for limited datasets, with studies demonstrating that generated medical examples could increase task performance.
- U-Net, originally designed for microscopic cell segmentation, efficiently and robustly learned features across many medical image segmentation tasks.
E. Emerging deep learning approaches
Emerging deep learning approaches address medical imaging’s architectural, annotation, privacy, knowledge, interpretability, and domain-shift challenges. The survey describes increasingly specialized models and learning strategies for extracting useful representations from limited, heterogeneous, and distributed data.
- Network architectures: Deep networks provide greater model capacity and generalization than shallow networks, while architectural research has pursued depth, skip connections, deep supervision, adversarial learning, and attention.
- Network architectures: Neural architecture search optimizes networks for specific medical imaging tasks, whereas lightweight design targets computational efficiency on resource-constrained devices while maintaining accuracy.
- Annotation-efficient learning: Annotation-efficient approaches address sparse and noisy labels through transfer learning, domain adaptation, self-supervised learning, semi-supervised learning, and weak or partial supervision.
- Annotation-efficient learning: Transfer learning reuses source-task knowledge for related medical tasks, but natural-image pretraining is limited because ImageNet models are 2D and may not suit medical images.
- Annotation-efficient learning: Domain adaptation aligns source and target domains with the same feature space, while related methods learn domain-invariant or cross-modality representations.
- Annotation-efficient learning: Self-supervised, semi-supervised, weakly supervised, and unsupervised methods reduce reliance on pixel-level annotations by learning from proxy tasks, pseudo-labels, weak labels, or unlabeled data.
- Knowledge and trustworthiness: Anatomy knowledge from unpaired CT can decompose chest x-rays into lung, bone, and remaining structures; augmented bone-suppressed images improved prediction for 11 of 14 common lung diseases.
- Federated learning: Federated learning trains shared models through distributed computation and aggregation without transferring hospital or laboratory data, supporting privacy-preserving collaboration.
II. CASE STUDIES WITH PROGRESS HIGHLIGHTS
Because deep learning spans too many medical imaging applications for exhaustive coverage, the survey selects common clinical case studies and highlights prominent progress in each.
- The survey covers selected clinical cases in chest, neuro, cardiovascular, abdominal, and microscopy imaging rather than attempting an exhaustive literature review.
A. Deep learning in thoracic imaging
Thoracic imaging case studies show deep learning advancing anatomical segmentation, lung-cancer screening, and COVID-19 decision support. Progress includes strong automated segmentation and systems that approach or exceed radiologist performance, while intertwined airway and vascular structures remain challenging.
- Chest imaging commonly uses radiography and CT, with CT providing high-quality, high-resolution lung images at low radiation dose.
- Anatomical segmentation: Deep learning became state of the art for lung and lobe segmentation, with fully automatic U-Net variants outperforming leading interactive methods in recent LOLA11 results.
- Anatomical segmentation: Accurate automated segmentation of intertwined arteries, veins, and airway trees remains an open challenge because these structures have complex geometry.
- COVID-19 case study: CO-RADS-AI combines coronal CT slices, lobe segmentation, abnormal-area detection, CO-RADS prediction, and per-lobe CT severity scoring for COVID-19 assessment.
- Detection and diagnosis: Public chest x-ray datasets totaling 868K images enabled extensive abnormality-detection research, but report-mined labels are noisy.
- Decision support: A deep learning lung-cancer screening model outperformed six radiologists following Lung-RADS, while its follow-up-scan performance was slightly below human experts.
- COVID-19 case study: A COVID-19 chest-radiograph system evaluated on 454 images from another hospital performed comparably to six chest radiologists.
B. Deep learning in neuroimaging
Deep learning has expanded neuroimaging across segmentation, registration, prediction, and image synthesis, while addressing scanner variation and computational demands. Important gaps remain in limited-data learning, domain adaptation, semi-supervised designs, hyperparameter selection, and robustness.
- Neuroimaging applications: Deep learning now supports neuroimaging segmentation, registration, prediction, and complex image synthesis tasks.Applications include brain extraction, cortical and subcortical segmentation, tumor and lesion segmentation, deformable registration, and predictive modeling.
- Neuroimage segmentation and tissue classification: Deep learning segmentation methods can reduce inference costs relative to atlas-based tools and target brain tissues, tumors, and lesions.Atlas-based methods require time-consuming deformable-registration optimization, whereas proposed deep learning methods are relatively computationally inexpensive during inference.
- Neuroimage segmentation and tissue classification: Adversarial training has been used to learn scanner-invariant feature representations for segmentation across datasets acquired on different scanners.The approach trains an adversarial network to classify scanner origin from downstream feature representations.
- The use of GANs in neuroimaging: GAN-based neuroimaging methods enable cross-modality synthesis, motion-artifact reduction, resolution upscaling, and low-dose PET image estimation while preserving important image details.One reported low-tracer approach used a ten-fold reduction in tracer material while preserving important edge, structural, and textural details.
- The use of GANs in neuroimaging: A U-Net-style mapping between MRI acquisition protocols reduced site-based variation and improved segmentation consistency between sites.The method paired subjects scanned with two protocols and applied super-resolution to lower-resolution images.
- Open challenges: Neuroimaging still lacks rigorous understanding of limited-data learning, hyperparameter selection, domain adaptation, semi-supervised designs, and robustness.These areas are identified as important lines of further research.
C. Deep learning in cardiovascular imaging
Deep learning in cardiovascular imaging addresses chamber segmentation, cardiac motion tracking, and vessel analysis. The reviewed approaches combine spatial, temporal, anatomical, biomechanical, recurrent, multi-task, and graph-based designs across cardiac modalities.
- Cardiac image segmentation: Cardiac chamber segmentation supports quantitative measures of morphology and function, including LV volume, mass, wall thickening, and ejection fraction.Deep learning work spans 2D and 3D cardiac images, with FCN- and U-Net-based strategies and added spatial or temporal context.
- Cardiac image segmentation: A fully convolutional network for 2D short-axis cardiac MR images significantly outperformed traditional methods in accuracy and speed.The network segmented the LV, myocardium, and RV.
- Cardiac motion tracking: Joint cardiac motion tracking and segmentation networks simultaneously estimate motion and generate segmentation masks, using shared task information and mutual regularization.The 4D semi-supervised echocardiography approach lets one branch influence and regularize the other, with biomechanically inspired displacement regularization.
- Cardiac vessel segmentation: Cardiac vessel segmentation supports aortic mechanical and hemodynamic characterization and coronary-artery analysis for stenosis assessment or fractional-flow-reserve simulation.Approaches include recurrent tracking, centerline extraction, lumen segmentation, and graph convolutional networks.
D. Deep learning in abdominal imaging
Deep learning has advanced abdominal imaging through organ and lesion segmentation, disease detection, and opportunistic screening. Progress is supported by public datasets, but clinical translation depends on generalization across patient populations and image-acquisition variations.
- Overview: Public datasets such as the MICCAI Data Decathlon and Deep Lesion have facilitated automated abdominal anatomy and disease analysis.Recent work spans localization, segmentation, detection, and classification.
- Organs and lesions: Liver segmentation Dice coefficients typically exceed 95%, while spleen segmentation has reached a Dice score of 0.962.Pancreas segmentation has reached the mid-80 percentile, and multi-organ networks can learn from partially labelled datasets.
- Organs and lesions: Deep learning methods address abdominal lymph nodes, bowel findings, kidney function, ovarian cancer, blood collections, and radiation-therapy organs at risk.Reported bowel applications include polyp detection, small-bowel segmentation, colonic inflammation, appendicitis, and small-bowel obstruction.
- Universal lesion detection: Universal lesion detectors identify, classify, and measure lymph nodes and multiple abdominal tumors using the Deep Lesion dataset.The approach is illustrated for abdominal CT.
- Opportunistic screening: Opportunistic abdominal screening has produced accurate measurements that generalize well to new patient populations.Examples include bone mineral densitometry, visceral fat, muscle volume and quality, and aortic atherosclerotic plaque quantification.
- Future directions: The most important advances sought for abdominal imaging include demonstrating generalizability across patient populations and image-acquisition variations.This is identified as a key requirement for continued progress and clinical translation.
E. Deep learning in microscopy imaging
Deep learning in digital pathology has progressed from detecting and segmenting nuclei and other primitives to cancer detection, grading, mutation-related analysis, and outcome prediction. These applications show strong reported performance, while annotation demands, interpretability, training-set size, and generalizability remain concerns.
- Nuclei detection and segmentation: Deep learning progressed from nuclei detection and segmentation to identifying cancer presence and extent in whole-slide breast cancer pathology images.The cited approach reported 100% accuracy for identifying cancer presence or absence at slide or patient level.
- Nuclei detection and segmentation: Weakly supervised learning analyzed over 44K whole-slide images from over 15K patients, achieving area under the curve values above 0.98 across three cancers.The authors suggested excluding 65–75% of slides while retaining 100% cancer detection sensitivity.
- Disease grading: Deep learning has been applied to disease grading and to extracting histologic measurements associated with breast cancer risk categories and biological pathways.Reported examples include tubule density, mitotic index, and associations with Oncotype DX risk categories.
- Survival and disease outcome prediction: In early-stage colorectal cancer, deep learning predicted cancer-specific survival with hazard ratios of 3.84 in validation and 3.04 after adjustment for established prognostic markers.The study used over 12M pathology image tiles from over 2,000 patients.
- Limitations and interpretability: Digital pathology applications remain constrained by interpretability, large training-set requirements, annotated-data needs, and generalizability.Visual attention mapping and hybrid methods combining learned primitives with handcrafted features were proposed to improve transparency.
III. DISCUSSION
The discussion frames clinical translation as requiring robust generalization across heterogeneous systems and effective integration with clinical workflows. It also highlights AI-assisted performance, pandemic-driven adaptation, emerging clinical support, and future patient-level, population-level, and discovery-oriented uses.
- Technical challenges ahead: Medical imaging systems face robustness and generalization challenges across acquisition protocols, machines, hospitals, datasets, and countries.Data preprocessing, continuous model learning, and cross-system fine-tuning are identified as developments ahead.
- How do we get new tools into the clinic?: Human experts consistently perform better with AI assistance than without it in studies estimating AI utility as a radiologist aid.This evidence motivates the question of how deep learning tools can enter clinical practice.
- How do we get new tools into the clinic?: COVID-19 experience may help bridge clinical need and AI adoption through rapid adaptation of pretrained models and more adaptive regulatory steps.The cited advances include detection, localization, quantification, and characterization for hospitalized patients.
- How do we get new tools into the clinic?: AI-based image evaluation can identify complex imaging patterns not perceptible through visual radiologic evaluation, supporting early detection and personalized risk estimates.Examples include anatomical, functional, and connectomic signatures of neurologic and neuropsychiatric diseases.
- Future promise: Future systems are envisioned to combine images with clinical context and enable population-level analyses of disease, treatment response, adverse reactions, and medication interactions.This direction requires complex infrastructure and new privacy and security regulations across institutions and consortia.