PaperlayerView exact revision on arXiv

Source-linked AI summary

A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises

S. Kevin Zhou, Hayit Greenspan, Christos Davatzikos, James S. Duncan, Bram van Ginneken, Anant Madabhushi, Jerry L. Prince, Daniel Rueckert, Ronald M. Summers

arXiv:2008.09104v2cs.CVeess.IV

TL;DR

Medical imaging offers important clinical evidence but poses challenges from distribution drift, sparse and noisy labels, and heterogeneous data. This survey links those traits and clinical needs to emerging deep-learning approaches, then reviews case studies and future directions, highlighting both successes and unresolved generalization, robustness, interpretability, and annotation concerns.

Abstract

from arXiv · show

Since its renaissance, deep learning has been widely used in various medical imaging tasks and has achieved remarkable success in many medical imaging applications, thereby propelling us into the so-called artificial intelligence (AI) era. It is known that the success of AI is mostly attributed to the availability of big data with annotations for a single task and the advances in high performance computing. However, medical imaging presents unique challenges that confront deep learning approaches. In this survey paper, we first present traits of medical imaging, highlight both clinical needs and technical challenges in medical imaging, and describe how emerging trends in deep learning are addressing these issues. We cover the topics of network architecture, sparse and noisy labels, federating learning, interpretability, uncertainty quantification, etc. Then, we present several case studies that are commonly found in clinical practice, including digital pathology and chest, brain, cardiovascular, and abdominal imaging. Rather than presenting an exhaustive literature survey, we instead describe some prominent research highlights related to these case study applications. We conclude with a discussion and presentation of promising future directions.

I. OVERVIEW

Medical imaging uses diverse physical phenomena to produce non-invasive or invasive visual representations of the body and supplies a major share of healthcare data. Its clinical importance motivates technological trends summarized in the review.

  • Medical imaging uses light, electromagnetic radiation, radioactivity, nuclear magnetic resonance, and sound to generate representations of body tissues.Source: §I · p. 1
  • X-ray radiography, CT, MRI, ultrasound, and digital pathology are among the most commonly used clinical imaging modalities.Source: §I · p. 1
  • 90% of healthcare data are accounted for by imaging data, making medical imaging an important source of evidence for clinical analysis and intervention.Source: §I · p. 1

A. Traits of medical imaging

Medical imaging combines heterogeneous, high-resolution data with fragmented acquisition, diverse diseases, sparse and noisy labels, and severe sample imbalance. These traits create a broad set of complex analysis tasks for deep learning systems.

  • Medical images span multiple modalities and dense pixel resolutions, with clinical CT and MRI reaching sub-millimeter spatial resolution.Ultrasound has even better spatial resolution and temporal resolution exceeding real-time.Source: A. Traits of medical imaging · p. 1
  • Non-standard acquisition equipment and settings create distribution drift, while privacy requirements scatter images across hospitals and imaging centers.Truly centralized open-source medical big data are rare.Source: A. Traits of medical imaging · pp. 1–2
  • Disease patterns are numerous and long-tailed, leaving most diseases infrequently observed despite sufficient cases for large-scale analysis of some common diseases.Source: A. Traits of medical imaging · p. 2
  • Medical-image labels are sparse, expensive, time-consuming, inconsistent across and within users, and lacking established gold standards.Source: A. Traits of medical imaging · p. 2
  • Samples are heterogeneous and imbalanced, with tumor pixels usually one to many orders of magnitude fewer than normal-tissue pixels.Source: A. Traits of medical imaging · p. 2
  • Medical imaging combines reconstruction, enhancement, restoration, classification, detection, segmentation, and registration across modalities and diseases into highly complex tasks.Source: A. Traits of medical imaging · p. 2

B. Clinical needs and applications

Medical imaging supports diagnosis, treatment planning, follow-up, and procedures, but human interpretation is constrained by workload, subjectivity, inter-reader variation, and fatigue. Deep learning can support physicians through automated detection, quantification, characterization, and decision-support tools.

  • Medical imaging informs diagnosis and treatment plans, verifies treatment success during follow-up, and supports surgical planning and real-time procedural imaging.Source: B. Clinical needs and applications · p. 2
  • CT scanners increased from 50–100 slices per case in 1990 to 1000–2500 slices today, while a prostate-biopsy whole-slide image can occupy 10GB at 40x magnification.Source: B. Clinical needs and applications · p. 2
  • Human interpretation is limited by subjectivity, variation across interpreters, fatigue, increasing image volume, missed findings, long turnaround times, and limited quantification.Source: B. Clinical needs and applications · p. 2
  • Deep learning can support physicians by automating pathological-finding detection, disease-extent quantification, pathology characterization, and decision support.Source: B. Clinical needs and applications · p. 2

C. Key technologies and deep learning

Medical-image technologies can be framed as function approximation tasks mapping images to task-specific outputs. The review emphasizes how deep learning connects emerging methods to medical-imaging needs rather than re-presenting established technical details.

D. Historical perspective

Deep learning became prominent in medical imaging after ImageNet-era advances, but adoption was initially constrained by limited labeled medical data. Transfer learning and synthetic augmentation addressed this data challenge, while U-Net became a key segmentation architecture.

  • The 2012 ImageNet challenge established CNN superiority and helped make deep learning a leading imaging and computer-vision method.Source: D. Historical perspective · p. 3
  • Limited labeled medical data was identified as the main challenge to applying deep learning in medical imaging.Source: D. Historical perspective · p. 3
  • Transfer learning enabled medical imaging models to reuse knowledge from source tasks, including natural-image pretraining, for related target problems.Source: D. Historical perspective · p. 3
  • Synthetic data augmentation emerged as a second strategy for limited datasets, with studies demonstrating that generated medical examples could increase task performance.Source: D. Historical perspective · p. 3
  • U-Net, originally designed for microscopic cell segmentation, efficiently and robustly learned features across many medical image segmentation tasks.Source: D. Historical perspective · p. 3

E. Emerging deep learning approaches

Emerging deep learning approaches address medical imaging’s architectural, annotation, privacy, knowledge, interpretability, and domain-shift challenges. The survey describes increasingly specialized models and learning strategies for extracting useful representations from limited, heterogeneous, and distributed data.

Network architectures

  • Deep networks provide greater model capacity and generalization than shallow networks, while architectural research has pursued depth, skip connections, deep supervision, adversarial learning, and attention.Source: E. Emerging deep learning approaches · pp. 3–4
  • Neural architecture search optimizes networks for specific medical imaging tasks, whereas lightweight design targets computational efficiency on resource-constrained devices while maintaining accuracy.Source: E. Emerging deep learning approaches · p. 4

Annotation-efficient learning

  • Annotation-efficient approaches address sparse and noisy labels through transfer learning, domain adaptation, self-supervised learning, semi-supervised learning, and weak or partial supervision.Source: E. Emerging deep learning approaches · p. 4
  • Transfer learning reuses source-task knowledge for related medical tasks, but natural-image pretraining is limited because ImageNet models are 2D and may not suit medical images.Source: E. Emerging deep learning approaches · p. 4
  • Domain adaptation aligns source and target domains with the same feature space, while related methods learn domain-invariant or cross-modality representations.Source: E. Emerging deep learning approaches · p. 4
  • Self-supervised, semi-supervised, weakly supervised, and unsupervised methods reduce reliance on pixel-level annotations by learning from proxy tasks, pseudo-labels, weak labels, or unlabeled data.Source: E. Emerging deep learning approaches · p. 4

Knowledge and trustworthiness

Federated learning

  • Federated learning trains shared models through distributed computation and aggregation without transferring hospital or laboratory data, supporting privacy-preserving collaboration.Source: E. Emerging deep learning approaches · p. 5

II. CASE STUDIES WITH PROGRESS HIGHLIGHTS

Because deep learning spans too many medical imaging applications for exhaustive coverage, the survey selects common clinical case studies and highlights prominent progress in each.

  • The survey covers selected clinical cases in chest, neuro, cardiovascular, abdominal, and microscopy imaging rather than attempting an exhaustive literature review.Source: §II · p. 5

A. Deep learning in thoracic imaging

Thoracic imaging case studies show deep learning advancing anatomical segmentation, lung-cancer screening, and COVID-19 decision support. Progress includes strong automated segmentation and systems that approach or exceed radiologist performance, while intertwined airway and vascular structures remain challenging.

Anatomical segmentation

COVID-19 case study

Detection and diagnosis

Decision support

COVID-19 case study

B. Deep learning in neuroimaging

Deep learning has expanded neuroimaging across segmentation, registration, prediction, and image synthesis, while addressing scanner variation and computational demands. Important gaps remain in limited-data learning, domain adaptation, semi-supervised designs, hyperparameter selection, and robustness.

Neuroimaging applications

  • Deep learning now supports neuroimaging segmentation, registration, prediction, and complex image synthesis tasks.Applications include brain extraction, cortical and subcortical segmentation, tumor and lesion segmentation, deformable registration, and predictive modeling.Source: B. Deep learning in neuroimaging · pp. 7–8

Neuroimage segmentation and tissue classification

  • Deep learning segmentation methods can reduce inference costs relative to atlas-based tools and target brain tissues, tumors, and lesions.Atlas-based methods require time-consuming deformable-registration optimization, whereas proposed deep learning methods are relatively computationally inexpensive during inference.Source: B. Deep learning in neuroimaging · p. 8
  • Adversarial training has been used to learn scanner-invariant feature representations for segmentation across datasets acquired on different scanners.The approach trains an adversarial network to classify scanner origin from downstream feature representations.Source: B. Deep learning in neuroimaging · p. 8

The use of GANs in neuroimaging

  • GAN-based neuroimaging methods enable cross-modality synthesis, motion-artifact reduction, resolution upscaling, and low-dose PET image estimation while preserving important image details.One reported low-tracer approach used a ten-fold reduction in tracer material while preserving important edge, structural, and textural details.Source: B. Deep learning in neuroimaging · p. 8
  • A U-Net-style mapping between MRI acquisition protocols reduced site-based variation and improved segmentation consistency between sites.The method paired subjects scanned with two protocols and applied super-resolution to lower-resolution images.Source: B. Deep learning in neuroimaging · p. 9

Open challenges

  • Neuroimaging still lacks rigorous understanding of limited-data learning, hyperparameter selection, domain adaptation, semi-supervised designs, and robustness.These areas are identified as important lines of further research.Source: B. Deep learning in neuroimaging · p. 9

C. Deep learning in cardiovascular imaging

Deep learning in cardiovascular imaging addresses chamber segmentation, cardiac motion tracking, and vessel analysis. The reviewed approaches combine spatial, temporal, anatomical, biomechanical, recurrent, multi-task, and graph-based designs across cardiac modalities.

Cardiac image segmentation

  • Cardiac chamber segmentation supports quantitative measures of morphology and function, including LV volume, mass, wall thickening, and ejection fraction.Deep learning work spans 2D and 3D cardiac images, with FCN- and U-Net-based strategies and added spatial or temporal context.Source: C. Deep learning in cardiovascular… · p. 9
  • A fully convolutional network for 2D short-axis cardiac MR images significantly outperformed traditional methods in accuracy and speed.The network segmented the LV, myocardium, and RV.Source: C. Deep learning in cardiovascular… · p. 9

Cardiac motion tracking

  • Joint cardiac motion tracking and segmentation networks simultaneously estimate motion and generate segmentation masks, using shared task information and mutual regularization.The 4D semi-supervised echocardiography approach lets one branch influence and regularize the other, with biomechanically inspired displacement regularization.Source: C. Deep learning in cardiovascular… · pp. 9–10, Figure 5 · p. 9

Cardiac vessel segmentation

  • Cardiac vessel segmentation supports aortic mechanical and hemodynamic characterization and coronary-artery analysis for stenosis assessment or fractional-flow-reserve simulation.Approaches include recurrent tracking, centerline extraction, lumen segmentation, and graph convolutional networks.Source: C. Deep learning in cardiovascular… · p. 10

D. Deep learning in abdominal imaging

Deep learning has advanced abdominal imaging through organ and lesion segmentation, disease detection, and opportunistic screening. Progress is supported by public datasets, but clinical translation depends on generalization across patient populations and image-acquisition variations.

Overview

  • Public datasets such as the MICCAI Data Decathlon and Deep Lesion have facilitated automated abdominal anatomy and disease analysis.Recent work spans localization, segmentation, detection, and classification.Source: D. Deep learning in abdominal imaging · p. 10

Organs and lesions

  • Liver segmentation Dice coefficients typically exceed 95%, while spleen segmentation has reached a Dice score of 0.962.Pancreas segmentation has reached the mid-80 percentile, and multi-organ networks can learn from partially labelled datasets.Source: D. Deep learning in abdominal imaging · p. 11
  • Deep learning methods address abdominal lymph nodes, bowel findings, kidney function, ovarian cancer, blood collections, and radiation-therapy organs at risk.Reported bowel applications include polyp detection, small-bowel segmentation, colonic inflammation, appendicitis, and small-bowel obstruction.Source: D. Deep learning in abdominal imaging · p. 11

Universal lesion detection

  • Universal lesion detectors identify, classify, and measure lymph nodes and multiple abdominal tumors using the Deep Lesion dataset.The approach is illustrated for abdominal CT.Source: D. Deep learning in abdominal imaging · p. 11

Opportunistic screening

  • Opportunistic abdominal screening has produced accurate measurements that generalize well to new patient populations.Examples include bone mineral densitometry, visceral fat, muscle volume and quality, and aortic atherosclerotic plaque quantification.Source: D. Deep learning in abdominal imaging · p. 11

Future directions

  • The most important advances sought for abdominal imaging include demonstrating generalizability across patient populations and image-acquisition variations.This is identified as a key requirement for continued progress and clinical translation.Source: D. Deep learning in abdominal imaging · p. 11

E. Deep learning in microscopy imaging

Deep learning in digital pathology has progressed from detecting and segmenting nuclei and other primitives to cancer detection, grading, mutation-related analysis, and outcome prediction. These applications show strong reported performance, while annotation demands, interpretability, training-set size, and generalizability remain concerns.

Nuclei detection and segmentation

  • Deep learning progressed from nuclei detection and segmentation to identifying cancer presence and extent in whole-slide breast cancer pathology images.The cited approach reported 100% accuracy for identifying cancer presence or absence at slide or patient level.Source: E. Deep learning in microscopy imaging · p. 11
  • Weakly supervised learning analyzed over 44K whole-slide images from over 15K patients, achieving area under the curve values above 0.98 across three cancers.The authors suggested excluding 65–75% of slides while retaining 100% cancer detection sensitivity.Source: E. Deep learning in microscopy imaging · p. 12

Disease grading

  • Deep learning has been applied to disease grading and to extracting histologic measurements associated with breast cancer risk categories and biological pathways.Reported examples include tubule density, mitotic index, and associations with Oncotype DX risk categories.Source: E. Deep learning in microscopy imaging · p. 12

Survival and disease outcome prediction

  • In early-stage colorectal cancer, deep learning predicted cancer-specific survival with hazard ratios of 3.84 in validation and 3.04 after adjustment for established prognostic markers.The study used over 12M pathology image tiles from over 2,000 patients.Source: E. Deep learning in microscopy imaging · p. 12

Limitations and interpretability

  • Digital pathology applications remain constrained by interpretability, large training-set requirements, annotated-data needs, and generalizability.Visual attention mapping and hybrid methods combining learned primitives with handcrafted features were proposed to improve transparency.Source: E. Deep learning in microscopy imaging · p. 12

III. DISCUSSION

The discussion frames clinical translation as requiring robust generalization across heterogeneous systems and effective integration with clinical workflows. It also highlights AI-assisted performance, pandemic-driven adaptation, emerging clinical support, and future patient-level, population-level, and discovery-oriented uses.

Technical challenges ahead

  • Medical imaging systems face robustness and generalization challenges across acquisition protocols, machines, hospitals, datasets, and countries.Data preprocessing, continuous model learning, and cross-system fine-tuning are identified as developments ahead.Source: §III · p. 13

How do we get new tools into the clinic?

  • Human experts consistently perform better with AI assistance than without it in studies estimating AI utility as a radiologist aid.This evidence motivates the question of how deep learning tools can enter clinical practice.Source: §III · p. 13
  • COVID-19 experience may help bridge clinical need and AI adoption through rapid adaptation of pretrained models and more adaptive regulatory steps.The cited advances include detection, localization, quantification, and characterization for hospitalized patients.Source: §III · p. 13
  • AI-based image evaluation can identify complex imaging patterns not perceptible through visual radiologic evaluation, supporting early detection and personalized risk estimates.Examples include anatomical, functional, and connectomic signatures of neurologic and neuropsychiatric diseases.Source: §III · p. 13

Future promise

  • Future systems are envisioned to combine images with clinical context and enable population-level analyses of disease, treatment response, adverse reactions, and medication interactions.This direction requires complex infrastructure and new privacy and security regulations across institutions and consortia.Source: §III · p. 13

Related papers

Loading 2008.09104v2…