PaperlayerView exact revision on arXiv

Source-linked AI summary

Recent advances and clinical applications of deep learning in medical image analysis

Xuxin Chen, Ximin Wang, Ke Zhang, Kar-Ming Fung, Theresa C. Thai, Kathleen Moore, Robert S. Mannel, Hong Liu, Bin Zheng, Yuchen Qiu

arXiv:2105.13381v3cs.CVeess.IV

TL;DR

Medical image analysis faces limited large, well-annotated datasets and variability in clinical interpretation. This paper reviews recent deep learning advances, emphasizing unsupervised and semi-supervised methods across four major tasks, and discusses technical challenges and future directions.

  • Problem

    Medical image analysis is limited by scarce large, well-annotated datasets and inter-reader variability that affects disease detection and diagnosis.

  • Method

    The paper synthesizes recent deep learning studies, emphasizing unsupervised and semi-supervised approaches across classification, segmentation, detection, and registration.

  • Results

    The review summarizes recent advances, architectures, performance-enhancement strategies, and applications of deep learning in medical image analysis.

  • Takeaways & Limitations

    Unsupervised and semi-supervised learning are presented as promising approaches for addressing medical imaging annotation constraints across multiple analysis tasks.

  • Takeaways & Limitations

    Self-supervised contrastive learning had not yet achieved wide application in medical image analysis when the paper was written.

Abstract

from arXiv · show

Deep learning has received extensive research interest in developing new medical image processing algorithms, and deep learning based models have been remarkably successful in a variety of medical imaging tasks to support disease detection and diagnosis. Despite the success, the further improvement of deep learning models in medical image analysis is majorly bottlenecked by the lack of large-sized and well-annotated datasets. In the past five years, many studies have focused on addressing this challenge. In this paper, we reviewed and summarized these recent studies to provide a comprehensive overview of applying deep learning methods in various medical image analysis tasks. Especially, we emphasize the latest progress and contributions of state-of-the-art unsupervised and semi-supervised deep learning in medical image analysis, which are summarized based on different application scenarios, including classification, segmentation, detection, and image registration. We also discuss the major technical challenges and suggest the possible solutions in future research efforts.

1. INTRODUCTION

Deep learning has become mainstream in computer-aided medical image analysis, supporting diverse clinical imaging tasks while addressing limitations of clinician-dependent interpretation. This survey reviews recent advances, especially unsupervised and semi-supervised methods, across four major tasks and selected imaging modalities.

  • Clinical image detection and diagnosis depend on individual clinicians’ expertise, producing substantial inter-reader variability and motivating computer-aided detection and diagnosis schemes.
  • Deep learning models progressively learn hidden image patterns through hierarchical neural networks, amplifying task-relevant properties while filtering irrelevant features.
  • Deep learning methods have become mainstream in computer-aided diagnosis and are widely applied to classification, ROI segmentation, and medical object detection.
  • The survey examines deep learning applications across classification, segmentation, detection, and registration, with detailed attention to object detection and chest X-ray, mammogram, CT, and MRI images.
  • The survey emphasizes recent unsupervised and semi-supervised deep learning advances and discusses attention mechanisms, domain knowledge, and uncertainty estimation as performance-enhancement strategies.

2. OVERVIEW OF DEEP LEARNING METHODS

Deep learning methods are broadly categorized as supervised, unsupervised, or semi-supervised according to whether training data have labels. Supervised learning optimizes models using labeled image–label pairs to predict class labels for test images.

  • 2. OVERVIEW OF DEEP LEARNING METHODS: Deep learning is roughly divided into supervised, unsupervised, and semi-supervised learning based on training-data labels.The passage distinguishes these categories by whether labels are present.
  • 2. OVERVIEW OF DEEP LEARNING METHODS: Supervised learning uses labeled training images and optimizes the model with image–label pairs.The model is trained from paired images and labels.
  • 2. OVERVIEW OF DEEP LEARNING METHODS: For each testing image, the optimized supervised model generates a likelihood score to predict its class label.The likelihood score supports class-label prediction for test images.

2.1. Supervised learning

Supervised medical image analysis commonly uses convolutional neural networks (CNNs), which transform input images through convolutional, pooling, and fully connected layers to produce class-based likelihoods. Convolution extracts features, pooling reduces feature-map dimensions and parameters, and sigmoid or softmax functions produce class-probability distributions.

  • Supervised learning: CNNs are widely used in medical image analysis and transform input images through convolutional, pooling, and fully connected layers into class-based likelihoods.The paper illustrates this architecture for disease classification from MRI images.
  • Supervised learning: Convolutional layers use kernels and biases to extract features, apply nonlinear activation functions, and pass resulting feature maps to subsequent layers.At each layer, kernels extract features from the input image, biases are added, and an activation function is applied.
  • Supervised learning: Pooling layers reduce feature-map dimensions and parameter counts, with average pooling and maximum pooling as common operations.The process is repeated across the network’s layers after convolution.
  • Supervised learning: Fully connected layers usually produce class-probability distributions through sigmoid or softmax functions.These layers are placed at the end of the network.

2.2. Unsupervised learning

Unsupervised learning methods learn representations or generate data without requiring externally provided labels. The section covers autoencoders and generative adversarial networks, alongside self-supervised pretext and contrastive learning, while noting the emerging role of self-supervised pre-training in medical image analysis.

  • Autoencoders: Autoencoders learn latent feature representations by minimizing reconstruction loss, although shallow architectures have limited representation power.The latent representation is learned by reconstructing the input from a hidden representation.
  • Variational autoencoders: VAEs probabilistically map observed data to lower-dimensional latent variables by maximizing sample likelihood with reconstruction and prior-divergence terms.The divergence term measures the difference between the approximated posterior and a Gaussian prior.
  • Generative adversarial networks: GANs learn generative models through a minimax game between a generator producing synthetic samples and a discriminator distinguishing them from training data.WGAN replaces the vanilla GAN’s Jensen–Shannon divergence with Earth-Mover distance, while cGANs condition generation and discrimination on additional information such as class labels.
  • Self-supervised learning: Self-supervised learning creates inputs and labels from unlabeled data itself, reducing reliance on expensive and time-consuming supervised tasks.The section describes pretext tasks that predict hidden image information and contrastive methods that maximize similarity for positive pairs while minimizing it for negative pairs.
  • Self-supervised learning: Although recent self-supervised contrastive methods such as MoCo and SimCLR lacked established medical imaging applications at the time of writing, self-supervised pre-training was identified as a promising alternative to supervised pre-training.The authors anticipated rapid growth in studies applying these methods to medical image analysis.

2.3. Semi-supervised learning

Semi-supervised learning combines limited labeled data with large-scale unlabeled data when the two are relevant, using the unlabeled data to compensate for limited annotation. The reviewed approaches include perturbation-consistency methods, pseudo-labeling, and semi-supervised generative models such as GANs and VAEs.

  • Semi-supervised learning: SSL combines labeled and unlabeled data during training, targeting settings with limited labeled data and large-scale unlabeled data.The two data types should be relevant so unlabeled information can compensate for limited labeled data.
  • Consistency-based methods: Perturbation-consistency methods require predictions for unlabeled examples to remain stable after noise or data augmentation.Their loss generally contains labeled- and unlabeled-data components.
  • Consistency-based methods: UDA and MixMatch have recently raised the performance of semi-supervised learning to a new level.The passage identifies UDA and MixMatch as recent examples of this progress.
  • Pseudo-labeling: Pseudo-labeling uses model-generated annotations for unlabeled examples alongside labeled examples, iteratively improving pseudo-label quality and model performance.Combining naïve pseudo-labeling with Mixup augmentation can further improve performance.
  • Generative models: Semi-supervised generative models such as GANs and VAEs focus on target tasks rather than only generating high-fidelity samples.A simple GAN adaptation modifies the discriminator to perform additional tasks, including image classification.

2.4. Strategies for performance enhancement

Performance enhancement strategies include attention mechanisms that focus models on relevant image regions and uncertainty estimation that improves trustworthiness. Medical imaging models also require adaptation because natural-image architectures may perform suboptimally on medical images with subtle class differences.

  • Attention mechanisms: Attention mechanisms adaptively learn what and where to attend, conditioning predictions on the most relevant image regions.The Transformer is a prominent architecture relying solely on self-attention to capture global dependencies without sequential computation.
  • Medical-image adaptation: Natural-image deep learning models may perform suboptimally on medical images because medical and natural images differ substantially.Medical images often exhibit high inter-class similarity, requiring fine-grained visual features to distinguish subtle differences important for correct prediction.
  • Uncertainty estimation: Clinical reliability requires quantifying uncertainty because data noise and inference errors can affect model predictions.Bayesian approximation and model ensembles are identified as common techniques for uncertainty estimation.

3. DEEP LEARNING APPLICATIONS

Deep learning applications in medical imaging have achieved major success in classification and computer-aided diagnosis, but their performance remains constrained by the availability and quality of annotated training data. End-to-end architectures such as AlexNet, VGG, GoogleLeNet, ResNet, and DenseNet have helped make deep learning mainstream in high-performing CADx and other medical image analysis tasks.

  • Classification: Medical image classification supports CADx by distinguishing malignant from benign lesions or identifying diseases from input images.Deep learning-based CADx schemes have achieved substantial success over the last decade.
  • Classification: AlexNet, VGG, GoogleLeNet, ResNet, and DenseNet exemplify increasingly deep end-to-end classification models with larger representation capacity.These architectures have produced superior results and contributed to deep learning becoming mainstream in CADx and other subfields.
  • Data limitations: Deep learning performance depends strongly on training-dataset size and annotation quality, especially in medical image analysis tasks involving 3D data.Data acquisition and annotation difficulties can prevent the creation of sufficiently large, high-quality training datasets.

I. Unsupervised image synthesis

Unsupervised image synthesis uses GANs to learn hidden data distributions and generate realistic medical images for augmentation. Studies applied DCGANs and conditional GANs to address limited datasets and the lack of positive cancer cases.

  • I. Unsupervised image synthesis: GANs provide a more complicated medical-data augmentation approach by learning hidden data distributions and generating realistic images.Classic augmentation is simple and effective but cannot bring much new information to existing training examples.
  • I. Unsupervised image synthesis: DCGAN synthesized high-quality examples to improve liver lesion classification using a dataset of 182 lesions.Classic augmentation expanded the dataset to nearly 90,000 examples before GAN training.
  • I. Unsupervised image synthesis: Conditional GANs generated realistic mammogram lesions to alleviate data scarcity, especially the lack of positive cancer cases.Conditioning on malignant/non-malignant labels controlled lesion generation, including synthesizing malignant lesions onto non-malignant patch images.

II. Self-supervised learning based classification

Self-supervised classification addresses limited annotation by pre-training on abundant unlabeled medical images before supervised fine-tuning. Recent approaches use pretext or contrastive tasks, with reported gains over ImageNet pre-training in chest X-ray and dermatology classification.

  • Self-supervised learning based classification: Self-supervised learning suits settings with many medical images but only a small percentage labeled, using self-supervised pre-training followed by supervised training.This two-step optimization aims to improve medical-task performance when annotations are insufficient.
  • Self-supervised learning based classification: SimCLR-based contrastive pre-training outperformed ImageNet-pre-trained counterparts by 1.1% in mean AUC for chest X-ray classification and 6.7% in top-1 accuracy for dermatology condition classification.The models used wider ResNet-50 and ResNet-152 architectures and were fine-tuned with fewer labeled dermatology images and chest X-rays.
  • Self-supervised learning based classification: MoCo-pretrained feature representations on large X-ray databases outperformed ImageNet pre-training on downstream tasks with small labeled X-ray datasets and generalized well to an external dataset.The approach used contrastive learning to learn representations from unlabeled data before downstream classification.
  • Self-supervised learning based classification: Context restoration was introduced as a self-supervised pretext task to learn high-quality features from unlabeled medical images.The approach was motivated by marginal improvements from relative position and local context prediction on medical image datasets.

I. U-Net and its variants

U-Net uses encoder–decoder architecture with skip connections to combine high-level semantic features and low-level localization details for precise image segmentation. Its variants modify skip connections, dimensionality, convolutional units, and training strategies for biomedical and volumetric segmentation.

  • Core U-Net design: U-Net combines high-level semantic features with low-level fine-grained details through skip connections to support precise pixel-level image segmentation.Higher layers capture whole-image semantics, while lower layers preserve localization details.
  • Skip-connection variants: Adding short skip connections helps train very deep segmentation networks, while redesigned connections reduce semantic gaps between fused encoder and decoder feature maps.Plain U-Net skip connections may fuse semantically dissimilar feature maps.
  • 3D and residual variants: 3D U-Net replaces 2D operations with 3D counterparts for volumetric segmentation using sparsely annotated images, while V-Net targets 3D MRI prostate-volume segmentation with residual units.V-Net’s major architectural distinction is replacing forward convolutional units with residual convolutional units.
  • Recurrent and residual variants: RU-Net replaces U-Net’s forward convolutional units with recurrent convolutional layers, and R2U-Net integrates recurrent and residual designs to accumulate useful features for segmentation.The recurrent and residual components derive from RCNN and ResNet architectural advantages.
  • Training and inference enhancements: U-Net-based segmentation methods also incorporate adversarial training, attention mechanisms, and uncertainty estimation alongside architectural and skip-connection modifications.These strategies extend U-Net beyond its basic encoder–decoder design.

II. Transformers for segmentation

Transformer-based medical image segmentation models combine self-attention with convolutional or UNet-like architectures to capture global relations while preserving spatial information. Recent approaches extend these designs to parallel CNN–Transformer branches, volumetric segmentation, computationally efficient attention, and pure Transformer architectures, with most models using external pre-training.

  • Hybrid CNN–Transformer models: Hybrid models use CNN feature maps, patch and positional embeddings, Transformer layers, and decoder upsampling to combine spatial localization with global relations.Skip connections retain spatial localization information, while multi-head self-attention and multilayer perceptrons process embedded feature sequences.
  • Hybrid CNN–Transformer models: TransFuse combines CNN and Transformer branches in parallel, with the Transformer branch processing embedded image patches to capture global context.This differs from sequential designs that first extract low-level CNN features and then pass them through Transformer layers.
  • 3D segmentation: Hybrid Transformer–UNet approaches also support 3D volumetric segmentation of MRI brain tumors, CT spleens, and multi-organ segmentation.Three-dimensional images are split into volumes before linear and positional embeddings are applied to the input volume sequence.
  • Efficient attention: CoTr replaces vanilla multi-head self-attention with deformable self-attention that attends to a small set of key positions, reducing computational complexity.CoTr follows the CNN-to-Transformer design in which CNN-generated feature maps provide Transformer inputs.
  • Transformer-only segmentation: Swin-Unet presents a pure Transformer UNet-like architecture with symmetric encoder–decoder structure, Swin Transformer blocks, and patch merging or expanding layers.Its shifted-window scheme supports the modeling capacity described for the Swin Transformer block without convolutional operations.
  • Training strategies: Most Transformer-based segmentation models are pre-trained on large external datasets, although efficient attention modules and new training strategies can enable good results without pre-training.The training strategies integrate high-level information and finer details.

III. Mask R-CNN for segmentation

Mask R-CNN has achieved good results in medical image segmentation, with volumetric attention reducing false positives and Mask R-CNN++ producing a large performance boost. Because annotated data are limited, recent approaches increasingly use unlabeled images through self-supervised pre-training and semi-supervised methods.

  • Mask R-CNN for segmentation: Mask R-CNN has achieved good results in medical image segmentation and is closely related to Faster R-CNN object-detection architectures.It was originally developed for pixelwise instance segmentation.
  • Mask R-CNN for segmentation: A volumetric attention module uses contextual relations along the z direction of 3D CT volumes and enables Mask R-CNN to achieve lower false positives in segmentation.Feature pyramids are extracted from target and neighboring groups of three CT slices before concatenation.
  • Mask R-CNN for segmentation: Replacing Mask R-CNN’s plain FPN skip connections with UNet++ nested and dense skip connections produced Mask R-CNN++ and a large performance boost.The redesign leverages the stronger segmentation performance of UNet++ connections.
  • Self-supervised and semi-supervised segmentation: Self-supervised pretext tasks and contrastive learning learn representations from unlabeled datasets, improving downstream segmentation accuracy and efficiency when annotated examples are limited.Taleb et al. extended pretext tasks from 2D to 3D and evaluated rotation prediction, jigsaw puzzles, and relative patch location for brain tumor segmentation using about 22,000 MRI scans.
  • Self-supervised and semi-supervised segmentation: Semi-supervised segmentation methods include uncertainty-aware mean teachers, pseudo labeling, and generative models that exploit unlabeled medical images alongside limited labeled data.A VAE-based framework for optic-cup segmentation outperformed counterparts, including fully supervised CNNs, across different labeled/unlabeled data splits.

I. Overview of the detection frameworks

Detection frameworks evolved from slow, multi-stage region-proposal pipelines toward simpler one-stage architectures. YOLO directly predicts classes and bounding boxes with a single network, improving speed and simplicity but initially sacrificing localization accuracy and detection performance relative to Faster RCNN.

  • Two-stage detectors: RCNN uses a multi-stage pipeline whose separate feature extraction for each region proposal makes training and testing slow, expensive, and difficult to optimize.The framework’s drawbacks include disk-space and time costs during training, as well as slower testing.
  • One-stage detectors: YOLO reformulates object detection as regression, using one network to directly predict object classes and bounding-box coordinates without a separate region-proposal network.Unlike region-proposal-based frameworks, YOLO learns features globally from the entire image rather than from local regions.
  • One-stage detectors: YOLO is faster and simpler than Faster RCNN but has more localization errors and lower detection accuracy.YOLOv2 and YOLO9000 later incorporated batch normalization, improved anchor boxes, fine-grained features, and multi-scale training to improve performance.

II. Specific-type medical object (e.g., lesion) detection

Medical lesion detection is challenging because lesions may be extremely small, limiting the effectiveness of general object-detection frameworks. Recent approaches address this by incorporating medical domain characteristics, semi-supervised learning, and uncertainty estimation.

  • General object-detection frameworks may perform poorly for medical lesions because lesions can be extremely small.
  • Radiology: Radiology detection methods exploit 3D spatial context and correlations across consecutive CT slices to improve lung-nodule detection.
  • Histopathology: Histopathology detection methods incorporate lymphocyte size and overlap priors and sample hard negatives in whole-slide images.
  • Semi-supervised detection: Semi-supervised methods adapt focal loss and MixMatch for soft labels and 3D medical image detection.
  • Uncertainty estimation: Uncertainty estimation has been explored to improve detection of small multiple-sclerosis lesions, whose uncertainties mainly arise from lesion size and boundaries.

III. Universal lesion detection

Universal lesion detection aims to identify and localize diverse lesion types across the whole body, but supervised detectors require predefined lesion categories and cannot detect lesions absent from training data. Recent work therefore includes comprehensive datasets, multi-branch and attention-based detectors, and unsupervised anomaly-detection approaches that model normal anatomy to identify deviations.

  • Supervised universal detection: DeepLesion provides a comprehensive dataset of 32K lesions spanning diverse types, including lung nodules, liver tumors, abdominal masses, and pelvic masses.
  • Supervised universal detection: MULAN extends Mask R-CNN with detection, tagging, and segmentation branches, predicting lesion status, bounding boxes, 185 lesion-related tags, and binary masks.
  • Supervised universal detection: Attention-based universal detection incorporates 3D context and spatial information into R-FCN architectures to improve lesion detection.
  • Limitations of supervision: Supervised one- and two-stage detectors require predefined lesion types and cannot detect lesions not represented in their training datasets.
  • Unsupervised anomaly detection: Unsupervised anomaly detection models normal image distributions so abnormal regions such as tumors can be detected as deviations, commonly using VAE- or GAN-derived models.
  • Unsupervised anomaly detection: Restoration-based anomaly detection reconstructs an optimal latent representation or a healthy counterpart of an anomalous image, enabling anomaly localization through reconstruction or dissimilarity maps.

4. DISCUSSIONS · 4.1. Toward better combinations of deep learning and medical image analysis

Improving deep learning for medical image analysis requires task-specific solutions, learning paradigms suited to limited annotation, more automated architectures and pipelines, and appropriate use of medical domain knowledge. Key challenges include differences from natural images, class imbalance, unreliable registration ground truth, weak task connections in GANs, and distribution mismatch in semi-supervised learning.

  • 4.1.1. On the task-specific perspective: Medical images differ from natural images, so computer-vision methods cannot always be directly applied effectively; task-specific challenges must be addressed.For classification, success depends on extracting highly discriminative features.
  • 4.1.1. On the task-specific perspective: Class imbalance and small targets complicate medical segmentation, while region-based metrics may omit diagnostically important structures, shapes, and contours.Context, attention, multi-scale features, anatomical cues, and Transformers are identified as useful strategies for improving segmentation accuracy.
  • 4.1.1. On the task-specific perspective: Medical image registration uniquely requires pixel-wise or voxel-wise correspondence, but reliable ground-truth registrations are difficult to obtain, making unsupervised methods promising.Ground truth may be synthetically generated or produced by conventional registration algorithms.
  • 4.1.2. On the perspective of different learning paradigms: Large annotated datasets remain a major bottleneck, while supervised transfer learning can alleviate this limitation by initializing target-task models with pretrained weights.Pretraining may use relevant or irrelevant datasets such as ImageNet.
  • 4.1.2. On the perspective of different learning paradigms: GANs show promise for synthesis and semi-supervised learning, but their generators must connect strongly to target tasks to improve performance beyond conventional augmentation.Differentiable augmentation and adaptive discriminator augmentation can generate high-fidelity images under data-limited conditions, but had not been applied to medical image analysis in the passage.
  • 4.1.2. On the perspective of different learning paradigms: Contrastive learning appears more promising than directly transferring computer-vision pretext tasks because radiological images require robust representations and novel pretext-task design is difficult.Self-supervision can be constructed through pretext tasks or contrastive learning.
  • 4.1.2. On the perspective of different learning paradigms: Semi-supervised learning can degrade when labeled and unlabeled medical images come from different distributions, violating its underlying assumption.Appropriate augmentation policies should be developed in dataset-driven or task-driven ways to support semi-supervised applications.
  • 4.1.3. Finding better architectures and pipelines: Progress is closely related to architectural advances, motivating further exploration of neural architecture search and automated pipelines that configure model components.Most existing NAS studies are confined to image classification, and fundamental revolutionary models have not emerged from this process.

4.2. Toward large-scale applications of deep learning in clinical settings

Deep learning has shown strong promise in medical image analysis but has not yet achieved the expected clinical impact, with methodological flaws, biases, and technical pitfalls limiting clinical applicability. Progress toward large-scale clinical use requires better data, clinically meaningful evaluation, clinician collaboration, and reproducible research.

  • Deep learning has not made the expected significant impact in clinical practice, as illustrated by its limited contribution during the early stages of the COVID-19 pandemic.
  • 62 studies selected from 415 were reviewed, and none of the identified deep learning or machine learning models were clinically applicable to COVID-19 diagnosis or prognosis.
  • High-risk bias recurs across medical image analysis tasks and applications, severely limiting deep learning’s potential in clinical radiology.
  • Medical image datasets are increasingly larger, usually containing at least several hundred images, while MICCAI challenge datasets have promoted progress but require caution about potential biases.
  • Accuracy, AUC, and Dice coefficient quantify technical performance but often fail to reflect clinical applicability, which depends on beneficial changes in patient care.
  • Deep learning used as a second opinion could correctly identify many breast cancer cases missed by radiologists in mammogram analysis.
  • Despite promising applications in classification, segmentation, detection, and registration, technical challenges and pitfalls still impede scientifically rigorous clinical CAD schemes.
Loading 2105.13381v3…