Recent advances and clinical applications of deep learning in medical image analysis

• Clinical image interpretation remains vulnerable to expertise differences, reader variability, fatigue, and limited medical resources, motivating computer-aided detection and diagnosis.CAD aims to help clinicians interpret images more efficiently, accurately, and objectively.Source: §1 · p. 1
• Deep learning supports classification, segmentation, detection, and registration by hierarchically learning relevant image features without manual feature engineering.The approach progressively identifies, amplifies, and assembles task-relevant patterns while filtering irrelevant variation.Source: §1 · p. 2
• Supervised deep learning is hindered by medical datasets that are often small, with fewer than 10,000 images and limited expert annotations.Unsupervised and semi-supervised methods can expand labeled data, learn from unlabeled images, and generate pseudo-labels.Source: §1 · p. 2
• The survey emphasizes self-supervised, unsupervised, and semi-supervised learning while retaining supervised methods across classification, segmentation, detection, and registration.It focuses especially on chest X-ray, mammogram, CT, and MRI applications, with detailed coverage of medical object detection.Source: §1 · pp. 2–3
• The review is technically oriented toward state-of-the-art architectures and recent not-so-supervised methods, including Transformers for medical image segmentation and emerging self-supervised learning.It also addresses applications beyond radiology when methods may transfer to radiological or MRI images.Source: §1 · p. 3
• The survey covers recent deep learning advances, attention mechanisms, domain knowledge, uncertainty estimation, task applications, and challenges for large-scale deployment.Its structure includes methods in Section 2, four application tasks in Section 3, and challenges and future directions in Section 4.Source: §1 · p. 3

Learning paradigms

• Supervised learning optimizes models using fully labeled image-label pairs, whereas unsupervised learning discovers hidden data structures without labels and semi-supervised learning combines limited labels with unlabeled-data features.Semi-supervised models learn input-output relationships from labeled data while strengthening semantic and fine-grained feature learning from unlabeled data.Source: §2 · p. 5
• The section primarily reviews unsupervised and semi-supervised methods because they support medical image tasks when annotated data are limited.Popular frameworks for both learning paradigms are introduced after a brief discussion of supervised learning.Source: §2 · p. 5

General strategies

• Attention mechanisms, domain knowledge, and uncertainty estimation are presented as general strategies that can be combined with different learning paradigms to improve medical image analysis.Source: §2 · p. 5

Supervised learning

• CNNs process medical images through convolutional, pooling, and fully connected layers before outputting class-based likelihoods (Anwar et al., 2018).Figure 2 illustrates this architecture for MRI disease classification.Source: §2.1 · p. 5, Figure 2 · p. 5
• Convolutional layers extract feature maps with learned kernels and biases, followed by nonlinear activation functions.The resulting feature maps are passed to subsequent layers.Source: §2.1 · p. 5
• Pooling layers reduce feature-map dimensions and parameter counts using average- or maximum-pooling operations.Fully connected layers then produce class probabilities through sigmoid or softmax functions.Source: §2.1 · p. 5
• Network parameters are iteratively optimized by minimizing a loss between the predicted probability distribution and the real labels.The predicted distribution assigns a label to each input instance.Source: §2.1 · p. 5

2.2.1. Autoencoders

• Autoencoders learn latent representations by reconstructing inputs, while deeper stacked and regularized variants improve nonlinear modeling and robustness to perturbations (Bengio et al., 2007; Ranzato et al., 2007; Vincent et al., 2010; Rifai et al., 2011).Variational autoencoders additionally approximate an intractable posterior using an encoder and optimize a tractable evidence lower bound combining reconstruction and KL-divergence terms (Kingma and Welling, 2013).Source: §2.2.1 · p. 6

2.2.2. Generative adversarial networks (GANs)

• GANs learn generative models through a minimax game in which a generator produces samples and a discriminator distinguishes them from training data (Goodfellow et al., 2014).WGAN replaces Jensen–Shannon divergence with Wasserstein distance to provide useful gradients, improve training stability, and alleviate mode collapse (Arjovsky et al., 2017).Source: §2.2.2 · pp. 6–7

2.2.3. Self-supervised learning

• Self-supervised learning creates inputs and labels from unlabeled data, reducing expensive annotation demands while potentially improving downstream supervised-task performance, efficiency, and real-world generalization.Its main approaches are pretext tasks and contrastive learning; pretext representations depend heavily on hand-crafted task quality, whereas contrastive learning separates similar and dissimilar pairs.Source: §2.2.3 · pp. 7–8
• MoCo and SimCLR learn contrastive representations from augmented image pairs, but broad medical-image applications had not yet been established when this review was written.MoCo uses a queued dictionary and momentum encoder, while SimCLR adds a projection head after augmentation and encoding; the authors anticipate rapid expansion of medical applications.Source: §2.2.3 · pp. 8–9

Overview

• Deep semi-supervised learning generally outperforms high-quality supervised baselines across varying labeled and unlabeled data amounts, supporting medical image analysis with lower annotation costs (Ouali et al., 2020).The review notes that unlabeled data can sometimes hurt performance, although recent deep-learning studies increasingly report consistent gains.Source: §2.3 · p. 9

Consistency regularization

• Consistency regularization enforces stable predictions under perturbations by combining an unlabeled consistency loss with a labeled cross-entropy loss; Mean Teacher uses an EMA teacher to guide the student.Ladder Networks, the Π-Model, Temporal Ensembling, and Mean Teacher exemplify this category; UDA and MixMatch further improved SSL performance (Xie et al., 2020; Berthelot et al., 2019).Source: §2.3 · pp. 9–10, Figure 4 · p. 10

Pseudo labeling

• Pseudo-labeling generates annotations for unlabeled examples and iteratively trains with labeled data, with Mixup and multi-view co-training providing extensions.Co-training learns separate classifiers for different views and maximizes agreement among their pseudo annotations.Source: §2.3 · p. 10

Generative models

• Semi-supervised GANs and VAEs focus on target tasks, while Triple-GAN adds an independent classifier because one discriminator may not optimize classification and generation simultaneously.In semi-supervised GANs, the discriminator estimates realness for unlabeled images and additionally predicts class for labeled images.Source: §2.3 · p. 10

2.4.1. Attention mechanisms

• Attention mechanisms adaptively identify relevant image regions and features, with soft attention favored because hard attention is nondifferentiable.Developed variants include spatial, channel, combined spatial-channel, and self-attention mechanisms (Jaderberg et al., 2015; Hu et al., 2018a; Wang et al., 2017; Woo et al., 2018; Wang et al., 2018).Source: §2.4.1 · pp. 10–11

2.4.2. Domain knowledge

• Medical models require customization because medical images have high inter-class similarity and datasets are smaller than benchmark natural-image datasets.Integrating anatomical information, 3D spatial context, patient-level multi-instance data, metadata, radiomic features, and accompanying text can improve representations and reduce model complexity (Zhou et al., 2021; Zhou et al., 2019a; Zhang et al., 2017; Zhuang et al., 2019; Zhu et al., 2020a; Azizi et al., 2021; Vu et al., 2021; Shorfuzzaman and Hossain, 2021; Zhang et al., 2020a).Source: §2.4.2 · p. 11

2.4.3. Uncertainty estimation

• Uncertainty estimation improves trustworthiness by quantifying prediction uncertainty caused by data noise and inference errors in high-safety clinical settings.Common approaches include Bayesian approximation, such as Monte Carlo dropout, and model ensembles (Gal and Ghahramani, 2016; Lakshminarayanan et al., 2017).Source: §2.4.3 · p. 11

3.1. Classification

• Medical image classification supports CADx by distinguishing malignant from benign lesions or identifying diseases, but its performance depends strongly on sufficient training data and high-quality annotations. (Shen et al., 2017; van Ginneken et al., 2011)Beyond transfer learning, unsupervised synthesis, self-supervised learning, and semi-supervised learning show potential for improving classification with limited annotated data.Source: §3.1 · p. 12

3.1.1. Supervised classification

• Deep learning classification models evolved from AlexNet to VGG, GoogleLeNet, ResNet, and DenseNet, yielding superior results and making deep learning mainstream in CADx and medical image processing.(Krizhevsky et al., 2012; Simonyan and Zisserman, 2014; Szegedy et al., 2015; He et al., 2016; Huang et al., 2017)Source: §3.1.1 · p. 12
• Transfer learning addresses limited medical-image annotations by pre-training models on large natural or medical-image datasets and fine-tuning them with fewer target-domain examples, becoming a cornerstone across CT, MRI, mammography, and X-ray classification.Tajbakhsh et al. (2016) showed that adequately fine-tuned pre-trained CNNs performed at least as well as CNNs trained from scratch. (Tajbakhsh et al., 2016; Donahue et al., 2014; de Bruijne, 2016; Shin et al., 2016; Yuan et al., 2019; Huynh et al., 2016; Minaee et al., 2020)Source: §3.1.1 · p. 12
• Attention-guided classification improves performance and interpretability by suppressing irrelevant areas, highlighting discriminative local regions, and fusing global and local image information. (Zhou et al., 2019b; Guan et al., 2018)Attention modules were also applied to VGG- and U-Net-variant networks, focusing on local structures containing salient features for fetal ultrasound plane classification and 3D CT pancreas segmentation. (Schlemper et al., 2019)Source: §3.1.1 · p. 12

I. Unsupervised image synthesis

• GANs address the limited novelty of rotation, scale, flip, and translation by learning hidden data distributions and generating realistic medical images.Classic augmentation creates more instances but cannot introduce much new information into existing examples.Source: §I · pp. 12–13
• Sensitivity and specificity for liver-lesion classification increased from 78.6% and 88.4% to 85.7% and 92.4%, respectively, after DCGAN-based synthetic augmentation on a limited dataset.The dataset contained 182 lesions, and classic augmentation expanded training examples to nearly 90,000 before GAN training.Source: §I · p. 13
• ACGAN-based lesion augmentation produced weaker classification performance than the unconditional DCGAN counterpart.ACGAN conditioned generation on lesion classes and also predicted classes through its discriminator.Source: §I · p. 13
• Conditional GAN augmentation for mammogram classification improved performance over traditional augmentation by less than 1%.The cGAN controlled malignant or non-malignant lesion synthesis using labels and segmentation masks to address the scarcity of positive cancer cases.Source: §I · p. 13

II. Self-supervised learning based classification

• 1.1% higher mean AUC for chest X-rays and 6.7% higher top-1 accuracy for dermatology classification were achieved over ImageNet pre-training using SimCLR-based self-supervision.With only 1% labeled data, metadata-guided MoCo pre-training further improved mean AUC by 3.4% over prior contrastive learning and 14.4% over ImageNet pre-training for pleural effusion classification.Source: §II · pp. 13–14

3.1.3. Semi-supervised learning

• Semi-supervised classification integrates labeled and unlabeled images through adversarial or consistency-based training, with semi-supervised GANs outperforming supervised CNNs as labeled examples increase.Applications include cardiac disease, lung nodule, and left ventricular hypertrophy classification, alongside Π-Model and Mean Teacher approaches.Source: §3.1.3 · p. 14

3.2. Segmentation

• Segmentation requires especially strong supervision because it identifies lesion, organ, and substructure pixels or voxels, motivating U-Net variants, Transformer combinations, and self- or semi-supervised methods.U-Net has been a dominant architecture since 2015, while combining U-Net with NLP Transformers has recently achieved state-of-the-art performance.Source: §3.2 · p. 15

U-Net foundations

• U-Net uses skip connections to concatenate coarse semantic and fine-grained features, recover input resolution, and generate pixel-class segmentation maps.The architecture takes 2D images as input and produces multiple segmentation maps corresponding to pixel classes.Source: §I · p. 16

Skip-connection variants

• UNet++ replaces plain encoder–decoder skips with nested dense connections to reduce semantic gaps and outperformed U-Net and wide U-Net across four medical segmentation tasks.Short skip connections were also reported as important for training very deep segmentation networks.Source: §I · p. 16

3D and residual architectures

• 3D U-Net extends operations to volumetric segmentation, while V-Net uses residual units and a Dice-based loss for imbalanced prostate MRI segmentation with sparse annotations.Training data were augmented using random nonlinear transformations and histogram matching to address annotated-volume scarcity.Source: §I · p. 16

Dense, recurrent, and residual units

• Dense V-Net supports multiorgan abdominal CT segmentation through dense feature stacks and achieved significantly higher Dice scores for all organs than multiatlas label fusion methods.RU-Net and R2U-Net further replaced forward units with recurrent and recurrent-residual units, respectively, and both outperformed U-Net and residual U-Net.Source: §I · pp. 16–17

Auxiliary learning mechanisms

• U-Net variants also incorporate adversarial training, attention gates, and uncertainty estimation to improve segmentation, suppress irrelevant features, and model annotation ambiguity.Attention U-Net consistently outperformed U-Net in CT pancreas segmentation, while hierarchical probabilistic models estimated multiresolution uncertainties in prostate MR and thoracic CT segmentation.Source: §I · p. 17

Hybrid architectures

• TransUNet combines CNN localization features with Transformer global context through cascaded encoder–decoder processing and achieves superior CT multi-organ segmentation performance over competing methods such as attention U-Net.CNN features are passed through skip connections, while patch and positional embeddings are processed by Transformer layers containing multi-head self-attention and MLP blocks before decoding.Source: §II · pp. 17–18, Figure 6 · p. 18
• Parallel fusion in TransFuse combines multiscale CNN features and Transformer representations to capture local details and global context for prostate MRI segmentation.Its two branches operate at corresponding resolution scales and are selectively fused before generating the final segmentation mask.Source: §II · p. 18

3D segmentation

• Hybrid Transformer designs also support 3D volumetric segmentation, while CoTr reduces computational and spatial complexity with deformable self-attention over selected key positions.The 3D architectures process embedded image volumes or multiscale CNN feature maps and merge Transformer representations with decoder features through skip connections.Source: §II · pp. 18–19

Transformer-only architectures

• Swin-Unet provides a pure Transformer, UNet-like alternative whose shifted-window attention lowers self-attention complexity while supporting multiorgan CT and cardiac MRI segmentation.Its symmetric encoder–decoder uses Swin Transformer blocks, patch merging or expanding layers, and skip connections without convolutional operations.Source: §II · p. 19

Pre-training

• Most Transformer segmentation models use large-dataset pre-training, but efficient attention and training strategies can enable good performance without it, and pre-training did not improve 3D segmentation in reported studies.These findings are associated with Wang et al. (2020a), Valanarasu et al. (2021), Hatamizadeh et al. (2022), and Xie et al. (2021b).Source: §II · p. 19

III. Mask R-CNN for segmentation

• Mask R-CNN combines region proposals, RoIAlign, parallel mask prediction, and FPN-based multiscale features for pixelwise medical image segmentation.FPN merges bottom-up and top-down features at multiple resolutions while making predictions independently at each scale.Source: §III · p. 19
• Mask R-CNN with volumetric attention uses neighboring CT slices and spatial-channel attention to model z-axis context, reducing false positives; Mask R-CNN++ replaces FPN skip connections with UNet++ nested dense connections and yields a large performance boost (Wang et al., 2019b; Zhou et al., 2019c).The volumetric-attention module concatenates feature pyramids from target and neighboring slices before mask prediction.Source: §III · pp. 19–20

3.2.2. Unsupervised learning based segmenting models

• Self-supervised pretraining transfers representations from unlabeled medical images to segmentation, improving accuracy with limited annotations through anatomical, 3D pretext, metadata-assisted, and local contrastive objectives (Bai et al., 2019; Taleb et al., 2020; Hu et al., 2020; Chaitanya et al., 2020).Taleb et al. (2020) pre-trained on about 22,000 unlabeled MRI scans and fine-tuned on 285 labeled scans; 3D pretext tasks outperformed 2D counterparts and sometimes supervised pretraining.Source: §3.2.2 · pp. 20–21

3.2.3. Semi- supervised learning based segmenting models

• Semi-supervised consistency regularization improves segmentation under noisy or limited labels by combining mean-teacher guidance, uncertainty estimation, adaptive updates, and generalized Dice loss (Yu et al., 2019; Wang et al., 2020b).The resulting framework demonstrated robustness to label noise and better pneumonia-lesion segmentation than other state-of-the-art methods.Source: §3.2.3 · p. 21
• Pseudo-labeling iteratively expands training data from 50 labeled to 1600 pseudo-labeled CT images, while Semi-InfNet’s PPD, reverse-attention, and edge-attention modules refine infection localization and boundaries (Fan et al., 2020).Semi-InfNet surpassed UNet++ and other cutting-edge segmentation models by a large margin.Source: §3.2.3 · p. 21
• Semi-supervised generative models learn from unlabeled images through reconstruction and shared representations, and one shared-encoder framework with separate foreground-background reconstruction outperformed fully supervised CNNs across labeled/unlabeled splits (Sedai et al., 2017; Chen et al., 2019c).Other semi-supervised methods incorporate anatomical, atlas, topological, semantic, and shape priors to improve segmentation.Source: §3.2.3 · pp. 21–22
• Semi-supervised segmentation methods improve learning from unlabeled images through teacher–student consistency, uncertainty filtering, pseudo-labeling, reconstruction, attention, and weak metadata supervision.Applications include cardiac, brain, pancreas, thyroid, liver, kidney, prostate, retinal, and COVID-19 lesion segmentation across CT, MRI, ultrasound, and fundus images.Source: §3.2.3 · pp. 23–24
• Self-supervised segmentation extends pretext tasks to 3D spatial context and comprehensively compares five pretext tasks for learning useful volumetric representations.Other approaches use local contrastive losses with volumetric structural similarity and compare contrastive, pretext-task, and other pre-training techniques.Source: §3.2.3 · p. 24

3.3. Detection

• Recent detection work reviews one-stage and two-stage frameworks, including Faster RCNN, YOLO, and RetinaNet, and covers supervised, semi-supervised, and GAN- or VAE-based unsupervised lesion detection.The section addresses specific and universal lesion detection after introducing milestone detection frameworks.Source: §3.3 · p. 25

Two-stage detection frameworks

• Two-stage detectors separate region proposal generation from bounding-box recognition, progressing from RCNN to Faster RCNN and Mask RCNN through shared features, RPNs, and added mask prediction.RCNN is slow and costly because of its multistage processing and separate feature extraction; Faster RCNN uses an RPN, while Mask RCNN adds a binary mask branch and FPN backbone.Source: §I · p. 25

One-stage detection frameworks

• One-stage detectors unify class and box prediction for greater speed and simplicity, but early YOLO showed more localization errors and lower detection accuracy than Faster RCNN.YOLO learns features globally from the entire image, while YOLOv2 and YOLO9000 incorporated batch normalization, improved anchor boxes, fine-grained features, and multiscale training.Source: §I · pp. 25–26
• RetinaNet achieved higher detection accuracy than contemporary state-of-the-art two-stage detectors by using focal loss to reduce easy-background dominance and emphasize hard examples.The class-imbalance problem arose because vast numbers of easy background examples dominated training.Source: §I · p. 26
• CornerNet removes anchor boxes by detecting paired corner key-points, whereas CenterNet adds a center key-point to extract more recognizable regional patterns and suppress inaccurate boxes.CornerNet’s anchor-free formulation avoids positive-negative imbalance, slow training, and extra hyperparameters, but it generates many incorrect boxes without sufficient cropped-region information.Source: §I · p. 26

Domain-specific characteristics

• Domain-specific adaptations exploit 3D context and fine-grained features for pulmonary nodule detection, with one method ranking first on the LUNA16 benchmark.The approach added deconvolution and FPN features, used 3D CNN context, and reduced false positives; 3D CNNs require more computation and 3D annotations.Source: §II · p. 26
• Correlations across consecutive CT slices and merged multi-scale features improved detection of both large and small lung nodules while reducing false positives.A non-local module captured long-range dependencies, while features from three shallow ResNet blocks were merged.Source: §II · p. 27
• In histopathology, prior knowledge and hard-negative sampling improved lymphocyte detection F1 score by 3% with a 4.3X speed-up, but modified YOLO was less robust than U-Net across cancer types and staining.The later YOLO-based method underperformed U-Net-based detection and was least robust to staining variation.Source: §II · p. 27

Semi-supervised methods

• Semi-supervised detection generated pseudo-labeled instances from unlabeled CT scans and mixed them with annotated examples, producing significant gains over supervised pulmonary-nodule baselines.The method adapted MixMatch and Mixup to bounding-box annotations and generalized focal loss to soft labels.Source: §II · p. 27

Uncertainty estimation

• Uncertainty estimation filtered highly uncertain candidates to improve multiple-sclerosis lesion detection, targeting uncertainty from small lesions and lesion boundaries.The method combined predicted variance, Monte Carlo variance, predictive entropy, and mutual information.Source: §II · p. 27

III. Universal lesion detection

• Universal lesion datasets such as DeepLesion contain 32K lesions across diverse types, enabling detection of multiple lesion categories from whole-body imaging.DeepLesion includes lung nodules, liver tumors, abdominal masses, pelvic masses, and other lesion types. (Yan et al., 2018b; Yan et al., 2018c)Source: §III · p. 27
• MULAN significantly surpassed ULDor and 3DCE by jointly performing lesion detection, tagging, and segmentation, while heterogeneous datasets and partial labels further improved detection.MULAN uses three branches: bounding-box detection, prediction of 185 lesion tags, and binary lesion segmentation. (Yan et al., 2019; Yan et al., 2020)Source: §III · pp. 27–28
• Attention-based universal detectors incorporate 3D context, spatially emphasize small-lesion features, combine multiple CT windows, and use lesion-position knowledge to improve performance with fewer slices.Tao et al. (2019) used contextual and spatial attention in an R-FCN framework, while Li et al. (2019) reweighted features from three CT windows with positional priors.Source: §III · p. 28
• Two-stage detectors remain prevalent for performance and robustness, but recent one-stage detectors show good performance and may become preferable for streamlined CADe schemes.Separately generating region proposals may hinder streamlined computer-aided detection, motivating interest in anchor-free one-stage detectors such as CenterNet. (Pisov et al., 2020; Lung et al., 2021; Zhu et al., 2021b)Source: §III · p. 28

3.3.2. Unsupervised lesion detection (non-prespecified type of lesion detection)

• Unsupervised lesion detection requires no annotations or prespecified lesion types and can identify arbitrary abnormalities, but its performance remains below supervised or semi-supervised methods.It can nevertheless provide rough suspicious-area detection and imaging biomarker candidates. (Baur et al., 2021)Source: §3.3.2 · p. 28
• Restoration-based anomaly methods generally outperform reconstruction-based approaches when runtime is not a concern, while GAN- and VAE-based models learn healthy-image distributions for anomaly detection.Reconstruction compares abnormal inputs with healthy-trained reconstructions, whereas restoration estimates either optimal latent representations or healthy counterparts using normative distributions and data consistency.Source: §3.3.2 · pp. 29–30
• AnoGAN introduced GAN-based anomaly detection through iterative latent-space inversion, f-AnoGAN accelerated inversion with an encoder, and AnoVAEGAN combined VAE-GAN modeling with whole-image global context.AnoGAN uses residual and discriminator losses; f-AnoGAN performs inverse mapping in one forward pass; AnoVAEGAN directly processes entire MR images. (Schlegl et al., 2017; Schlegl et al., 2019; Baur et al., 2018)Source: §3.3.2 · pp. 29, 31
• A newer restoration method replaced ELBO gradients with local-Gaussian derivatives, achieving higher accuracy at low false-positive rates while incorporating 3D MRI information.The method uses a GMVAE-based normative distribution, total-variation data consistency, and MAP estimation to restore healthy counterparts of anomalous images. (Chen et al., 2021c)Source: §3.3.2 · pp. 30–31

3.4. Registration

• Deep learning registration is organized into deep iterative, supervised, and unsupervised approaches for rigid or deformable alignment of medical images.Deep iterative methods learn similarity metrics for use with classical optimization, while deformable registration has received particular attention.Source: §3.4 · p. 32

3.4.1. Deep iterative registration

• Deep similarity metrics replace manually defined similarity measures and can outperform mutual information for multimodal registration, including challenging 3D MR–TRUS prostate alignment.Simonovsky et al. (2016) used a 5-layer CNN metric for deformable T1–T2 brain MRI registration, while Haskins et al. (2019) extended deep similarity to 3D MR–TRUS prostate images with large modality differences.Source: §3.4.1 · p. 32

3.4.2 Supervised registration

• Supervised registration directly predicts deformation fields in one step, avoiding slow iterative optimization but requiring ground-truth fields from simulation, annotation, or classical registration.Sokooti et al. (2017) generated approximately 1 million augmented training examples and achieved performance close to conventional B-spline registration; dual supervision further refined accuracy (Fan et al., 2019a).Source: §3.4.2 · pp. 32–33

3.4.3. Unsupervised registration

• Unsupervised registration avoids cumbersome ground-truth fields and limited deformation coverage; VoxelMorph operated orders of magnitude faster than classical SyN while achieving competitive performance.VoxelMorph uses a U-Net and spatial transformer to generate and apply deformation fields, while auxiliary anatomical segmentation improved accuracy (Balakrishnan et al., 2019). DLIR progressively combines affine and deformable stages, and adversarial similarity supports mono-modal and multimodal registration (Fan et al., 2019b).Source: §3.4.3 · pp. 33–34

4.1.1. On the task-specific perspective

• Medical classification and detection require discriminative features, while detection additionally needs multi-scale representations and better handling of small objects and severe class imbalance.Fine-grained visual classification, attention, local/global features, domain knowledge, feature pyramids, and anchor-free detectors are identified as promising directions [Liu et al., 2020b; Lin et al., 2017a; Duan et al., 2019].Source: §4.1.1 · p. 36
• Segmentation challenges include class imbalance and small structures; region-based losses help, but complementary non-region-based metrics are needed to capture diagnostic shapes and contours.Transformers may improve segmentation by modeling intra-slice and interslice long-range dependencies, although their medical applications, especially in 3D, remain early [Li et al., 2020e; Chen et al., 2021b; Valanarasu et al., 2021].Source: §4.1.1 · pp. 36–37
• Registration lacks reliable ground-truth correspondences, motivating unsupervised methods, but simpler end-to-end frameworks are desirable because multi-stage coarse-to-fine designs increase complexity and training difficulty.Unsupervised registration has shown promise despite these ground-truth limitations [de Vos et al., 2019].Source: §4.1.1 · p. 37

4.1.2. On the perspective of different learning paradigms

• Limited annotated data motivates transfer learning, GAN-based augmentation, self-supervision, and semi-supervision, but GANs need stronger task connections and can require thousands of examples to converge.Differentiable and adaptive discriminator augmentation may enable high-fidelity GAN synthesis under data-limited conditions, but these methods had not yet been applied to medical image analysis [Zhao et al., 2020; Karras et al., 2020].Source: §4.1.2 · pp. 37–38
• Contrastive self-supervision is more promising than directly transferred pretext tasks for radiological images, while integrating labeled and unlabeled data could improve feature learning with less unlabeled data.Source: §4.1.2 · p. 38
• Semi-supervised learning needs dataset- or task-driven augmentation and robustness to labeled–unlabeled distribution mismatch, which can otherwise degrade performance below a supervised baseline.Automated augmentation remains largely unexplored in medical imaging, while domain adaptation may help address mismatched distributions (Oliver et al., 2018; Guo et al., 2020).Source: §4.1.2 · pp. 38–39

4.1.3. Finding better architectures and pipelines

• Better architectures may draw on biological mechanisms such as attention and long-range dependencies, while neural architecture search and automatically configurable pipelines could reduce manual design and improve cross-dataset robustness.Existing medical pipelines require choices across preprocessing, architectures, losses, and augmentation, and performance may vary across hospitals; examples include NiftyNet and nnU-Net (Gibson et al., 2018b; Isensee et al., 2021).Source: §4.1.3 · pp. 39–40

4.1.4. Incorporating domain knowledge

• Domain knowledge can reduce time and computation costs, but strong radiological expertise—such as bilateral differences and unilateral correspondences in mammography—remains difficult to incorporate reliably.Potential inputs include anatomy, multi-instance patient data, metadata, radiomic features, and text reports (Zhou et al., 2021; Zhou et al., 2019a; Azizi et al., 2021; Vu et al., 2021; Zhang et al., 2020a).Source: §4.1.4 · p. 40

4.2. Toward large-scale applications of deep learning in clinical settings

• COVID-19 reviews found none of the examined deep learning or machine learning models clinically applicable because methodological flaws, bias, overfitting, and inappropriate evaluation undermined reported performance (Wynants et al., 2020; Roberts et al., 2021).Wynants et al. (2020) reviewed over 200 prediction models from 169 studies, while Roberts et al. (2021) reviewed 62 studies selected from 415.Source: §4.2 · pp. 40–41

4.2.1. Image datasets

• Federated learning can combine scattered, sensitive, multi-institutional datasets without exchanging data, helping produce less biased, more generalizable, and robust clinical algorithms (Li et al., 2020f; Rieke et al., 2020).Using multiple public or private datasets reduces community-wide bias, while federated learning enables collaborative training under privacy and regulatory constraints.Source: §4.2.1 · p. 41

4.2.2. Performance evaluation

• Clinicians should help evaluate algorithms because technical metrics may not reflect clinical applicability, including whether deep learning improves clinician performance or efficiency (Kelly et al., 2019).As a second opinion, a mammography model identified cancers missed by radiologists and reduced second-reader workload while maintaining performance comparable to consensus opinion (McKinney et al., 2020).Source: §4.2.2 · pp. 41–42

4.2.3. Reproducibility

• Reproducible clinical translation requires public code and explicit data-selection reporting, because sample choices can affect model superiority and allow advanced models to underperform simple baselines (Baltatzis et al., 2021).Despite rapid progress across classification, segmentation, detection, and registration, technical pitfalls still impede scientifically rigorous CAD schemes acceptable to clinicians (Roberts et al., 2021).Source: §4.2.3 · p. 42