PaperlayerView exact revision on arXiv

Source-linked AI summary

Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review

Reza Azad, Amirhossein Kazerouni, Moein Heidari, Ehsan Khodapanah Aghdam, Amirali Molaei, Yiwei Jia, Abin Jose, Rijo Roy, Dorit Merhof

arXiv:2301.03505v3cs.CV

TL;DR

Medical imaging research needs a coherent account of rapidly expanding Transformer applications, especially as Transformers model long-range dependencies beyond conventional CNN approaches. This paper systematically reviews the literature across major medical imaging tasks, organizing methods, taxonomies, benchmarks, challenges, and future directions. It reports broad application of Transformer strategies, while highlighting data, computational, and explainability constraints.

  • Problem

    Rapidly expanding Transformer research in medical imaging is difficult to track, while the field needs a comprehensive account of applications, methods, strengths, weaknesses, and benchmarks.

  • Method

    The authors conduct a systematic search and review of Transformer literature across classification, segmentation, reconstruction, synthesis, detection, registration, and clinical report generation.

  • Results

    The review organizes more than 200 papers and analyzes state-of-the-art Transformer approaches across diverse medical imaging tasks.

  • Takeaways & Limitations

    The review provides taxonomies, comparisons, challenges, and future directions for understanding Transformer-based medical imaging research.

  • Takeaways & Limitations

    Transformer medical imaging research remains constrained by scarce data, high computational costs, and numerical-instability challenges for explainability methods.

Abstract

from arXiv · show

The remarkable performance of the Transformer architecture in natural language processing has recently also triggered broad interest in Computer Vision. Among other merits, Transformers are witnessed as capable of learning long-range dependencies and spatial correlations, which is a clear advantage over convolutional neural networks (CNNs), which have been the de facto standard in Computer Vision problems so far. Thus, Transformers have become an integral part of modern medical image analysis. In this review, we provide an encyclopedic review of the applications of Transformers in medical imaging. Specifically, we present a systematic and thorough review of relevant recent Transformer literature for different medical image analysis tasks, including classification, segmentation, detection, registration, synthesis, and clinical report generation. For each of these applications, we investigate the novelty, strengths and weaknesses of the different proposed strategies and develop taxonomies highlighting key properties and contributions. Further, if applicable, we outline current benchmarks on different datasets. Finally, we summarize key challenges and discuss different future research directions. In addition, we have provided cited papers with their corresponding implementations in https://github.com/mindflow-institue/Awesome-Transformer.

1. Introduction

This review motivates Transformer use in medical imaging by emphasizing long-range modeling and surveys applications, methods, taxonomies, challenges, and future directions.

  • Motivation: Transformers can model long-range interactions and have become alternatives to CNNs across computer-vision tasks, motivating their adoption in medical imaging.The review frames this adoption as difficult to track because innovation is rapid.
  • Scope: The review systematically examines Transformer applications across diverse medical imaging tasks and analyzes each task’s research progress and limitations.Covered areas include classification, segmentation, reconstruction, synthesis, detection, registration, and report generation.
  • Contributions: More than 200 papers are organized hierarchically, with comparisons and taxonomies intended to characterize state-of-the-art approaches for each task.The review also highlights strengths, weaknesses, parameters, modalities, and technical design choices where applicable.
  • Future directions: The paper discusses open challenges and identifies new trends, open questions, and future research directions for Transformer-based medical imaging.Its stated aim is to guide researchers and encourage further investigation in the vision community.
  • Review methodology: The authors searched DBLP, Google Scholar, and Arxiv Sanity Preserver using task-oriented queries and selected publications based on novelty, contribution, and significance.The search included peer-reviewed, non-peer-reviewed, conference, workshop, and preprint literature.

2. Background

The background introduces Transformer and Vision Transformer mechanisms, then relates pure and hybrid designs to medical imaging needs, data constraints, robustness, and clinical applications.

  • Transformer fundamentals: Transformers use self-attention to relate positions within a sequence, while Vision Transformers process flattened image patches with positional and class embeddings.A Transformer encoder then produces the classification output from the patch sequence.
  • Self-attention: Self-attention projects an input X into query, key, and value representations using learnable matrices, with scaling and softmax producing normalized attention weights.The matrices W_K, W_Q, and W_V are learnable parameters.
  • Multi-Head Self-Attention: Multi-head self-attention combines attention heads from multiple representation sub-spaces to model complex token relationships more finely than a single head.The original Transformer used h = 8 heads, followed by concatenation and linear mapping.
  • Architectural variants: Vision Transformer designs can be pure or hybrid, with hybrid models modifying CNN structures by placing Transformer modules in components such as encoders, decoders, or skip connections.Pure Transformer models address limitations of convolutional locality, while CNN features can support localization in hybrid designs.
  • Medical relevance: Medical imaging motivates Transformers through the need to reduce reader variation and labor, while data scarcity, privacy, and ownership constrain model development.ViT structures can support shared bodies and task-specific heads for multi-task or decentralized solutions.
  • Reported applications: The background cites evidence that ViTs can be more robust to adversarial attacks than CNNs and describes clinical Transformer applications including Med-PaLM 2 and SurgicalGPT.SurgicalGPT reportedly outperformed unimodality text-generation models in accuracy across several robotic surgical datasets.

3. Medical Image Classification

This section reviews Vision Transformer approaches for medical image classification, organizing them into pure and hybrid architectures while examining their designs, applications, and limitations. It also highlights strategies addressing limited data, computational cost, multimodal fusion, interpretability, and privacy.

  • Architectures: ViT classification architectures commonly comprise a feature-extracting backbone, an information-modeling encoder, and a task-specific classification head.Transformers may be adopted in each module, while hybrid designs combine CNN locality with Transformer long-range dependency modeling.
  • Hybrid Models: Hybrid models can improve multimodal medical image classification by combining CNN low-level feature extraction with Transformer modeling of relationships between patch sequences.TransMed overlays neighboring multimodal slices, partitions them into patches, and feeds patch embeddings through CNN and ViT components.
  • Hybrid Models: 10.1%: TransMed-S enhances average accuracy on the PGT dataset over BoTNet while requiring fewer parameters and FLOP count.The reported comparison identifies BoTNet as the nearest counterpart.
  • Training Strategies: Self-supervised contrastive pre-training can outperform ImageNet-supervised ViT models as model size increases and can be competitive with convolutional ResNets.The review notes that removing position embeddings causes only a slight accuracy decrease in the reported findings.
  • Taxonomy: Vision Transformer classification methods are organized into pure and hybrid architectures according to whether they retain vanilla ViT structure or introduce new variants.The review also records architecture type, modality, organ, pre-training strategy, datasets, metrics, publication year, model size, and contributions.
  • Challenges: Medical classification applications remain constrained by scarce labeled data, privacy concerns, costly annotation, and the computational expense of deploying Transformers in real time.The review discusses generative models and federated learning as approaches to data scarcity, while noting continuing privacy concerns in federated communication.

4. Medical Image Segmentation

Transformer-based medical image segmentation methods are organized around pure, hybrid, and specialized architectures that combine global-context modeling with local-detail recovery. Across reviewed studies, these designs report improved segmentation accuracy, edge prediction, robustness, and efficiency on multiple medical imaging tasks.

  • 4.1. Pure Transformers: Pure Transformer architectures such as Swin-Unet use encoder-decoder designs to learn local and long-range dependencies for medical image segmentation.Swin-Unet is reported to improve edge prediction and generalize across Synapse CT and ACDC MRI datasets.
  • 4.1. Pure Transformers: MISSFormer produces more accurate edge prediction and segmentation than TransUNet and Swin-Unet, while its bridge reduces errors by integrating multi-scale information.The comparison specifically attributes fewer segmentation errors to the bridge component relative to MISSFormer-S.
  • 4.2.2. Transformer: Decoder: Other reviewed strategies include Segtran’s squeezed and expanded attention, TransFuse’s CNN-Transformer fusion, and pre-training methods such as Swin UNETR++ and MAE.These approaches target contextual feature integration, anatomical representation learning, or improved downstream classification and segmentation performance.
  • 4.2. Transformer-based Hybrid Architectures: Hybrid architectures pair CNNs for local features with Transformers for global context, as in TransBTS, TransFuse, TransUNet, and UNETR.TransBTS processes volumetric images in a single pass, while UNETR uses a Transformer encoder connected to a CNN decoder through multi-resolution skip connections.
  • 4.2.1. Transformer: Encoder: TransFuse evaluates parallel CNN and Transformer branches fused by BiFusion across four segmentation tasks and reports strong accuracy with faster inference and fewer parameters for polyp segmentation.TransFuse-L* reaches 45.3 FPS, approximately 12% faster than TransUNet, while other tasks also show superior segmentation performance.
  • 4.2.1. Transformer: Encoder: UNETR reports state-of-the-art performance across all organs on BTCV, especially small organs, and outperforms close methods on brain tumor and spleen segmentation.Qualitative comparisons also show that UNETR captures brain-tumor details well on the MSD dataset.

5. Medical Image Reconstruction

This section reviews Transformer-based reconstruction methods for low-dose, sparse-view, and undersampled imaging, emphasizing long-range context and combinations with convolutional or self-supervised designs. Across reviewed studies, Transformers support competitive reconstruction quality, reduced computational demands, and improved handling of limited or shifted data.

  • Overview: ViT reconstruction studies are categorized into low-dose enhancement, sparse-view reconstruction, undersampled reconstruction, and super-resolution reconstruction.The taxonomy also organizes methods by imaging modality and Transformer placement within each pipeline.
  • Low Dose Enhancement: Eformer combines Sobel edge enhancement, LeWin Transformer blocks, convolutional sampling layers, and residual learning to denoise low-dose CT images.The residual scheme learns the noise representation rather than the denoised image.
  • Discussion and Conclusion: The review covers 17 Transformer-based reconstruction studies, but most use the original Transformer as a plug-and-play module and only a limited number use hierarchical or efficient Transformers.On NIH-AAPM-Mayo low-dose CT benchmarks, Eformer is reported superior to TransCT and CTformer by higher SSIM and lower RMSE.
  • Sparse-View Reconstruction: DuDoTrans reconstructs sparse-view CT using enhanced and raw sinogram streams, a Sinogram Restoration Transformer, a DuDo Consistency layer, and a residual image pathway.Its design models long-range dependencies in sinogram sampling rather than relying only on conventional iterative reconstruction.
  • Undersampled Reconstruction: ViT-based fastMRI reconstruction can match or exceed U-Net accuracy while requiring two times faster inference and fewer memory resources.Pre-training weights improve low-data performance and robustness to anatomy shifts.
  • Low Dose Enhancement: PET/MRI-guided denoising addresses low-count PET degradation by using simultaneous modalities to recover texture detail and accommodate PET–MRI shape and modality disparities.The reported Siamese structure has lower computational burden and a simplified network compared with U-Net and residual U-Net structures.
  • Super-Resolution Reconstruction: T2Net introduces task Transformers for simultaneous MRI reconstruction and super-resolution, while SLATER uses cross-attention Transformers and unsupervised pretraining for self-supervised zero-shot reconstruction.T2Net reports competitive FLOPS with SOTA PSNR, and SLATER reports improved reconstruction performance over CNN and self-attention GAN models.

6. Medical Image Synthesis

This section surveys ViT-based medical image synthesis across intra-modality and inter-modality settings, including missing-slice, missing-contrast, and modality-translation tasks. The reviewed methods combine Transformers with U-Net, CNN, GAN, multiscale, residual, and self-supervised designs to generate diagnostically relevant images from incomplete or low-quality inputs.

  • Overview: Medical image synthesis is organized into intra-modality methods that improve target quality and inter-modality methods that map source images to a different target modality.The taxonomy includes methods for missing MRI or CT slices, missing contrasts, and cross-modality translation.
  • Intra-Modality: PTNet reconstructs realistic T1WI images from T2WI using an end-to-end multi-resolution U-Net-like architecture with Performer-based encoder and decoder components.Its skip connections preserve fine-grained and localization features, while the pyramidal design captures local and global information.
  • Inter-Modality: ResViT unifies varying source-target modality settings in one conditional GAN by combining CNN operators with residual Transformer blocks in the generator bottleneck.Aggregated residual Transformer blocks integrate contextual, local, and hybrid local-contextual features.
  • Intra-Modality: MMT synthesizes unavailable MRI contrasts from existing contrasts by formulating the task as sequence-to-sequence learning.The framework is described as a multi-contrast multiscale Transformer.
  • Inter-Modality: CyTran performs unsupervised translation between contrast and non-contrast CT scans while aligning contrast-enhanced and non-enhanced images.SLMT-Net combines paired and unpaired data with Edge-MAE pretraining and patch-wise loss to generate missing MR modalities.
  • Discussion and Conclusion: The review reports synthesis methods using FID and KID for fundus-to-angiogram translation and highlights self-supervised learning for reducing labeling costs.Architectural type, modality, input size, training setting, datasets, metrics, and year are additionally cataloged for analyzed techniques.

7. Medical Image Detection

Medical image detection methods place Transformers in the backbone, neck, or head, often combining them with CNNs. Reviewed approaches address 2D and emerging 3D detection challenges through hybrid architectures, end-to-end set prediction, context modeling, and self-supervised learning.

  • Detection Transformers: DETR reframes detection as end-to-end set prediction using bipartite matching, while later variants target computational complexity, refinement, attention, denoising, and training efficiency.Deformable DETR addresses DETR’s high computational complexity; Efficient DETR, Conditional DETR, DN-DETR, and DINO introduce further modifications.
  • Medical Applications: Medical detection research remains more developed for 2D data than 3D data, although Spine-Transformer and TR-Net demonstrate adaptations for vertebrae and coronary stenosis.Only very few attempts had adapted Detection Transformers to 3D medical images in the reviewed literature.
  • Taxonomy: Detection approaches are categorized by whether Transformers operate in the backbone, neck, or head, with hybrid strategies also represented.The review organizes methods according to the Transformer’s position in the detection architecture.
  • Backbone Strategies: TR-Net combines 3D convolutional feature extraction with 12 Transformer encoders and softmax classifiers to detect significant stenosis from coronary CT angiography.The CNN extracts local features, while Transformer encoders model relationships among flattened image features before classification.
  • Hybrid Models: Hybrid detectors use CNNs for local information and Transformers for global information, as illustrated by CellCentroidFormer’s cell-centroid and whole-cell-shape modeling.This division of labor combines convolutional local representations with Transformer-based global context.

8. Medical Image Registration

Transformer-based registration methods use self-attention to model global dependencies between moving and fixed images across deformable, rigid, and affine transformations. Reviewed systems combine Transformer representations with convolutional encoders, decoders, patch embeddings, or cross-attention to improve spatial alignment and volumetric registration.

  • Registration Task: Medical image registration aligns images acquired with different poses, times, or modalities by estimating a spatial transformation that maximizes image similarity.The task produces geometrically aligned and spatially corresponding images for medical analysis.
  • Registration Taxonomy: Transformer-based registration is organized into deformable, rigid, and affine categories according to the transformation type.Deformable registration optimizes a transformation model, rigid registration preserves relative point distances, and affine registration adds non-isometric scaling.
  • Deformable Registration: ViT-V-Net performs self-supervised volumetric registration by converting convolutional features into patch embeddings, applying a Transformer encoder, retaining localization through skip connections, and warping the moving image.Its V-Net-style decoder produces the output used by a spatial Transformer for warping.
  • Deformable Registration: Dual Transformer networks model inter- and intra-image dependencies alongside global dependencies, then decode fused features into a diffeomorphic deformation field.The framework uses a CNN-based 3D U-Net encoder and decoder around the dual Transformer modules.
  • Affine and Alternative Designs: Multi-stage Transformer designs downsample and concatenate fixed and moving images to learn affine registration matrices, while other systems use cross-attention backbones or spatial priors.The reviewed approaches also include full Transformer backbones and spatial-prior attention for registration.
  • Assessment: The review reports improved Dice scores and Jacobian determinants over CNNs, but Transformer-based registration systems remain in their infancy and require further research.The cited conclusion links gains to better global-dependency identification while retaining a broad maturity caveat.

9. Medical Report Generation

Transformer-based medical report generation targets clinically accurate descriptions from medical images despite data imbalance, data requirements, and model opacity. The review organizes methods by reinforcement learning, graph, memory, and other mechanisms that support reliable, coherent, and bias-aware report generation.

  • Task and Motivation: Medical report generation converts medical-image content into diagnostic captions or reports, with the practical aim of reducing tedious report-production work.The task is framed as an image-to-text problem for clinical use.
  • Taxonomy: The review groups approaches into reinforcement learning, graph-based, memory-based, and other systems according to their report-generation mechanisms.Other systems include curriculum learning, contrastive learning, and alternate learning.
  • Memory-based Systems: Memory mechanisms support longer and more coherent text generation by sharing diagnostic information and prior knowledge during the generation process.Knowledge graphs similarly provide structured prior information for long medical narratives.
  • Reinforcement Learning-based Systems: RTMIC was the first reported Transformer-based medical report-generation system and used reinforcement learning to mitigate exposure bias in sequence-to-sequence models.It extracts bottom-up visual features with DenseNet before visual encoding and caption generation, and achieved state-of-the-art results on IU X-Ray.
  • Graph-based Systems: KERP constructs an abnormality graph, retrieves templates based on detected abnormalities, and paraphrases template terms into a report.Its Encode, Retrieve, and Paraphrase modules connect visual features, structured abnormalities, and report wording.
  • Limitations and Outlook: Current medical report-generation systems still need further research and progression before robust incorporation into practical settings.The review also discusses methods evaluated on IU Chest X-ray and MIMIC-CXR using NLG metrics.

10. Open Challenges and Future Perspectives

The review identifies explainability, representation, computational cost, registration maturity, generalizability, and infrastructure as open challenges for Transformer-based medical image analysis. Future work emphasizes interpretable models, efficient architectures for large or 3D data, broader learning regimes, and open-source ecosystems.

  • Explainability: Attention maps can support interpretability, but numerical instability in propagation-based XAI and vague token associations remain open problems for medical Vision Transformers.The review presents interpretable Vision Transformers as an ongoing research opportunity.
  • Representation Learning: CNNs capture local context while Transformers capture global dependencies, motivating hybrid and multi-scale representations for clinical and dense-prediction tasks.The review notes that single-scale ViT representations can hinder dense prediction improvements.
  • Computational Complexity: Large Transformer models require substantial training and inference resources, and ViT pretraining can be especially burdensome when medical datasets are small or volumetric.A ViT-L/16 ImageNet pretraining example took approximately 30 days on an 8-core cloud TPUv3.
  • Efficient Transformers: Self-attention has O(n^2) complexity in the token count, creating difficulties for million-pixel images and volumetric data.Proposed remedies include windowing, region-of-interest computation, efficient attention, cross-covariance attention, and spatial token reduction.
  • Deployment: Efficient Transformer design remains necessary for challenging 3D data, large-scale images, and deployment on low-cost or resource-constrained devices.The review calls for balancing computational complexity with practical deployment requirements.
  • Registration: Transformer-based registration can improve global spatial-correlation modeling, but registration systems using these architectures remain in an early stage of development.The review identifies further research effort as necessary for this application area.
  • Data-Driven Predictions: Supervised learning can hinder generalizability because annotation-driven features favor task-specific assumptions over adaptability to unseen settings and datasets.Unsupervised learning is presented as a route toward broader robustness and generalizability.
  • Future Directions: An open-source environment with reusable Transformer libraries and community contributions is proposed to support multiple medical tasks and future development.The proposed ecosystem would allow developers to add tasks and methods while organizations provide computational resources.

11. Discussion and Conclusion

The paper presents an encyclopedic review of Transformer applications across diverse medical imaging tasks, organizing techniques and state-of-the-art approaches through taxonomies and comparison tables. It also outlines future research directions.

  • The review covers Transformers in classification, segmentation, detection, reconstruction, synthesis, registration, and clinical report generation.
  • For each application, it provides taxonomies and high-level abstractions of the core techniques and state-of-the-art approaches.
  • Comparison tables summarize strengths, weaknesses, network parameters, imaging modalities, organs, and evaluation metrics.
  • The review concludes by outlining possible avenues for future research.
Loading 2301.03505v3…