PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformers in Medical Imaging: A Survey

Fahad Shamshad, Salman Khan, Syed Waqas Zamir, Muhammad Haris Khan, Munawar Hayat, Fahad Shahbaz Khan, Huazhu Fu

arXiv:2201.09873v1eess.IVcs.CV

TL;DR

Transformers address the limited long-range context of CNNs in medical imaging, motivating a synthesis of this rapidly expanding field. The paper surveys Transformer applications across medical imaging tasks, architectures, challenges, and trends, finding broad progress while identifying unresolved deployment and interpretability constraints.

  • Problem

    CNNs’ local receptive fields limit long-range relationships, while the rapidly expanding Transformer literature in medical imaging lacks a comprehensive, up-to-date synthesis.

  • Method

    The survey organizes more than 125 papers across medical imaging applications, developing taxonomies and reviewing task-specific challenges, solution insights, trends, and open problems.

  • Results

    Transformer models have been applied across segmentation, detection, classification, reconstruction, synthesis, registration, clinical report generation, and other medical imaging tasks, with hybrid segmentation architectures achieving a 13% Dice-score gain over a baseline Transformer model.

  • Takeaways & Limitations

    The survey provides an up-to-date reference and roadmap for exploring Transformer models in medical imaging while outlining promising future directions.

  • Takeaways & Limitations

    Interpretability for ViT-based medical imaging remains in its infancy, and intensive memory and computation requirements hamper deployment on resource-constrained edge devices.

Abstract

from arXiv · show

Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as {de facto} operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of the applications of Transformers in medical imaging covering various aspects, ranging from recently proposed architectural designs to unsolved issues. Specifically, we survey the use of Transformers in medical image segmentation, detection, classification, reconstruction, synthesis, registration, clinical report generation, and other tasks. In particular, for each of these applications, we develop taxonomy, identify application-specific challenges as well as provide insights to solve them, and highlight recent trends. Further, we provide a critical discussion of the field's current state as a whole, including the identification of key challenges, open problems, and outlining promising future directions. We hope this survey will ignite further interest in the community and provide researchers with an up-to-date reference regarding applications of Transformer models in medical imaging. Finally, to cope with the rapid development in this field, we intend to regularly update the relevant latest papers and their open-source implementations at \url{https://github.com/fahadshamshad/awesome-transformers-in-medical-imaging}.

1 INTRODUCTION

Transformers are gaining traction in medical imaging because they model long-range dependencies that CNNs’ local convolutions can miss. This survey organizes the rapidly expanding literature across applications, architectures, challenges, and future directions.

  • CNNs provide efficient, translationally equivariant representations, but local receptive fields limit their ability to capture long-range pixel relationships.CNN filters also use stationary weights that are not adapted to the input image at inference time.
  • Attention-based Transformer models encode long-range dependencies and can replace standard convolutions by operating on sequences of image patches.This design gives rise to Vision Transformers, which have achieved strong results across multiple computer vision tasks.
  • The medical imaging literature has grown rapidly, making a comprehensive survey timely and providing researchers with a roadmap for exploring the field.The survey identifies exponential growth in Transformer-based techniques, particularly following the introduction of ViTs.
  • The survey comprehensively covers more than 125 relevant papers on Transformer applications in medical imaging.
  • It categorizes works by medical imaging application, develops taxonomies, highlights task-specific challenges, and provides literature-based insights for addressing them.
  • The paper critically discusses the field’s current state, key challenges, open problems, and promising future directions.
  • Beyond its primary focus on Vision Transformers, the survey extensively covers language modeling for clinical report generation.It presents this coverage as a first since the original Transformer’s inception about half a decade earlier.

2 BACKGROUND

Medical imaging has progressed from hand-crafted models and CNNs toward Transformer-based approaches that model global interactions through image patches and attention. This survey background outlines ViT construction, self-attention mechanisms, computational limitations, and application-oriented categorization.

  • Hand-Crafted Approaches: Hand-crafted medical imaging models often struggle to represent high-dimensional imaging nuances because reliable labeled data are difficult and their expressive power is limited.Examples include total variation, non-local self-similarity, sparsity, Markov-tree, and related models.
  • CNN-based Methods: CNNs learn discriminative features effectively but depend on large labeled datasets and can be difficult to interpret as black-box solutions.These limitations motivated hybrid approaches combining CNNs with domain-specific prior information.
  • Vision Transformers: A Vision Transformer splits an image into fixed-size patches, flattens and linearly embeds them, adds positional information, and processes the sequence with Transformer encoders.The described workflow includes pre-training on a large-scale image dataset followed by fine-tuning for medical image classification.
  • Self-Attention: Self-attention projects patch embeddings into queries, keys, and values to model interactions among all sequence elements, producing contextualized outputs.For an image, the number of patches is N = HW/P^2, and the embeddings are projected to dimension D.
  • Multi-Head Self-Attention: Multi-head self-attention concatenates multiple self-attention heads, each with separate learnable projections, to model complex dependencies across the input sequence.The output heads are combined through a learned linear transformation.
  • Efficient Attention and Applications: Self-attention has quadratic complexity in sequence length, motivating sparse, linearized, low-rank, memory-compressed, and improved attention methods for high-resolution medical images.The survey focuses on Transformer-block attention and organizes subsequent methods by medical imaging application.

3 MEDICAL IMAGE SEGMENTATION

Transformers have become prominent in medical image segmentation because their global-context modeling captures relationships across spatially distant pixels. Surveyed approaches span hybrid CNN-Transformer architectures, organ-specific and multi-organ tasks, and mechanisms addressing boundaries, scale, computational cost, and modality-specific context.

  • Global-context modeling makes Transformers well suited to segment organs spanning large receptive fields and scattered medical-image backgrounds.
  • Organ-specific segmentation: GT U-Net combines Transformer and convolutional layers for global and local context, while Fourier Descriptor loss addresses fuzzy tooth boundaries and grouping reduces computation.
  • Organ-specific segmentation: TransBridge uses a lightweight parameter-efficient hybrid encoder-decoder to reduce the computational and parameter demands of Transformer-based cardiac segmentation.
  • Organ-specific segmentation: COTR-Net combines convolution-transformer blocks, skip connections, deep supervision, pretrained initialization, and morphological refinement for kidney, cyst, and tumor segmentation.
  • Organ-specific segmentation: Cell-DETR extends DETR with an attention branch that predicts instance segmentation masks alongside bounding boxes for biological cells.
  • 3D medical segmentation: Surveyed 3D models capture global and local context through ViT layers, attention refinement, hierarchical processing, window-based attention, and post-processing strategies.
  • Organ-specific segmentation: SpecTr learns contextual features across hyperspectral bands, removes irrelevant bands through sparsity, and uses band-specific normalization to address distribution mismatch.
  • Organ-specific segmentation: RAT-Net fuses breast-tumor region information across multiple scales to obtain precise ultrasound segmentation.

4 MEDICAL IMAGE CLASSIFICATION

ViTs are surveyed across COVID-19, tumor, and retinal disease classification, including black-box, interpretable, 2D, and 3D approaches. The literature reports competitive or superior performance, while limited data and interpretability remain important challenges.

  • ViT-based classification approaches are organized around COVID-19, tumor, and retinal disease applications with distinct task-specific challenges.
  • COVID-19 Classification: Black-box COVID-19 models prioritize accuracy, but their limited interpretability can hinder user trust and clinical deployment.
  • COVID-19 Classification: Lightweight and hybrid designs address deployment and volumetric challenges by reducing attention complexity or combining 2D and 3D information.
  • Tumor and Retinal Disease Classification: Transformer-based models report better or competitive classification results than CNN baselines across tumor and retinal disease tasks.
  • Challenges and Trends: The lack of large, diverse COVID-19 datasets constrains ViT applicability, motivating domain-specific architectures and loss functions.
  • Discussion: ImageNet-pretrained ViTs perform comparably to CNNs, while medical-image pretraining yields substantial gains over ImageNet pretraining.

5 MEDICAL IMAGE DETECTION

Medical image detection localizes and identifies clinically relevant regions, and Transformer approaches are primarily derived from DETR. The area has fewer recent developments than segmentation and classification, with generic multimodal detectors performing poorly on medical datasets.

  • Medical image detection targets localization and identification of regions of interest such as lung nodules for computer-aided diagnosis.
  • Recent Transformer-based detection methods mainly build on the DETR framework, including hybrid convolutional-transformer architectures.
  • COTR combines convolutional-transformer encoding with Transformer object queries and outperforms DETR on ETIS-LARIB and CVC-ColonDB.
  • Discussion: Transformer-based detection research is less frequent than segmentation and classification, while generic class-agnostic multimodal ViTs pretrained on natural image-text pairs perform poorly on medical datasets.

6 MEDICAL IMAGE RECONSTRUCTION

Transformer reconstruction methods address enhancement and restoration of degraded medical images, especially MRI and CT, using hybrid, supervised, unsupervised, and pretrained designs. Key challenges include ill-posed inversion, data requirements, high-resolution complexity, and limited coverage beyond MRI and CT.

  • Medical image reconstruction seeks clean images from degraded inputs and is categorized into image enhancement and image restoration.
  • Image Enhancement: ViT methods improve low-dose CT and low-dose PET quality by combining Transformer components with CNN, GAN, frequency, or perceptual-processing designs.
  • MRI Reconstruction: MRI restoration methods use cross-attention, multitask learning, and CNN–ViT hybrids for reconstruction and super-resolution in high-data regimes.
  • Low-Data Regime Approaches: Zero-shot SLATER reduces dependence on paired MRI data through inference-time optimization constrained by the MRI forward model.
  • Other Restoration Tasks: Sparse-view CT and surgical-scene reconstruction use Transformer-based designs to address reduced measurements, occlusion, viewpoint changes, and tissue deformation.
  • Pretraining: Pretrained ViTs produce sharp reconstructions with robustness to anatomical shifts, including when fine-tuned using only 100 fastMRI images.
  • Challenges and Trends: Most reconstruction work focuses on MRI and CT, while other modalities and application-specific architectures remain comparatively underexplored.

7 MEDICAL IMAGE SYNTHESIS

Transformer-based synthesis methods generate or translate medical images within or across modalities using supervised, semi-supervised, and unpaired strategies. They improve image realism and quality, but paired-data requirements, adversarial instability, and unsupervised inter-modality translation remain challenges.

  • Medical image synthesis is divided into intra-modality and inter-modality tasks because their challenges differ.
  • Intra-Modality Synthesis: Intra-modality synthesis generates higher-quality images from lower-quality inputs, with supervised methods often requiring paired source and target images.
  • Supervised Methods: PTNet combines performer encoders and decoders with a Transformer bottleneck and surpasses pix2pix and pix2pixHD qualitatively and quantitatively.
  • Supervised Methods: PTNet additionally achieves an execution rate of around 30 slices per second for infant brain MRI synthesis.
  • Semi-Supervised and Unpaired Methods: Semi-supervised and unpaired approaches use ViT discriminators or cycle-consistent hybrid architectures to synthesize images without fully paired training data.
  • Inter-Modality Synthesis: Inter-modality synthesis transfers structural information between modalities such as CT and MRI, but the reviewed approaches are supervised.
  • Discussion: Transformer synthesis can generate realistic images useful when a desired modality is costly or infeasible to acquire, while adversarial loss risks mode collapse.
  • Open Problems: Unsupervised Transformer-based inter-modality synthesis remains unaddressed, partly because CT and MRI have substantially different appearances.

8 MEDICAL IMAGE REGISTRATION

Transformers are being introduced into medical image registration to model correspondences and predict dense deformations, but the evidence base remains too early for firm conclusions.

  • Medical image registration aligns fixed and moving images by estimating dense per-voxel displacement, including across time, viewpoints, or modalities.
  • ViT-V-Net combines convolutional feature extraction, transformer processing, and a V-Net decoder to produce a dense displacement field for warping.
  • TransMorph uses a Swin Transformer encoder and convolutional decoder, with Bayesian inference added for uncertainty estimation.
  • TransMorph outperformed twelve handcrafted, CNN-based, and transformer-based approaches on inter-patient brain MRI and XCAT-to-CT registration.
  • The application of Transformers to medical image registration remains at an early stage, making broad conclusions difficult.

9 CLINICAL REPORT GENERATION

Transformer-based clinical report generation addresses long, diverse, and biased medical narratives through reinforcement learning and supervised or unsupervised strategies. The survey organizes methods by training mechanism and challenges including dataset bias, explainability, feature alignment, and clinical evaluation.

  • Automatic report generation is difficult because radiologists produce diverse reports, sequences are long, and datasets contain more normal than abnormal cases.
  • The survey divides transformer-based report generation into reinforcement-learning and supervised or unsupervised approaches.
  • Reinforcement learning: Self-critical reinforcement learning lets models use evaluation metrics or clinical terminology as rewards and update parameters through policy gradients.
  • Supervised and unsupervised learning: Supervised and unsupervised methods target dataset bias, explainability, feature alignment, and miscellaneous challenges using differentiable losses.
  • Dataset bias: Hierarchical transformer decoding addresses normality bias by attending across report, image, and tag features for abnormality-aware generation.
  • Feature alignment: Cross-modal memory networks improve alignment between encoded medical images and corresponding text for report generation.
  • Clinical evaluation: Natural-language metrics such as CIDEr and BLEU often fail to represent clinical efficacy, motivating disease- and anatomy-aware rewards and human evaluation criteria.
  • Future directions: Transformer report-generation research mainly uses X-ray images, while MRI and PET introduce modality-specific challenges and available multimodal datasets remain limited.

10 OTHER APPLICATIONS

Transformers have been applied beyond core medical imaging tasks, including survival outcome prediction, visual question answering, and medical point-cloud analysis. In survival prediction, models learn relationships between imaging and clinical or molecular features.

  • The survey highlights transformer applications in survival outcome prediction, visual question answering, and medical point-cloud analysis.
  • Survival outcome prediction: MCAT learns co-attention between whole-slide histology and genomics features while predicting cancer survival outcomes.
  • Survival outcome prediction: Clinical Transformer models relationships between clinical and molecular features to predict survival outcomes from a cancerous lung dataset.

11 OPEN CHALLENGES AND FUTURE DIREC-

The survey identifies data, interpretability, robustness, efficiency, and distribution shift as major obstacles to deploying Vision Transformers in medical imaging. It also reports that initialization and pre-training strongly affect performance, while domain-specific generalization remains unresolved.

  • Pre-training: ViTs lack intrinsic local visual inductive biases and therefore require large-scale pre-training, difficult to obtain from typically small medical datasets.
  • Pre-training: CNNs outperform ViTs with random initialization, whereas ImageNet initialization brings ViTs to performance on par with CNNs for medical image classification.
  • Pre-training: Self-supervised methods such as DINO and BYOL improve both model families, with ViTs outperforming CNNs by a small margin in medical image classification.
  • Pre-training: Pre-trained Swin UNETR achieves comparable performance using 60% of labeled data versus 100% for random initialization, reducing manual annotation effort by 40%.
  • Pre-training: Swin UNETR pre-trained on CT performs unsatisfactorily when directly applied to MRI, reflecting a substantial domain gap.
  • Interpretability: Interpretability remains an open problem because ViTs are often black boxes, despite self-attention’s potential to expose interactions between image regions.
  • Adversarial robustness: Medical ViT robustness to adversarial attacks requires dedicated evaluation because natural-image robustness findings cannot be directly transferred to medical imaging.
  • Efficient architectures: ViTs’ intensive memory and computation requirements hinder deployment on resource-constrained edge devices.

12 DISCUSSION AND CONCLUSION

The survey finds that Vision Transformers have spread across medical imaging and presents a comprehensive review intended to guide further research in the field.

  • Vision Transformers have pervaded every area of medical imaging, reflecting the field’s rapid development.
  • The survey recommends workshops and special journal issues to disseminate rapidly developing Transformer research in medical imaging.
  • The survey reviews Transformers across segmentation, detection, classification, reconstruction, synthesis, registration, clinical report generation, and other medical imaging tasks.
  • For each application, it develops a taxonomy, identifies application-specific challenges, offers insights to address them, and specifies recent trends.
  • The authors anticipate substantial room for further exploration and position the survey as a roadmap for researchers progressing the field.
Loading 2201.09873v1…