Source-linked AI summary
Transformers in Medical Image Analysis: A Review
Kelei He, Chen Gan, Zhuoyuan Li, Islem Rekik, Zihao Yin, Wen Ji, Yang Gao, Qian Wang, Junfeng Zhang, Dinggang Shen
TL;DR
Medical image analysis needs methods that can address diverse clinical tasks and limited annotated data. This review synthesizes Transformer architectures and applications across the field, examines learning paradigms and model improvement, and identifies unresolved challenges and future directions.
Problem
Transformers require large-scale annotated data, while medical imaging often has scarce labels and complex multi-modal, multi-task clinical requirements.
Method
The paper reviews more than 170 Transformer-based medical image methods, covering architectures, clinical applications, learning paradigms, efficiency, interpretation, and coupling with other techniques.
Results
Transformers have been successfully applied across diagnosis, registration, segmentation, detection, and synthesis, with comparable or better performance than CNNs in most tasks.
Takeaways & Limitations
The review identifies weakly supervised, multi-modal, multi-task, efficiency, interpretability, quantification, parallelization, and safety research as important directions for medical Transformers.
Takeaways & Limitations
Few medical image analysis studies address Transformer model efficiency despite medical images being larger and available in smaller quantities.
Abstract
from arXiv · showhide
Transformers have dominated the field of natural language processing, and recently impacted the computer vision area. In the field of medical image analysis, Transformers have also been successfully applied to full-stack clinical applications, including image synthesis/reconstruction, registration, segmentation, detection, and diagnosis. Our paper aims to promote awareness and application of Transformers in the field of medical image analysis. Specifically, we first overview the core concepts of the attention mechanism built into Transformers and other basic components. Second, we review various Transformer architectures tailored for medical image applications and discuss their limitations. Within this review, we investigate key challenges revolving around the use of Transformers in different learning paradigms, improving the model efficiency, and their coupling with other techniques. We hope this review can give a comprehensive picture of Transformers to the readers in the field of medical image analysis.
1 INTRODUCTION
Transformers evolved from NLP into medical image analysis, where they have been applied across diagnosis and other clinical tasks. This review surveys more than 170 methods, their architectures, learning paradigms, efficiency, interpretation, and open challenges.
- Transformers capture long-term dependencies with stacked self-attention layers while processing sequential learning problems in one shot.
- CNNs learn hierarchical image representations through progressively enlarged receptive fields but neglect non-local correlations within images.
- Transformers have been applied to disease diagnosis, segmentation, detection, and synthesis in medical imaging, achieving state-of-the-art results in reported studies.
- The review examines more than 170 Transformer-based methods and discusses weakly supervised, multi-task, and multi-modal learning, efficiency, and interpretation.
- The paper organizes its coverage around Transformer preliminaries, medical image applications, future directions, and conclusions.
2 TRANSFORMERS
Transformers use attention to weight relationships among input elements and combine these weighted values into representations. Self-attention extends this operation to matrices, while multiple heads capture hierarchical features in parallel.
- The paper introduces attention as a core principle before describing how Transformer architectures operate.
- Attention filters information by emphasizing meaningful components while assigning less importance to others.
- Self-attention defines outputs from queries, keys, and values derived from the input, using query-key attention weights to form weighted value sums.
- Each input x_i is linearly projected into q_i, k_i, and v_i vectors, with learned dimensions d_q, d_k, and d_v.
- For matrix inputs X, parallel computation produces attention matrix A and output matrix Z from stacked query, key, and value representations.
- Multi-head self-attention concatenates multiple self-attention outputs to capture hierarchical features across different learned subspaces.
2.2 Architecture
A typical Transformer uses stacked encoder and decoder blocks built from multi-head attention, feed-forward layers, residual connections, and layer normalization. The decoder adds masked self-attention and output projection for sequence generation.
- The encoder maps an input sequence to an output sequence of the same length, while the decoder generates outputs from encoded representations and previous outputs.
- Each encoder block combines multi-head attention, layer normalization, residual connections, and a feed-forward layer.
- The decoder contains an additional self-attention layer and masks subsequent positions so predictions use only known prior states.
- The architecture is illustrated as a typical Transformer encoder-decoder design.
- A linear layer and Softmax layer follow the decoder output to generate the final output.
2.3 Vision Transformer
Vision Transformers adapt Transformer processing to images by tokenizing patches and using positional information for classification. Related architectures address data requirements, high-resolution attention costs, and alternative neural design choices.
- Vision Transformer: DETR uses a CNN backbone, Transformer encoder and decoder, and feedforward network for end-to-end object detection.
- Vision Transformer: ViT converts an image into positionalized patches, processes them with Transformer attention, and classifies the class-token representation through an MLP.
- Vision Transformer: DeiT uses teacher-student knowledge distillation and a distillation token to address ViT’s large-scale training-data requirement.
- Vision Transformer: Swin-Transformer reduces attention costs for high-resolution images and accommodates varied patch sizes in scene-understanding tasks.
- Other techniques: MLP-based models such as MLP-Mixer, gMLP, ResMLP, ASMLP, and CycleMLP have shown competitive performance alongside CNNs and Transformers.
3 TRANSFORMERS IN MEDICAL IMAGE APPLICA-
Transformers have been applied across full-stack medical image analysis, spanning classification, segmentation, image-to-image translation, detection, registration, and video-based applications.
- Medical image applications of Transformers include classification, segmentation, image-to-image translation, detection, registration, and video-based tasks.
3.1 Classification
For medical image classification, reviewed methods use pure ViTs, convolutional hybrids, or graph-based hybrids across multiple imaging modalities and diseases. Transformers achieve comparable or better performance than CNNs in most tasks, while remaining constrained by data and computational demands.
- Classification methods address disease diagnosis and prognosis through three categories: pure ViTs, ViT-convolution hybrids, and ViT-graph hybrids.
- Pure Transformers: Pure Transformers are ViTs similar to the original architecture, with applications organized by modalities including X-ray, CT, MRI, ultrasound, and OCT.
- Pure Transformers: Reviewed classification studies apply Transformers to diseases including COVID-19, intracranial hemorrhage, pancreatic cancer, breast cancer, leukemia, melanoma, and fundus diseases.
- Hybrid Transformers: Hybrid approaches combine CNNs with Transformers for local feature learning, feature aggregation, multi-view analysis, pretrained backbones, or data resampling.
- Transformers achieve comparable or better performance than CNNs in most tasks, while hybrid Transformers benefit from both conventional networks and Transformers.
3.2 Segmentation
Transformer-based segmentation methods address Unet’s limited modeling of long-term dependencies through hybrid, pure-Transformer, and feature-fusion designs. The reviewed approaches vary in Transformer placement, CNN coupling, and multi-scale or attention-based processing.
- Transformer-based methods cover abdominal, thoracic, cardiac, pancreas, brain, tumour, tissue, and polyp segmentation tasks.
- Unet achieves strong medical image segmentation performance, but convolution operations limit its ability to model long-term dependencies.
- Hybrid Transformers: Hybrid architectures insert Transformer layers at different U-shaped levels, combine Transformers and CNNs through varied strategies, or use multi-scale features and attention mechanisms.
- Location of Transformer in U-shaped architecture: TransUNet places Transformer layers in the encoder, then upsamples self-attentive features and combines them with multi-scale encoder features through skip connections for localization.
- Hybrid Transformers: nnFormer uses an interleaved architecture that empirically combines self-attention and convolution, addressing concerns about how the two operations are combined.
- Hybrid Transformers: Other designs process multi-scale or high-resolution features, use pure Transformer encoders, apply attention in encoder and decoder blocks, or fuse inter-slice and intra-slice information.
- The reviewed methods generally leverage additional features learned through feature-fusion strategies for more effective learning.
3.3 Image-to-image translation
Transformers have been applied to medical image synthesis, reconstruction, super-resolution, and denoising, often alongside generative adversarial or encoder-decoder designs. These applications target challenges including anatomical hallucinations, broad PET intensity ranges, and limited low-dose CT denoising work.
- Medical image-to-image translation research includes image synthesis, reconstruction, and super-resolution applications using Transformer models.
- Image synthesis: Medical image synthesis is challenging because of inter-subject variability and potentially harmful anatomical hallucinations.
- Image synthesis: Transformer-based image synthesis methods have been combined with generative adversarial learning, including architectures with global Transformer and local CNN discriminators.
- Image synthesis: GANBERT generates PET images from MRI images, while 3D-Transformer GAN reconstructs high-quality PET images from low-dose acquisitions.
- Image synthesis: ResViT uses aggregated residual Transformer blocks in an encoder-decoder bottleneck for multi-modal medical image synthesis.
- Image super-resolution: T2Net jointly learns MRI reconstruction and super-resolution through separate branches linked by a Transformer module.
- Image denoising: TED-net and Eformer apply Transformer-based encoder-decoder architectures to medical image denoising, while low-dose CT denoising applications remain scarce.
3.4 Detection
In this review, detection refers to object detection, while related terminology can mean disease checking or diagnosis in technical and clinical contexts. Medical detection Transformers are often combined with CNN feature extractors.
- “Detection” may refer to disease or lesion existence in technical contexts, diagnosis or classification in clinical practice, and object localization with category prediction in computer vision.
- Medical image detection Transformers commonly use CNNs for feature extraction and Transformers to enhance features for downstream detection.
- COTR is a DETR-based polyp-detection model that combines a CNN feature extractor with Transformer components.
3.5 Registration
Transformers have been applied to medical image registration because self-attention can support more precise spatial mapping between moving and fixed images. ViT-V-Net combines a Vision Transformer with V-Net by modifying VoxelMorph’s conventional registration architecture.
- Self-attention enables more precise spatial mapping between moving and fixed images in Transformer-based registration.
- ViT-V-Net combines ViT and V-Net by altering VoxelMorph’s conventional registration-network architecture.
3.6 Video-based applications
Transformers address CNNs’ limited receptive fields in video by modeling global temporal and spatial information, supporting several medical video-analysis applications.
- Video-based applications: Transformers can overcome CNNs’ limited receptive field for continuous video frames by modeling global temporal and spatial information.The passage motivates Transformer use for video analysis through their ability to capture information across frames.
- Video-based applications: PNS-Net applies progressively normalized self-attention to accurate polyp segmentation in colonoscopy videos.
- Video-based applications: LapFormer detects surgical tools in laparoscopic surgery videos.
- Video-based applications: OperA predicts surgical phases from long videos.
4 DISCUSSION
The discussion frames medical Transformer research around clinical data scarcity, multimodal and multitask learning, weak supervision, and efficiency. Existing methods address these challenges, but efficiency remains underexplored and ViTs have not consistently surpassed CNNs, especially with few-shot or low-resolution data.
- Challenges: Label scarcity and noisy labels constrain clinical deployment, particularly for segmentation and detection requiring precise pixel-wise annotations.
- Multi-modal and multi-task learning: Combining multiple modalities and tasks can support diagnosis and generalizability, but designing such versatile models remains difficult.Examples include OCT–visual-field glaucoma diagnosis and joint skin-lesion segmentation and classification.
- Weakly-supervised learning: Weakly supervised approaches use bag-level labels, cross-teaching, adversarial losses, or teacher structures to learn from limited annotations or unlabeled data.Multiple-instance learning models relations among instances, while semi-supervised methods exchange pseudo-labels or use adversarial learning.
- Weakly-supervised learning: Self-supervised learning transfers knowledge from unlabeled upstream tasks to downstream medical-image classification, detection, and segmentation.The reviewed practice commonly pretrains on ImageNet and fine-tunes on the target medical dataset.
- Model efficiency: Efficient attention can reduce self-attention complexity from O(n^2) to approximate O(n), while hybrid designs can initialize Transformers as convolutional networks without pretraining.
- Model efficiency: Fewer medical-imaging studies address efficiency than computer-vision studies, despite medical images being larger and available in smaller quantities.
- Comparison with convolutional neural networks: ViTs have not consistently outperformed CNNs across medical-imaging settings, particularly few-shot and low-resolution scenarios, motivating hybrid convolutional models.
5 CONCLUSION
The conclusion identifies rapid growth in medical Transformer research but emphasizes that advanced learning paradigms and broader model concerns remain insufficiently explored.
- Conclusion: Most medical Transformer methods apply existing architectures to imaging problems without drastic changes.
- Conclusion: Weakly supervised, multimodal, multitask, and model-improvement methods are rarely explored in current medical Transformer research.
- Conclusion: Parallelization, interpretability, quantification, and safety are additional general model problems identified as future directions.