Source-linked AI summary
Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives
Jun Li, Junyu Chen, Yucheng Tang, Ce Wang, Bennett A. Landman, S. Kevin Zhou
TL;DR
The paper addresses whether Vision Transformers can improve medical imaging relative to CNN-based approaches and clarifies their status across medical image-analysis tasks. It introduces Transformer fundamentals and key properties, then comparatively reviews state-of-the-art methods by task and architecture. The review reports broad progress, including generally stronger segmentation performance, while emphasizing dependence on pre-training and unresolved computational, fairness, and interpretability challenges.
Problem
The review asks whether Vision Transformers are better than CNNs for understanding medical images and whether Transformers can transform medical imaging.
Method
The paper provides a comparative review organized by Transformer properties and by architectures combining Transformers with CNNs across major medical-imaging tasks.
Results
Transformer-based models generally outperform ConvNets for medical image segmentation, while classification benefits heavily from pre-training and remains less effective without large-scale data or pre-training.
Takeaways & Limitations
Transformer properties and CNN–Transformer architectures offer a framework for understanding current medical-imaging methods across segmentation, classification, detection, registration, reconstruction, and enhancement.
Takeaways & Limitations
Medical-imaging Transformers face quadratic self-attention costs on large images, data-bias and domain gaps, and continuing black-box interpretability concerns.
Abstract
from arXiv · showhide
Transformer, the latest technological advance of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to inquire about the status quo of Transformers in medical imaging and ask the question: can the Transformer models transform medical imaging? In this paper, we attempt to make a response to the inquiry. After a brief introduction of the fundamentals of Transformers, especially in comparison with convolutional neural networks (CNNs), and highlighting key defining properties that characterize the Transformers, we offer a comprehensive review of the state-of-the-art Transformer-based approaches for medical imaging and exhibit current research progresses made in the areas of medical image segmentation, recognition, detection, registration, reconstruction, enhancement, etc. In particular, what distinguishes our review lies in its organization based on the Transformer's key defining properties, which are mostly derived from comparing the Transformer and CNN, and its type of architecture, which specifies the manner in which the Transformer and CNN are combined, all helping the readers to best understand the rationale behind the reviewed approaches. We conclude with discussions of future perspectives.
1. Introduction
Medical imaging is a major source of clinical evidence, while deep learning and CNNs have become prevalent for analyzing medical images. This review examines whether Vision Transformers can improve medical image analysis and surveys their applications, properties, architectures, and future directions.
- Medical images account for about 90% of healthcare data and provide critical evidence for clinical decision making.
- Deep learning has achieved strong performance in classification, object detection, and semantic segmentation of medical images.
- CNNs are the customary deep-learning methods for image data, with medical-imaging applications dating to the 1990s and widespread state-of-the-art deployment emerging in the last decade.
- The review asks whether Vision Transformers are better than CNNs for understanding medical images and whether they can transform medical imaging.
- The paper reviews Transformer-based methods across segmentation, recognition, classification, detection, registration, reconstruction, and enhancement, alongside challenges and future potential.
2. CNN and RNN for Medical Image Analysis
CNNs have been highly successful in medical imaging, especially through U-Net-like architectures, but their local convolutions limit explicit modeling of long-range dependencies. Transformers address this limitation through self-attention, while introducing data and computational considerations.
- CNNs suit grid-like image data because convolution shares the same weights across pixels, imposing a strong spatial prior.
- U-Net uses a convolutional encoder and decoder with upsampling to produce full-resolution outputs and became the de facto paradigm for pixel-level medical imaging tasks.
- CNNs using small kernels are biased toward local structures and are less effective at explicitly modeling long-range spatial dependencies.
- RNN- and LSTM-based models have been used for medical image segmentation, classification, detection, and longitudinal or spatiotemporal data.
- Transformers use self-attention to capture explicit long-range dependencies, scale more easily, and show encouraging medical-imaging results.
3. Fundamentals of Transformer
Transformers compute representations and global dependencies using self-attention and feedforward modules. Vision Transformer adapts this framework to images by converting them into sequences and modeling visual features sequence-to-sequence.
- A Language Transformer combines self-attention mechanisms with a feedforward module to compute representations and global dependencies.
- Vision Transformer partitions and flattens images into sequences, then applies Transformer modeling in a sequence-to-sequence paradigm.
3.1. Self-attention in Transformer
Self-attention projects input features into queries, keys, and values, then uses query–key affinities to aggregate values across all elements. Multi-head self-attention performs this process in parallel across multiple subspaces and combines the resulting outputs.
- Self-attention: Self-attention maps input X into query Q, key K, and value V representations using learnable parameters.The representations have n elements and dimensions determined by the input and projection spaces.
- Self-attention: The similarity between Q and K is normalized into an attention distribution A.This distribution defines the affinities used to weight the value features.
- Self-attention: The attention distribution weights V to produce the self-attention output Z.The output is a weighted sum of values, allowing the block to connect all elements.
- Self-attention: Because the attention matrix connects all elements, self-attention can model long-range dependencies in language and vision tasks.Keys act as an embedding matrix, queries as lookup vectors, and values provide the aggregated content.
- Multi-head self-attention: Multi-head self-attention computes multiple self-attention blocks in parallel and concatenates and projects their outputs.The heads project Q, K, and V into multiple subspaces; increasing the number of heads does not necessarily improve performance.
3.2. Vision Transformer pipeline
A Vision Transformer converts an image into position-aware patch tokens, processes them with stacked Transformer blocks, and uses a task-specific decoder. Its design preserves positional information while combining global attention with localized MLP processing.
- Overview: A Vision Transformer splits an image into N non-overlapping patches, vectorizes each patch, and linearly projects the resulting vectors into tokens.For an image X ∈ R^{C×H×W}, each patch has dimensions C×P×P.
- Positional embedding: Positional embeddings are added to patch tokens so their locations in the original image remain available throughout the network.Sinusoidal, learnable, and relative embeddings encode absolute positions or offsets between patches.
- Transformer encoder: The resulting tokens enter an encoder of L stacked blocks, each containing multi-head self-attention, Layer-Norm, and an MLP.The encoder is paired with a task-specific decoder in the typical Vision Transformer design.
- Patch processing: Non-overlapping patches avoid computational redundancy, while self-attention integrates information across patches to alleviate the loss of internal image structure.This design follows the standard Transformer with minimal vision-specific modifications.
- MLP blocks: Transformer MLPs act locally and translation-equivariantly because their linear projections are analogous to 1×1 convolutions.The typical weight dimensions are D×4D and 4D×D.
3.3. Transformer vs. CNNs
The review contrasts CNN and Transformer image processing and organizes combined designs into Conv-like Transformers, Transformer-like CNNs, and Conv-Transformer hybrids. These approaches differ in how they combine local structure, global context, and feature fusion.
- Transformer vs. CNNs: CNN receptive fields expand with depth, whereas Transformers can capture global relationships through self-attention.CNN features are analyzed layer by layer, with global information introduced progressively.
- Conv-like Transformers: Conv-like Transformers retain MLP and self-attention blocks while arranging attention locally and hierarchically, often with neighboring-window merging.Swin Transformer, HaloNets, and DAT exemplify this category.
- Transformer-like CNNs: Transformer-like CNNs retain convolutional building blocks but incorporate Transformer traits such as self-attention into convolutional processing.CoT and BoTNet are examples of this category.
- Conv-Transformer hybrids: Conv-Transformer hybrids use convolutions, MLPs, and self-attention together to combine local processing with long-distance relationships.Some methods retain attention for long-range dependencies, while others fuse parallel CNN and Transformer branches.
- Role of MSA: Self-attention is not uniquely responsible for global feature extraction, since repeated MLPs or large-kernel depth-wise convolutions can also aggregate features.The review frames CNN convolutions and Transformer MLPs as pattern learners, with pooling and other operations integrating feature maps.
3.4. Key properties
The review identifies modeling and computational properties that distinguish Transformers from CNNs, including long-range dependency modeling, detail preservation, weaker inductive bias, flatter loss landscapes, scaling, integration, and computational cost.
- M1: Long-range dependency: Self-attention connects all image patches, giving Transformers large theoretical and effective receptive fields for modeling long-range spatial relationships.The review associates this property with broader contextual understanding than CNNs.
- M2: Detail modeling: Transformers refine local patch embeddings at the same scale, enabling dense feature modeling that can preserve contextual details while deeper layers add semantic information.CNN features instead pass through pooling and strided convolutions across different scales.
- M3: Inductive bias: CNN locality supplies stronger inductive bias, supporting faster convergence and better performance on small datasets, whereas Transformers generally have weaker inductive bias.Original ViT manually injects positional information as its stated inductive bias.
- M4: Loss landscape: Transformer self-attention tends to produce flatter loss landscapes, which the review links to improved performance and generalizability under the same training conditions.The loss-landscape comparison includes hybrid CNN-Transformer models.
- C1–C2: Scaling and integration: Transformers improve performance as computation, model capacity, and data size scale together, and they can be integrated with CNNs through three flexible architecture families.The families are Conv-like Transformers, Transformer-like CNNs, and Conv-Transformer hybrids.
- C3: Computational intensiveness: Typical Transformers require substantial time and memory, particularly during training.This computational intensiveness is identified as a key limitation despite their promising results.
4. Current Progresses
The review surveys Transformer-based medical image analysis across major tasks, organizing approaches by defining properties and by how Transformer and CNN components are combined. It reports broad progress in segmentation, classification, detection, registration, and reconstruction while identifying data, scale, and evaluation challenges.
- Review scope: The review identified over 2000 Transformer-related manuscripts and projects across medical imaging research venues and databases.The search covered IEEE, PubMed, Xplore, Springer, ScienceDirect, major medical imaging conferences, and Google Scholar.
- Medical image segmentation: Transformer-based models generally outperform ConvNets in medical image segmentation, supported by long-range context modeling, scaling behavior, and flexible hybrid architectures.The review attributes these advantages to Properties M1, M2, C1, M3, and C2.
- Medical image segmentation: Segmentation architectures range from Transformer bottleneck encoders and CNN–Transformer hybrids to pure Transformer encoder–decoder networks.Examples include TransUNet-like, Swin UNETR, TransFuse, and nnFormer-like U-shaped designs.
- Medical image segmentation: Swin UNETR achieves state-of-the-art performance on 10 Medical Segmentation Decathlon tasks and BTCV benchmarks, while hierarchical Transformer models achieve state-of-the-art kidney sub-component segmentation with CT.These results illustrate progress in 3D medical image segmentation.
- Medical image recognition and classification: Transformer-based classification remains heavily dependent on large-scale supervised or self-supervised pre-training, and may not exceed CNN effectiveness without pre-training and large training data.Most reviewed classification models are 2D, with further work anticipated for 3D classification.
- Detection, registration, and reconstruction: Transformer methods show promise in detection, registration, and reconstruction, but registration remains in its infancy and existing methods have not evaluated advanced training strategies or more complex self-attention designs.Detection may use encoder–decoder designs to reduce post-processing, while registration commonly uses Transformers for correspondence modeling and ConvNets for deformation-field refinement.
5. Future Perspectives
The paper gives a qualified positive outlook for Transformers in medical imaging while emphasizing unresolved challenges involving data, computation, interpretability, 3D modeling, and domain knowledge.
- Future outlook: Transformers are gaining momentum across medical imaging, but their benefits remain qualified by inherited machine-learning limitations.The paper describes the answer as both potentially positive and negative because Transformers inherit the pros and cons of deep learning.
- Challenges: Sparse data and annotations constrain Transformer performance, making self-supervised learning a promising response.Performance improves with larger datasets and degrades when data or annotations are sparse; proxy tasks such as contrastive learning and reconstruction can use unlabeled data.
- Challenges: Scaling Transformers may support learning across modalities and domains, but domain gaps, bias, fairness, and clinical interpretability remain active concerns.The paper discusses adaptation methods, fairness concerns, and the need for interpretability suited to clinical users and contexts.
- Challenges: Efficient 3D modeling remains difficult because volumetric medical images are computationally and memory intensive, with positional information potentially lost across patches.Existing methods address spatial-feature modeling, yet heterogeneous tissues and 3D positional information remain challenging.
- Challenges: Self-attention creates quadratic computational cost, so medical-image Transformers often require compact designs, patching, downsampling, or smaller batches.Medical images are substantially larger than common natural images, increasing the practical burden of self-attention.
- Debate: The field continues to debate whether Transformers’ advantages arise from global dependencies, large receptive fields, locality, data specificity, or hybrid convolutional designs.The review presents competing explanations and notes that newer CNNs increasingly incorporate Transformer-like components such as large kernels and depthwise convolution.
- Task-specific perspectives: Detection particularly benefits from modeling and aggregating long-range dependencies because lesion localization and categorization require contextual information about abnormalities and organs.The review identifies this capability as potentially the most critical Transformer property for medical-image detection.
Appendix .1. Inductive bias
CNNs encode strong locality through convolution and pooling, whereas Transformers can perform better with sufficient pretraining and can incorporate locality or CNN-derived inductive bias.
- Inductive bias: CNN convolution and pooling impose strong intrinsic inductive biases by tying neighboring weights and constraining representations.These operations act like strong priors over network weights.
- Inductive bias: Transformers achieve superior performance on some smaller-data tasks when pretrained on sufficiently large datasets.The passage attributes this comparison to several studies across vision and Transformer research.
- Inductive bias: Researchers introduce locality into Transformers, distill CNN inductive bias, or combine CNNs and Transformers to address weak inductive bias.These approaches seek to retain useful convolution-like properties while using Transformer architectures.
Appendix .2. Loss landscapes
The appendix uses loss landscapes to examine parameter sensitivity and reports flatter landscapes for Vision Transformers than CNNs in registration and segmentation.
- Loss landscapes: Loss-landscape flatness measures how sensitive network parameters are to perturbations.The appendix describes loss landscapes as plots generated relative to neural-network parameters.
- Loss landscapes: ViTs tend to promote flatter loss landscapes than CNNs and thus generalize better on unseen data.The paper empirically confirms this finding by comparing CNNs and Transformers on registration and segmentation tasks.
Appendix .3. Computational complexity of Transformers
Transformer self-attention updates each token using all other tokens, producing quadratic complexity in sequence length and greater resource demands than CNN convolution.
- Self-attention: Self-attention updates each token by attending to all other tokens.This all-token interaction is identified as the main computational bottleneck in Transformers.
- Complexity: For sequence length n and embedding size d, self-attention has O(n^2d) matrix-multiplication cost and overall O(n^2) complexity.Both Q × K^⊤ and Softmax(·) × V have O(n^2d) complexity.
- Comparison: CNN convolution has linear O(n) complexity, so Transformer training generally requires more time and resources.The appendix motivates modifications to self-attention that reduce computational complexity.