PaperlayerView exact revision on arXiv

Source-linked AI summary

A survey of the Vision Transformers and their CNN-Transformer based Variants

Asifullah Khan, Zunaira Rauf, Anabia Sohail, Abdul Rehman, Hifsa Asif, Aqsa Asif, Umair Farooq

arXiv:2305.09880v4cs.CV

TL;DR

Vision Transformers face limited ability to capture low-level local correlations, motivating architectures that combine local and global modeling. This survey organizes Vision Transformers and hybrid architectures into taxonomies, examines their architectural features, and reports improved image-recognition performance alongside local-global learning capability.

  • Problem

    Vision Transformers face limited ability to capture low-level features through local-neighborhood correlations, motivating systematic study of hybrid architectures.

  • Method

    The survey develops taxonomies of Vision Transformers and hybrid architectures based on their integration methodology and examines their architectural organization.

  • Results

    Hybrid Vision Transformers show improved performance in image recognition and support learning at both local and global levels.

  • Takeaways & Limitations

    Hybrid Vision Transformers have substantial potential because they can perform learning at both local and global levels.

  • Takeaways & Limitations

    Efficiently merging learned features remains a major challenge for hybrid vision transformer architectures.

Abstract

from arXiv · show

Vision transformers have become popular as a possible substitute to convolutional neural networks (CNNs) for a variety of computer vision applications. These transformers, with their ability to focus on global relationships in images, offer large learning capacity. However, they may suffer from limited generalization as they do not tend to model local correlation in images. Recently, in vision transformers hybridization of both the convolution operation and self-attention mechanism has emerged, to exploit both the local and global image representations. These hybrid vision transformers, also referred to as CNN-Transformer architectures, have demonstrated remarkable results in vision applications. Given the rapidly growing number of hybrid vision transformers, it has become necessary to provide a taxonomy and explanation of these hybrid architectures. This survey presents a taxonomy of the recent vision transformer architectures and more specifically that of the hybrid vision transformers. Additionally, the key features of these architectures such as the attention mechanisms, positional embeddings, multi-scale processing, and convolution are also discussed. In contrast to the previous survey papers that are primarily focused on individual vision transformer architectures or CNNs, this survey uniquely emphasizes the emerging trend of hybrid vision transformers. By showcasing the potential of hybrid vision transformers to deliver exceptional performance across a range of computer vision tasks, this survey sheds light on the future directions of this rapidly evolving architecture.

1. Introduction

The introduction contrasts CNNs’ local processing with ViTs’ global self-attention, motivating Hybrid Vision Transformers that combine convolutional local features with transformer-based global context. It presents the survey’s taxonomies of ViTs and HVTs and outlines the paper’s coverage of architectures, applications, challenges, and future directions.

  • CNNs and ViTs: CNN filters capture distributed local feature motifs, while pooling reduces dimensionality and improves robustness but may lose spatial correlation for larger patterns.This limitation can affect performance when images contain larger and more complex patterns.
  • CNNs and ViTs: ViTs divide images into patches and use self-attention to model long-range relationships, retaining global relationships and identifying complex visual patterns.Their global receptive field supports global-scale feature modeling, unlike CNNs’ local receptive fields.
  • Motivation for HVTs: ViTs may underperform CNNs with small training data and require significantly more computation because of their large receptive field.The introduction also notes that ViTs depend heavily on large datasets because they assume minimal prior knowledge.
  • Hybrid Vision Transformers: Hybrid Vision Transformers combine CNNs’ local feature extraction with ViTs’ self-attention-based global context, yielding improved performance in many image-recognition tasks.CNN-derived local features are fed to ViTs to obtain global context.
  • Survey contributions: The survey emphasizes HVT developments, introduces a taxonomy for general ViTs, and classifies HVTs by their architectural composition and use of convolution with self-attention.Reported ViTs are grouped into six categories, while HVTs are divided into seven major groups.

2. Fundamental Concepts in ViTs

Vision Transformers convert images into patch-token sequences, add positional information, and process them through repeated attention-based encoder blocks. Their architecture combines self-attention for global relationships with feed-forward, residual, and normalization components for representation learning.

  • Patch Embedding: ViTs partition images into fixed-size non-overlapping patches, flatten them, and linearly project them into D-dimensional token embeddings.For an image of size A×B×C, the number of patches is N = A/P × B/P, with D = P×P×C.
  • Positional Embeddings: Positional embeddings preserve sequential information between patches by incorporating positional data into patch embeddings throughout the network.Absolute positional embeddings are learnable and share dimensions (N+1) × D with the patch embeddings.
  • Self-Attention: Self-attention represents relationships among sequence entities using query, key, and value spaces and global contextual information.It computes outputs as weighted sums of values, with weights determined by a scoring function followed by softmax.
  • Multi-Head Attention: Multi-head self-attention addresses single-head limitations by stacking attention blocks in parallel across diverse representation subspaces.The block outputs are concatenated and projected into the output space using the learnable parameter W_O.
  • Encoder Architecture: A ViT encoder repeatedly combines MSA, FFN, residual connections, and normalization layers to learn complex representations of the input sequence.The FFN uses fully connected layers with a nonlinear activation such as GELU and follows self-attention in each encoder block.

3. Architectural level modifications in ViTs · 3.1. Patch-based approaches

The survey organizes ViT architectural modifications into five classes and separately categorizes hybrid vision transformers by structural design. Within patch-based approaches, it reviews fixed, dynamic, overlapping, iterative, multi-level, and adaptive patch extraction strategies.

  • 3. Architectural level modifications in ViTs: ViT architectural modifications are classified into patch-based, knowledge transfer-based, shifted window-based, attention-based, and multi-transformer-based approaches.The survey also reports seven structural categories for hybrid vision transformers and notes performance gains from introducing CNN inductive bias.
  • 3.1. Patch-based approaches: ViT divides images into patches, flattens them into token embeddings, adds positional and class tokens, and passes the sequence to an encoder for feature learning.The survey discusses fixed-size, dynamic, and overlapping patch extraction mechanisms as modifications intended to improve ViT performance.
  • 3.1.1. Tokens-to-Token Vision Transformer (T2T-ViT): T2T-ViT uses a fixed-size, iterative Token-to-Token module to generate patches before feeding them into the network for final predictions.The module iteratively generates patches from the input images.
  • 3.1.2. Transformer in Transformer (TNT-ViT): TNT-ViT divides images into patches and sub-patches, using different transformer blocks to model relationships between both levels and learn multi-scale object representations.Extensive experiments showed TNT-ViT’s efficiency for image classification on ImageNet.
  • 3.1.3. Deformable Patch-based Transformer (DPT): DPT introduces the adaptive DePatch embedding module, which splits images into variable-sized patches to preserve stronger semantic information than fixed-size patching.The survey identifies semantic-information loss from fixed-size patching as a performance problem addressed by DePatch.
  • 3.1.4. CrowdFormer: CrowdFormer applies overlapping patching with a sliding window to capture global contextual information and retain relative context for effective crowd counting.The overlap patching layer considers images at different scales in a top-down manner.

3.2. Knowledge transfer-based approaches

Knowledge transfer-based approaches distill information from larger teacher ViT models into smaller student models, improving efficiency for resource-limited deployment. DeiT and TinyViT exemplify this category through distillation strategies that support competitive performance, generalization, and large-scale evaluation.

  • Knowledge transfer-based approaches: Knowledge distillation transfers distinctive features from a complex teacher network to a simpler student model for more efficient deployment under limited computational resources.The student incorporates important teacher features while retaining lower computational demands.
  • Knowledge transfer-based approaches: DeiT uses a pre-trained ViT teacher and smaller student, combining supervised and unsupervised learning to produce similar results.Teacher supervision supports the student during knowledge transfer.
  • Knowledge transfer-based approaches: DeiT offers fast inference and limited computational requirements while improving generalization by learning important data features and patterns instead of memorizing training data.Its competitive performance is attributed to capturing salient rather than memorized information.
  • Knowledge transfer-based approaches: TinyViT transfers learned features from large pre-trained models during pre-training using stored instructor logits, reconstructed augmentations, and independently trained models.Reduced logits and encoded augmentations are stored beforehand to save memory and computational resources, while a decoder reconstructs the augmentations.
  • Knowledge transfer-based approaches: TinyViT demonstrated effectiveness on large-scale test sets through its fast distillation methodology and novel architecture.The approach transfers knowledge via output logits while the teacher and student train independently.

3.3. Shifted window-based approaches

Shifted window-based architectures constrain self-attention to local windows while using window shifts to create cross-window connections and improve global relationship modeling. Extensions such as CSWinTT apply cyclic shifting and multi-scale attention for more accurate and efficient visual object tracking.

  • Swin Transformer: Swin Transformer computes self-attention within non-overlapping local windows while retaining cross-window connections through a shifted windowing scheme.The method uses two consecutive blocks: regular window attention followed by attention on shifted windows.
  • Swin Transformer: Shifting windows enables cross-window connections that improve the network’s capability to model global relationships.
  • CSWinTT: CSWinTT uses cyclic shifting window-based attention at multiple scales to aggregate cross-window attention and preserve tracking-object integrity.It enhances pixel attention to window attention and generates fine-scale target matches.
  • CSWinTT: Positional information in CSWinTT expands window samples and helps track objects through positional changes over time with greater accuracy and computational efficiency.The architecture showed promising results for improving the accuracy and efficiency of ViT-based visual object tracking.

3.4. Attention-based approaches

Attention-based vision transformers modify self-attention to capture image dependencies, using dense or sparse mechanisms despite their computational expense. Recent approaches improve attention through delayed class-token integration, data-dependent regional focus, and separate local-global attention modules.

  • Attention-based approaches: Dense and sparse attention mechanisms capture global-level image dependencies but are computationally expensive.Dense global attention and sparse attention are both used to model global relationships without spatial correlation.
  • Attention-based approaches: CaiT separates self-attention from class attention by adding class embeddings later rather than in the initial layers.Its self-attention module resembles a normal ViT, while class information is introduced in a later class attention module.
  • Attention-based approaches: DAT uses data-dependent attention to focus on reliable regions through local attention followed by deformable attention at each stage.The modular architecture showed exemplary performance on benchmark datasets.
  • Attention-based approaches: SeT combines pixel-wise attention for local interactions with patch-wise attention for global information, achieving superior results on ImageNet and MS COCO.The two attention modules are applied sequentially, with pixel-wise attention in initial layers and patch-wise attention later.

3.5. Multi-transformer-based approaches

Multi-transformer architectures use multiple transformer pathways or branches to learn multi-scale and multi-view representations. The reviewed models improve feature quality or computational efficiency through cross-attention, parallel processing, and multi-scale patching.

  • CrossViT: CrossViT combines different-sized image patches in dual branches, repeatedly fusing them with cross-attention for efficient knowledge transfer and linear attention-map generation.Separate branches process smaller and larger patch tokens with different computational complexities, while cross-attention creates a non-patch token.
  • Dual-ViT: Dual-ViT uses parallel semantic and pixel pathways to capture global and local information while reducing self-attention computational cost.The semantic branch learns coarse details and the pixel pathway captures fine image details; both branches are integrated and trained in parallel.
  • Dual-ViT: DualViT achieved good ImageNet results with fewer parameters than existing models.The supplied passage reports this result without providing a numerical accuracy value.
  • MMViT: MMViT processes multiple views at multiple resolutions in parallel and uses cross-attention to merge perspectives into robust multiscale representations.Its feature-extraction stages produce high-dimensional representations at multiple resolutions.
  • MPViT: MPViT uses CNN-generated multiscale feature maps, multiple transformer encoders, and output aggregation to learn representations across scales.The architecture applies multi-scale patching before processing patch embeddings with multiple encoders.

3.6. Taxonomy of HVTs (CNN-Transformer architectures)

Hybrid vision transformers combine CNNs and transformers to address ViTs’ limited local modeling, computational cost, and fixed-size token constraints. The survey classifies HVTs by how convolution is integrated and illustrates these categories through representative architectures.

  • Taxonomy: The taxonomy identifies eight integration patterns: early-layer, lateral-layer, sequential, parallel, block, hierarchical, attention-based, and channel boosting.These categories are defined according to the pattern of convolutional integration within ViT architectures.
  • Motivation: HVTs combine CNN locality and inductive bias with transformer modeling of global context and long-range dependencies.This hybridization targets ViTs’ difficulty capturing local correlations and CNNs’ limited global perspective.
  • Representative architectures: Early hybrid designs used CNN feature maps or convolutional stems before transformer processing, as demonstrated by ViT hybrids, DETR, and LeViT.DETR uses a CNN backbone before the ViT encoder, while LeViT uses initial convolution layers to reduce resolution and obtain local features.
  • Representative architectures: CPVT uses depth-wise convolutions in positional encoding generators to make positional embeddings more local and translation-equivalent.Its global average pooling layer above the final MLP improved performance relative to using a class token.
  • Representative architectures: Other HVTs integrate convolution sequentially, within attention, or through complementary pathways, including CoAtNet, CvT, ViTAE, CeiT, and CB-HVT.Reported designs include convolutional blocks followed by transformer blocks, convolution-enhanced attention, and CNN-transformer feature exchange.

3.7. Empirical comparison of different methods

This section compares ViT and HVT architectures using benchmark performance, strengths, weaknesses, primary modifications, and their underlying rationale. The comparison reports the best-performing variants of the listed architectures.

  • Comparison scope: Tables 3 and 4 compare ViT and HVT architectures by strengths, weaknesses, rationale, modifications, and benchmark performance.The reported results correspond to the best-performing variants of the mentioned architectures.
  • Strengths and limitations: The comparisons show recurring trade-offs: hybrid designs combine local convolutional features with global self-attention, while complex attention, multi-scale structures, or large search spaces can increase computational or training difficulty.Reported limitations include higher memory or computational cost, implementation complexity, and greater training requirements.

EA-AA-ResNet

EA-AA-ResNet improves feature representation through evolving attention and convolutional operations, while achieving strong Top-1 accuracy at 224×224 and 384×384 resolutions. Its limitations include increased computational cost and complexity compared with traditional CNNs.

  • Method: EA-AA-ResNet uses evolving attention and convolutional operations to improve feature representation.The passages also mention improving local features and incorporating convolutional components.
  • Limitations: The architecture may be computationally more expensive than traditional CNN-based approaches.The passages also identify increased computational cost and potentially higher model complexity.
  • Results: 83.6% Top-1 Acc @ 224x224 is reported for the architecture.
  • Results: 83.3% Top-1 Acc @ 384x384 is reported for the architecture.

4. Applications of ViTs and HVTs

ViTs and HVTs are increasingly applied across recognition, generation, restoration, segmentation, detection, medical imaging, and pose estimation. Across these applications, hybrid designs combine CNN-based local representations with transformer-based global or contextual information.

  • Overview: ViTs and HVTs have gained prominence across image and video recognition, object detection, segmentation, image restoration, and medical image analysis.The section identifies these as major application areas for both architecture families.
  • Recognition: Hybrid recognition models combine ViT global information with CNN local representations to improve fine-grained 3D object recognition.The described multimodal approach uses both RGB and depth views, with global encoding from ViT and local representation from CNN.
  • Recognition: HA-ViT integrates depth-wise convolution and convolution-based MSA in parallel to combine local and global features for face-periocular cross identification.The methodology outperforms three benchmark datasets in FPCI accuracy.
  • Image generation and restoration: ViT- and HVT-based GANs support image generation, while hybrid generators combine UNet or CNN components with transformer bottlenecks or layers.Reported applications include general image generation, MRI reconstruction, remote-sensing reconstruction, facial reconstruction, and medical image generation.
  • Other vision tasks: Hybrid CNN-transformer methods are used for segmentation, restoration, detection, and pose estimation, with reported gains over CNN-only, transformer-only, or existing approaches.The supplied results mention tunnel-crack segmentation, ADE20K, image restoration, super-resolution, denoising, medical segmentation, object detection, and human-joint estimation.

5. Challenges

HVTs face challenges in combining convolution and self-attention while maintaining efficient, interpretable, and broadly deployable vision processing. Key limitations include computational demands, difficult feature fusion, weak equivariance, and substantial data requirements.

  • Equivariance and invariance: Equipping ViTs with inductive biases that represent objects and video frames with appropriate rotation equivariance or invariance remains complex.Deep learning models have difficulty maintaining the equivariance and invariance that humans use to understand the world in context.
  • Computational complexity: Dense matrix multiplication in MSA and convolution can create high computational complexity and memory overhead, hindering dense applications such as volumetric analysis and segmentation.These costs affect both ViT and HVT architectures.
  • Interpretability: ViT interpretability is difficult because multi-head attention blends attention weights across layers, obscuring which input-token segments drive decisions.The intricate architecture makes decision-making processes difficult to comprehend.
  • Feature fusion: HVTs must efficiently merge transformer layers’ spatially independent global features with convolutional layers’ spatially correlated local features.Unifying MSA and CNN layers may improve performance across vision tasks, but efficient integration remains challenging.
  • Data requirements: High learning capacity creates a need for large training datasets, making HVT development difficult in medical imaging where annotated data is scarce and time-consuming to obtain.Extensive labeled data consumes valuable resources and time and can impede medical-imaging applications.

6. Future directions

The section identifies interpretability, lightweight deployment, and principled CNN–self-attention integration as key future directions for ViTs and HVTs. It also highlights knowledge distillation and new hybrid operators or blocks as promising approaches.

  • Interpretability: Future research should improve ViT interpretability through feature visualization, activation maps, and Grad-CAM to reveal decision-making, biases, and errors.These methods are intended to make models less of a black box and improve transparency.
  • Efficiency and deployment: ViTs and HVTs need lightweight, efficient architectures that reduce inference latency and energy overhead for edge devices and computationally limited systems.Knowledge distillation can produce data-efficient compact models by transferring knowledge from high-capacity models to simpler ones.
  • CNN–self-attention integration: Research should analyze how convolution and self-attention can be integrated for specific applications through early, lateral, sequential, parallel, hierarchical, and attention-based designs.The section emphasizes selecting integration methods according to the context and vision task.
  • Hybrid architecture design: Future HVT development may combine hand-crafted operators with hybrid and dynamic feature extraction, while designing effective blocks using convolution and self-attention.Understanding image content and associated operations is presented as important for devising better hybrid and deep architectures.
  • Expanded capabilities: ViT and HVT architectures should also investigate multimodality and multitasking alongside improved integration methods and convolution–self-attention blocks.These directions are proposed to extend the potential applications of hybrid vision transformers.

7. Conclusion

The survey reviews ViT fundamentals and taxonomy, emphasizing hybrid vision transformers (HVTs/CNN-Transformers) that combine self-attention and convolution to model global and local image relations. It also categorizes HVT integration methodologies and surveys their computer vision applications and potential.

  • Vision Transformer limitations: ViTs model global interactions through their integrated multi-head self-attention module but may generalize poorly because they lack inductive bias for local image relations.The survey attributes ViTs’ performance to global interaction modeling while identifying limited local-relation modeling as a limitation.
  • Vision Transformer taxonomy: ViT architectural improvements are categorized into patch-based, knowledge distillation-based, attention-based, multi-transformer-based, and hybrid approaches.The paper examines both the taxonomy of ViT architectures and the fundamental concepts underlying them.
  • Hybrid vision transformers: HVTs, also known as CNN-Transformers, combine self-attention and convolution mechanisms to learn local and global representations.These architectures address ViTs’ limited local inductive bias by integrating convolution with self-attention.
  • HVT integration methodologies: HVT architectures are taxonomized by integration methodology, including early-layer, lateral-layer, sequential, parallel, hierarchical, and channel boosting-based integration.The survey presents a taxonomy based on how convolution-specific inductive bias is integrated into transformers.
  • Applications and potential: The survey overviews HVT use in real-world computer vision applications and identifies substantial potential from their ability to learn at both local and global levels.This potential is noted despite current challenges.
Loading 2305.09880v4…