A survey of the Vision Transformers and their CNN-Transformer based Variants

Convolutional Neural Networks

• CNNs learn local feature motifs through convolutional filters and use pooling for dimensionality reduction and robustness, but local processing can lose spatial correlation.This limitation affects performance on larger and more complex patterns.Source: §1 · p. 2

Vision Transformers

• ViTs divide images into patches and use transformer layers with self-attention to model long-range relationships and identify globally distributed visual patterns.Their global receptive field can outperform CNNs in various applications, but ViTs depend more heavily on large datasets and computation.Source: §1 · p. 3

Motivation for Hybrid Vision Transformers

• HVTs combine CNN layers for local features with ViT self-attention for global context, improving image-recognition performance.This hybrid design addresses CNNs’ loss of spatial correlation for complex patterns and ViTs’ limited performance with small training data and high computational demands.Source: §1 · pp. 2, 4

Survey Contributions

• The survey emphasizes HVTs, introduces taxonomies for general ViTs and HVTs, and classifies emerging architectures according to their core designs and convolution–self-attention composition.The HVT taxonomy contains seven major groups, while reported ViT models are broadly classified into six categories.Source: §1 · pp. 4–5

Scope and Organization

• The survey covers HVT architectures and applications across computer-vision tasks, emphasizing how convolutional operations and multi-attention mechanisms jointly provide their advantages.It also organizes the paper around ViT fundamentals, ViT and HVT taxonomies, HVT usage, challenges, future directions, and conclusions.Source: §1 · p. 5

2.1. Patch embedding

• ViT divides images into fixed-size patches, flattens and linearly projects them into D-dimensional token embeddings, then processes the sequence through transformer blocks for prediction.For an image of size A × B × C, N = A/P × B/P patches and D = P × P × C.Source: §2.1 · p. 9

2.2. Positional embedding

• Positional embeddings preserve patch order, using absolute embeddings added to tokens, relative embeddings based on interpatch distances, or convolutional embeddings that exploit 2D structure.RPE supports variable-length sequences but may increase training and testing time, whereas CPE can be inserted at multiple stages, including attention, FFN, or between encoder layers.Source: §2.2 · p. 9, §2.2.1 · pp. 9–10, §2.2.2 · p. 10

2.3. Attention Mechanism

• Self-attention represents relationships among sequence elements by transforming inputs into query, key, and value spaces and computing context-dependent weighted value combinations.Its dynamically computed filters adapt to input context and remain robust to changes in the number or permutation of input points, unlike static convolutional filters.Source: §2.3 · p. 11, §2.3.1 · p. 12

2.3.1. Multi-Head Self-Attention (MSA)

• Multi-head self-attention addresses single-head limitations by stacking parallel attention blocks that learn diverse query, key, and value subspaces and capture multiple interactions.The block outputs are concatenated and projected into the output space using a learnable parameter W_O.Source: §2.3.1 · p. 11

2.4. Transformer layers

• A ViT encoder repeatedly combines MSA, an FFN, residual connections, and normalization to learn complex representations of the input sequence.The FFN uses fully connected layers with nonlinear activation such as GELU and typically has a hidden dimensionality of 2048.Source: §2.4 · p. 12, §2.4.1 · p. 13

2.4.2. Residual connection

• Residual links add sublayer inputs to outputs to strengthen information flow, after which layer normalization is applied within the encoder or decoder block.Pre-LN places normalization before MSA or FFN inside the residual connection, while batch normalization may be less efficient because feature values change.Source: §2.4.2 · p. 13, §2.4.3 · pp. 13–14

2.5. Hybrid Vision Transformers (CNN-Transformer Architectures)

• ViTs’ lack of CNN-like image-specific inductive bias motivates hybrid CNN-Transformer architectures that combine transformer modeling with locality, translational equivariance, and 2D neighborhood structure.CNNs encode these properties through shared weights across spatial locations, including translation and scale invariance.Source: §2.5 · p. 14
• Hybrid designs address ViTs’ limited local-correlation modeling by combining convolutional local features with self-attention-based global relationships.Convolutions leverage neighboring-pixel correlations, whereas ViTs process linear patches through encoder blocks to model global relationships but are less effective at local correlation.Source: §2.5 · pp. 14–15
• CvT uses convolutional projections and hierarchical token reduction, while CeiT and CCT apply convolution during image-to-token conversion.These mechanisms capture spatial and low-level information while progressively reducing token numbers or performing tokenization through conv-pool-reshape blocks.Source: §2.5 · pp. 14–15
• CCT achieved about 95% accuracy on CIFAR10 when trained from scratch, a result generally difficult for traditional ViTs to achieve.CCT integrates conv-pool-reshape blocks for tokenization and uses sequence pooling.Source: §2.5 · pp. 14–15
• LocalViT, LeViT, and ResT enhance local modeling or efficiency using depthwise or CNN blocks, with ResT additionally using adaptive position encoding for varying image sizes.LeViT’s initial four-layer CNN block gradually increases channels to improve inference efficiency.Source: §2.5 · p. 15
• Recent hybrids combine depthwise convolution with relative or conditional positional mechanisms, cross-attention, shuffling, or multi-scale token interactions.Examples include CoAtNet, Shuffle Transformer, CoaT, and Twins, which respectively target ImageNet accuracy, stronger cross-patch connections, cross-scale token relationships, or conditional position encoding.Source: §2.5 · p. 15
• MaxViT uses MBConv followed by block-wise and grid-wise self-attention, extracting local and global features while producing hierarchical representations for image generation and segmentation.Its multi-axis attention alternates block-wise and grid-wise processing within a repeated hybrid block.Source: §2.5 · p. 15

3. Architectural level modifications in ViTs

• ViT modifications are classified into patch-based, knowledge transfer-based, shifted window-based, attention-based, and multi-transformer-based approaches, while HVTs are categorized into seven structural types.The taxonomy of ViT architectures is presented in Fig. 4.Source: §3 · p. 16

3.1. Patch-based approaches

• ViT divides images into patches, flattens them into token embeddings, adds positional and class tokens, and feeds the resulting sequence to an encoder.Patch extraction variants include fixed-size, dynamic, and overlapping patching.Source: §3.1 · p. 19

3.1.1. Tokens-to-Token Vision Transformer (T2T-ViT)

• T2T-ViT (Yuan et al. 2021b) iteratively generates fixed-size patches through a Token-to-Token module before producing final predictions.The generated patches are passed to the T2T-ViT network.Source: §3.1.1 · p. 19

3.1.2. Transformer in Transformer (TNT-ViT)

• TNT-ViT (Han et al. 2021) uses multi-level patching, modeling relationships between image patches and their sub-patches with different transformer blocks.Extensive experiments showed TNT-ViT’s efficiency for ImageNet image classification.Source: §3.1.2 · p. 20

3.1.3. Deformable Patch-based Transformer (DPT)

• DPT introduces the adaptive DePatch module (Chen et al. 2021e), which produces variable-sized patches with strong semantic information instead of fixed-size patches.The approach addresses semantic-information loss associated with fixed-size transformer patching.Source: §3.1.3 · p. 20

3.1.4. CrowdFormer

• CrowdFormer (Yang et al. 2022b) uses overlapping patches from a sliding window to retain relative contextual information for effective crowd counting across image scales.Its overlap patching transformer block captures global crowd context in a top-down manner.Source: §3.1.4 · p. 20

3.2. Knowledge transfer-based approaches

• Knowledge distillation transfers knowledge from a complex teacher network to a simpler student model, which can be deployed more efficiently when computational resources are limited.The student learns and incorporates the teacher’s distinctive features rather than reproducing the teacher’s full complexity.Source: §3.2 · p. 21

3.2.1. Data-efficient Image Transformers (DeiT)

• DeiT uses a pre-trained ViT teacher to supervise a smaller student through combined supervised and unsupervised learning, achieving competitive performance with fast inference and improved generalization.The student captures important data features and patterns instead of merely memorizing the training data.Source: §3.2.1 · p. 21

3.2.2. Target-aware Transformer (TaT)

• TaT transfers teacher information to student features through one-to-many relations, averaging features within patches instead of correlating all spatial regions.Feature maps are divided into patches, and each patch’s teacher features are transferred to all corresponding student features before averaging into one vector.Source: §3.2.2 · pp. 21–22

3.2.3. Tiny Vision Transformer (TinyViT)

• TinyViT introduces fast distillation by transferring pre-trained large-model features through stored logits and encoded augmentations, with a decoder reconstructing the saved augmentations.The independently trained student and instructor transfer knowledge via output logits, and TinyViT is effective on large-scale test sets.Source: §3.2.3 · p. 22

Swin Transformer

• Swin Transformer introduced shifted windowing, computing self-attention within non-overlapping local windows while preserving cross-window connections to improve global relationship modeling (Liu et al. 2021c).It applies regular window attention followed by attention over shifted windows in two consecutive blocks.Source: §3.3 · p. 23

CSWinTT

• CSWinTT applies cyclic shifting window-based attention at multiple scales to enhance pixel-to-window attention and aggregate cross-window information for visual object tracking (Song et al. 2022b).Multi-scale aggregation preserves tracking-object integrity and generates fine-scale target matches.Source: §3.3 · p. 24
• Positional information in CSWinTT improves handling of object-position changes over time, supporting more accurate and efficient visual object tracking (Song et al. 2022b).The architecture is reported to improve both tracking accuracy and efficiency.Source: §3.3 · p. 24

3.4. Attention-based approaches

• Dense and sparse attention mechanisms capture global-level image dependencies without spatial correlation, but they are computationally expensive.Recent work also targets improvements in attention performance and computational complexity.Source: §3.4 · p. 24

3.4.1. Class attention layer (CaiT)

• CaiT delays class-token integration until later class-attention layers, and reports good results with few modifications (Touvron et al. 2021).Its self-attention module otherwise follows a normal ViT architecture.Source: §3.4.1 · pp. 24–25

3.4.2. Deformable attention transformer (DAT)

• DAT uses data-dependent attention to focus on reliable regions, combining local and deformable attention layers within each stage, and achieves exemplary benchmark performance (Xia et al. 2022).Each stage contains a local attention layer followed by a deformable attention layer.Source: §3.4.2 · p. 25

3.4.3. Patch-based Separable Transformer (SeT)

• SeT combines pixel-wise attention for initial local interactions with later patch-wise attention for global information, achieving superior results across datasets including ImageNet and MS COCO (Sun et al. 2022).Its architecture introduces two diverse attention blocks in the MSA layer (Fig. 7).Source: §3.4.3 · pp. 25–26, Figure 7 · p. 25

3.5. Multi-transformer-based approaches

• Multi-transformer-based approaches employ multiple ViTs to improve performance on tasks requiring multi-scale features.The section covers architectures that use multiple transformer branches, views, or paths to obtain complementary representations.Source: §3.5 · p. 26

3.5.1. Cross Vision Transformer (CrossViT)

• CrossViT (Chen et al. 2021a) combines different-sized image patches in separate branches and repeatedly fuses them through efficient cross-attention for linear attention-map generation.The branches have different computational complexities, exchange knowledge through a non-patch token, and produce domain-relevant features.Source: §3.5.1 · p. 26

3.5.2. Dual Vision Transformer (Dual-ViT)

• Dual-ViT (Yao et al.) reduces self-attention cost through parallel semantic and pixel pathways that capture global coarse information and local fine details.The branches are integrated and trained in parallel, achieving good ImageNet results with fewer parameters than existing models.Source: §3.5.2 · pp. 26–27

3.5.3. Multiscale Multiview Vision Transformer (MMViT)

• MMViT processes multiple views at multiple resolutions in parallel, using cross-attention at each scale to merge perspectives into complex, robust representations.It incorporates multiscale feature maps and multiview encodings through several feature-extraction stages.Source: §3.5.3 · p. 27

3.5.4. Multi-Path Vision Transformer (MPViT)

• MPViT demonstrated superior ImageNet results by using CNN-generated multi-scale feature maps, multiple transformer encoders, and aggregated outputs across paths.Its multi-scale patching and multi-path architecture process patch embeddings at different scales.Source: §3.5.4 · pp. 27–28

3.6. Taxonomy of HVTs (CNN-Transformer architectures)

• HVTs combine CNNs’ local-pattern modeling with transformers’ global-context modeling to address ViTs’ limited locality, expensive MSA, and fixed-sized input embeddings.The survey identifies eight integration patterns: early-layer, lateral-layer, sequential, parallel, block, hierarchical, attention-based, and channel boosting integration.Source: §3.6 · p. 28

3.6.1. Early-layer integration

• Early-layer integration applies CNN processing before transformer encoding, as in ViT hybrids, DETR, LeViT, CPVT, and convolutional-stem variants.This ordering first captures local patterns and then learns long-range dependencies; DETR uses CNN features with positional embeddings, LeViT combines an initial four-layer CNN with MSA, and CPVT uses depth-wise-convolutional PEGs for more local, translationally equivalent positional embeddings.Source: §3.6.1 · pp. 29–30

3.6.2. Lateral-layer integration

• Lateral-layer integration places CNN processing at the transformer’s end or in post-processing, exemplified by DPT’s ViT encoder-CNN decoder and LocalViT’s depth-wise-convolutional FFN.DPT decodes ViT representations at different levels for dense prediction, while LocalViT incorporates local information into the feed-forward network using depth-wise convolution.Source: §3.6.2 · p. 32

3.6.3. Sequential integration

• Sequential integration vertically stacks convolutional and transformer blocks, with CoAtNet using MBConv blocks and reporting efficient results from two convolutional blocks followed by two transformer blocks.CoAtNet was designed to merge convolution and attention to increase generalization and capacity, while CMT was proposed as another hybrid ViT to address ViTs’ locality, MSA cost, and fixed-size input limitations.Source: §3.6.3 · p. 33
• Sequential integration stacks CNN and transformer components, exemplified by CMT’s lightweight MSA, IRFFN, and LPU and BoTNet’s MSA-equipped final ResNet blocks.BoTNet replaces the 3x3 convolution with MSA to capture long-term dependencies alongside local features, while CMT adds an LPU to increase representation capacity.Source: §3.6.3 · pp. 33–34

3.6.4. Parallel integration

• Parallel integration runs CNN and transformer pathways concurrently and combines their predictions, as in Conformer, Mobile-Former, and BossNAS.Mobile-Former synchronizes branches so CNN features retain global information and transformer features retain local information; BossNAS searches parallel blocks across resolutions and forms.Source: §3.6.4 · pp. 34–35
• Conformer uses parallel CNN and transformer branches to capture local and global features, achieving better performance than DeiT and ViT.Connections make both branches local-global context-aware, and separate CNN and transformer classifiers produce the final predictions.Source: §3.6.4 · p. 35

3.6.5. Hierarchical integration

• Hierarchical HVTs repeat unified CNN–ViT blocks across stages, as illustrated by MaxViT, Visformer, ViTAE, and ConTNet.These designs use CNN-like multi-stage backbones or stacked hybrid blocks for computer vision tasks.Source: §3.6.5 · pp. 36, 39
• MaxViT combines MBConv with blocked local and dilated global attention, providing efficient, scalable attention and local-global visibility across stages.Its hybrid block is repeated over multiple stages for classification, detection, segmentation, and generative modeling.Source: §3.6.5 · pp. 36–37
• CvT uses hierarchical stages, convolutional token embeddings, and depth-wise separable projections, and outperformed other approaches for image recognition.Convolution adds locality, progressively shortens sequence length, and replaces linear projections before self-attention blocks.Source: §3.6.5 · p. 37

3.6.6. Attention-based integration

• Attention-based integration incorporates CNNs directly into attention mechanisms to add locality, motivated by limited generalizability of independent self-attention layers.This category includes architectures such as EA-AA-ResNet.Source: §3.6.6 · p. 39
• Evolving Attention in EA-AA-ResNet bridges attention maps across layers and uses convolutional modules to learn general attention patterns.The design exploits knowledge inherited from previous layers through residual connections.Source: §3.6.6 · pp. 39–40
• ResT integrates convolution into attention to capture global and local features, using depth-wise convolution to reduce token-map spatial dimensions before efficient self-attention.Its efficient transformer block replaces conventional MSA with an efficient variant.Source: §3.6.6 · p. 40
• CeiT combines CNN and ViT benefits through Image-to-Tokens, Locally-enhanced Feed-Forward, and class-token attention modules, achieving promising recognition results while being computationally efficient versus ViT.Its CNN-based patch extraction uses low-level features, while depth-wise convolutions capture spatial correlation and LCA combines outputs across ViT layers.Source: §3.6.6 · pp. 40–41

3.6.7. Channel boosting-based integration

• CB-HVT generates boosted channels with CNN and ViT-based auxiliary learners, capturing local diversity through CNNs and global context through PVT-based channels.It showed reasonable performance on a lymphocyte assessment dataset; Figure 16 depicts PVT combined within a CNN through channel boosting.Source: §3.6.7 · p. 41, Figure 16 · p. 42

Empirical comparison of ViT architectures

• The cited passages report Top-1 accuracies of 85.2% at 384x384, 72.41% at 513x513, and 86.5% at 512x512, without identifying corresponding models.A teacher-student model and one-to-many knowledge transfer are mentioned.Source: §3.7 · p. 44

Shifted Window-Based Approaches

• 84.5% Top-1 Acc @ 384x384 is reported for Swin Transformer (Liu et al. 2021c), whose shifted-window scheme enables efficient computation and cross-window connections.Its computational expense may limit applicability.Source: §3.7 · p. 44

Attention-Based Approaches

• 86.5% Top-1 Acc @ 448x448 is reported for CaiT, while DAT (Xia et al. 2022) reaches 84.8% Top-1 Acc @ 384x384 using deformable attention.The attention-based approaches emphasize improved attention or local image-structure modeling, with deformable attention potentially introducing computational costs.Source: §3.7 · p. 44

Multi-Transformer-Based Approaches

• The cited results report 82.8% Top-1 Acc @ 224x224 and 83.4% Top-1 Acc @ 384x384, with multi-scale patch embedding and multi-path structure mentioned.Source: §3.7 · p. 45

Hybrid Vision Transformer architectures

• 44.9% Box mAP @ 384x384 is reported for DETR (Carion et al.), demonstrating transformer effectiveness for dense prediction without separate region proposal networks.The approach remains computationally expensive, especially for dense prediction tasks.Source: §3.7 · p. 46
• CoAtNet (Dai et al. 2021) reports 86.0% Top-1 Acc @ 512x512, while CMT (Guo et al. 2021), with lightweight MSA, reaches 84.8% Top-1 Acc @ 288x288.Source: §3.7 · p. 46

Architectural strategies

• Attention-based and convolutional variants introduce mechanisms including bottlenecks, cross-attention, multi-axis attention, hierarchical processing, and residual convolutions.These architectural choices target efficiency, local-global interaction, linear-time feature extraction, or improved generalization.Source: §3.7 · pp. 47–48

Empirical results

• Reported results span 79.3%–87.7% Top-1 accuracy across 224x224, 384x384, and 512x512 inputs.The listed results are 84.7% and 84.1% at 224x224, 79.3% at 224x224, 87.7% at 384x384, and 82.5% and 86.70% at 512x512.Source: §3.7 · p. 47

Limitations

• Reported limitations include complex or expensive training, difficulty searching large hybrid spaces, configuration sensitivity, increased model complexity, and greater computational cost.Some approaches may also require more data or fail to capture all interactions between local and global representations.Source: §3.7 · p. 47

Hybrid mechanisms

• Hybrid designs combine convolution and self-attention to fuse local features with global representations and mitigate their respective weaknesses.The passages describe efficient fusion, bidirectional fusion, and a balance between efficiency and representation power.Source: §3.7 · p. 47

Generalization

• Inductive-bias-aware designs are reported as improving generalization and robustness to input changes, although their effectiveness can depend on the task.The passages also note that complexity may increase and that some configurations may require more data for good results.Source: §3.7 · pp. 47–48

• The approach improves feature representation through evolving attention and convolutional operations.Source: EA-AA-ResNet · p. 48
• 79.63% Top-1 Acc @ 224x224 is also reported.Source: EA-AA-ResNet · p. 48
• The architectures may incur increased computational cost or model complexity compared with traditional CNNs.Source: EA-AA-ResNet · p. 48
• 83.6% Top-1 Acc @ 224x224 is reported for one evaluated model.Source: EA-AA-ResNet · p. 48
• Convolutional components are added to improve local features and support feature extraction in vision transformers.Source: EA-AA-ResNet · p. 48
• 83.3% Top-1 Acc @ 384x384 is reported for another evaluated model.Source: EA-AA-ResNet · p. 48

4. Applications of ViTs and HVTs

• ViTs and HVTs support image and video recognition, object detection, segmentation, image restoration, and medical image analysis, while HVTs combine CNNs and transformers for intricate visual patterns.The survey identifies broad application coverage and describes HVTs as an effective combination of convolutional and transformer modules.Source: §4 · p. 49

4.1. Image/video recognition

• Xiong and Kasaei’s hybrid approach combines ViT and CNN representations, encoding global object information with ViT and local RGB-depth information with CNN, and outperforms CNN-only and ViT-only baselines.Source: §4.1 · p. 50

4.2. Image generation

• A hybrid architecture combining a UNet with a ViT bottleneck demonstrated superior performance compared with earlier best-performing models while retaining a strong correlation between original and generated images.Source: §4.2 · p. 51

4.3. Image segmentation

• HVT segmentation methods combine global transformer features with local CNN features and outperform competing CNN-only, transformer-only, or other techniques on tunnel cracks, ADE20K, and Cityscapes.SCDeepLab outperformed CNN-only and transformer-only models for tunnel-crack segmentation, while Mask2Former achieved better results than other techniques on ADE20K and Cityscapes (Zhou et al. 2023b; Xia and Kim 2023).Source: §4.3 · p. 52

4.4. Image Restoration

• Image restoration has shifted from CNNs toward ViTs and HVTs, with Dual-former delivering superior performance on multiple restoration tasks while using significantly fewer GFLOPs.Dual-former combines convolutional local modeling with self-attention global modeling (Chen et al. 2022d). Hybrid convolution-transformer blocks, multi-stage training, and a mixed error function also support infrared deblurring (Yi et al. 2023).Source: §4.4 · p. 53
• Hybrid restoration models combine CNN-derived local features with transformer-based non-local or global information, improving super-resolution, denoising, and infrared-visible fusion while controlling complexity.HNCT reports improved super-resolution results with fewer parameters, TECDNet achieves strong real-image denoising at relatively low computational cost, and the fusion method uses a specialized loss to retain textural and spatial information.Source: §4.4 · pp. 53–54

4.5. Feature extraction

• CNN-Transformer feature extractors achieve good or outstanding performance for crop-disease, recaptured-image, strip-steel-defect, and seafloor classification while supporting real-time computation.The architectures use CNN layers or modules for local features and transformer blocks or attention for global or high-domain representations; the seafloor model replaces the ViT MLP block with CNN-based extraction.Source: §4.5 · p. 54

4.6. Medical image analysis

• CNN-Transformer medical architectures capture complementary local and global information, yielding strong segmentation performance across medical-image applications and benchmark datasets.FCT combines CNN representation learning with transformer long-term dependencies, HiFormer uses a Swin Transformer and CNN encoder for multi-scale features, and TSEDeepLab reports exceptional segmentation accuracy and strong generalization.Source: §4.6 · p. 55

4.7. Object Detection

• Hybrid detectors improve object detection by combining CNN local modeling with transformer long-range or global features across natural, remote-sensing, UAV, and other images.ViT-FRCNN improves detection and generalization, MDCT outperforms other single-stage detectors, the UAV method uses FPN for multi-scale learning, and a fusion network reports satisfactory comparative performance.Source: §4.7 · pp. 56–57

4.8. Pose Estimation

• Pose-estimation systems increasingly integrate CNN local features with transformer global dependencies, and reported hybrid methods improve or achieve exemplary human-joint estimation results.DPIT uses two CNN branches followed by transformer encoder blocks, while Keypoint Transformer combines CNN and transformer modules to estimate human joints as 2D keypoints (Hampali et al. 2021).Source: §4.8 · p. 57
• The approach was evaluated on the InterHand2.6M dataset.Source: §4.8 · p. 58

• ViTs struggle to preserve the rotation invariance and equivariance needed to represent objects and video frames effectively.Designing appropriate inductive biases for equivariant representations remains complex.Source: §5 · p. 58
• Dense matrix multiplication in MSA and convolution creates high computational and memory overhead, hindering volumetric analysis and segmentation.Both ViTs and HVTs may therefore struggle with dense applications.Source: §5 · p. 58
• Training ViTs and HVTs requires powerful hardware, limiting deployment on edge devices because of hardware constraints and associated costs.Source: §5 · p. 58
• ViT interpretability is difficult because multi-head attention blends weights across layers, obscuring which input tokens drive decisions.Source: §5 · p. 59
• HVTs must efficiently merge transformer global features with convolutional spatially correlated features while unifying MSA and CNN layers.Effective fusion could improve performance across vision tasks, but remains a major architectural challenge.Source: §5 · p. 59
• ViTs and HVTs require large training datasets, making medical-imaging development difficult because annotated data are scarce, costly, and time-consuming to obtain.Source: §5 · p. 59

Interpretability

• Interpretability methods such as feature visualization, activation maps, and Grad-CAM can clarify ViT decisions, improve transparency, and help identify biases and errors.Understanding model behavior supports more effective model development and deployment.Source: §6 · pp. 59–60

Efficient architectures

• Lightweight ViT and HVT designs are needed to reduce inference latency and energy overhead for practical deployment on edge devices and computationally limited systems.Knowledge distillation may produce data-efficient compact models by transferring knowledge from high-capacity models to simpler ones.Source: §6 · p. 60

CNN-transformer integration

• Research should analyze how convolution and self-attention are integrated—including early, lateral, sequential, parallel, hierarchical, and attention-based approaches—for specific vision contexts.HVTs combine CNN and transformer strengths, but their potential depends on selecting suitable integration methods.Source: §6 · p. 60

Architectural extensions

• Future HVTs may benefit from deeper image-content understanding, hand-crafted operators, dynamic feature extraction, and new blocks combining convolution with self-attention.Multimodality and multitasking are also proposed for ViT and HVT architectures.Source: §6 · pp. 60–61

• ViTs model global image interactions through multi-head self-attention but may generalize poorly when they lack inductive bias for local relations.These limitations motivate architectural improvements across patch-based, knowledge distillation-based, attention-based, multi-transformer-based, and hybrid approaches.Source: §7 · p. 61
• The paper develops a ViT architectural taxonomy and examines the fundamental concepts underlying ViT architectures.The taxonomy includes patch-based, knowledge distillation-based, attention-based, multi-transformer-based, and hybrid approaches.Source: §7 · p. 61
• HVTs, or CNN-Transformers, combine self-attention and convolution to learn local and global image representations.Source: §7 · pp. 61–62
• HVT architectures are categorized by convolution-integration methodology, including early-layer, lateral-layer, sequential, parallel, hierarchical, and channel boosting-based integration.Source: §7 · p. 62
• The survey reviews HVT use in real-world computer vision applications and highlights their potential to learn at both local and global levels despite current challenges.Source: §7 · p. 62