A Survey of Visual Transformers

Motivation

• Visual Transformers emerged from Transformer successes in NLP and subsequent efforts to model long-range dependencies in computer vision.Prior approaches added spatial or channel self-attention to CNNs, or replaced convolutions with global or local self-attention blocks.Source: §I · p. 1

Scope

• The survey categorizes visual Transformers across classification, detection, segmentation, and image, point-cloud, and multi-stream data.Methods are grouped by vision task, data-stream type, motivation, and structural characteristics.Source: §I · pp. 1–2

Survey design

• It reviews over one hundred models, selecting representative methods for detailed descriptions while briefly introducing related works.The review also builds connections through progressive, contrastive, and multi-view analyses.Source: §I · p. 2

Evaluation

• Because methods use different training schemes and hyper-parameters, the survey compares them laterally across datasets and restrictions.It also summarizes task-specific components such as hierarchical local convolution, sparse attention, and general-purpose mask prediction.Source: §I · p. 2

Analysis

• The survey examines why visual Transformers work in vision, their correspondence with other neural networks, their double edges, and their learnable embeddings.It identifies query embeddings as a possible mechanism for unifying multiple visual tasks and data-stream types within an encoder-decoder backbone.Source: §I · p. 2

Scaled Dot-Product Attention

• The transformation layer maps input sequences X and Y into query Q, key K, and value V vectors using linear matrices.The paper identifies n and d as input length and dimension, respectively.Source: A. (Multi-Head) Attention Mechanism · pp. 2–3

Attention variants

• When Y = X, the mechanism is self-attention; otherwise, the two inputs form a cross-attention mechanism.Self-attention is used in both encoder and decoder, while cross-attention connects the decoder to encoder outputs.Source: A. (Multi-Head) Attention Mechanism · p. 3

Scaled Dot-Product Attention

• Attention computes query-key dot products, scales them by √d_k, applies softmax, and uses the normalized weights to aggregate value elements.This produces the final output vector from the corresponding values.Source: A. (Multi-Head) Attention Mechanism · p. 3

Multi-Head Attention

• Multi-head self-attention projects inputs into multiple feature subspaces and processes independent attention heads in parallel.The resulting head vectors are concatenated and mapped to the final output.Source: A. (Multi-Head) Attention Mechanism · pp. 3–4
• Each of h heads uses d_model/h-dimensional vectors, enriching feature-subspace diversity without extra costs.The head outputs are integrated dependently after parallel processing.Source: A. (Multi-Head) Attention Mechanism · p. 4

• Each encoder block feeds multi-head self-attention output into successive feed-forward networks with ReLU activation.Source: B. Position-wise Feed-Forward Networks · p. 4, D. Transformer Model · p. 4
• The position-wise feed-forward layer treats every sequence position equally while using different parameters between layers.It can be viewed as a point-wise convolution.Source: B. Position-wise Feed-Forward Networks · p. 4

• SANet replaces spatial 3 × 3 convolutions in ResNet bottlenecks with locally spatial self-attention.Source: A. Original Visual Transformer · p. 5
• A fully vanilla Transformer with quadratic positional encoding can approximate convolution through multi-head self-attention with relative positional encoding.Source: A. Original Visual Transformer · p. 5
• ViT splits images into non-overlapping patches, projects them into embeddings, adds learnable positional encoding, and processes them with a Transformer.Source: A. Original Visual Transformer · p. 5, Figure 6 · p. 5
• Hybrid models such as BoTNet integrate multi-head self-attention into CNN bottlenecks and outperform similarly parameterized CNNs on ImageNet.Source: B. Transformer-enhanced CNNs · p. 5
• DeiT transfers convolutional inductive bias through teacher-student distillation, reaching 85.2% top-1 accuracy on ImageNet-1k without external data.Source: C. CNN-enhanced Transformer · p. 5
• Swin uses shifted local windows to enable cross-window interaction while reducing one-layer attention complexity from O(2n^2C) to O(4M^2nC).Source: D. Local Attention Enhanced Transformer · p. 6, Figure 7 · p. 6

• T2T-ViT introduces hierarchical processing with overlapping unfolding for down-sampling, while PVT is developed to reduce its memory and computation costs.Source: E. Hierarchical Transformer · p. 6
• Deep Transformers face attention collapse and increasingly indistinguishable patch representations as depth increases.Source: F. Deep Transformer · p. 6
• CaiT separates patch self-attention from later class-attention stages, while Patch Diversity losses regularize deep Transformers against patch over-smoothing.Source: F. Deep Transformer · p. 7
• Self-supervised visual Transformer methods include autoregressive pixel prediction, masked latent-space reconstruction, and discriminative teacher-student training.Source: G. Transformers with Self-Supervised… · p. 7
• Basic training strategies such as DeiT and LV-ViT are more universal across models, tasks, and input settings than many structure-specific improvements.Source: 1) Algorithm Evaluation and… · p. 7
• The survey identifies locality as indispensable, with VOLO dominant for classification and Swin dominant for dense prediction.Source: 1) Algorithm Evaluation and… · p. 7
• Convolutional patchify stems and early convolutional stages significantly boost Transformer accuracy, especially for large models.Source: 1) Algorithm Evaluation and… · p. 7
• The survey concludes that hybrid convolution-Transformer designs remain promising because shallow convolutions process locality and semantic features effectively.Source: 2) Brief Discussion on Alternatives · p. 9

• Subsequent DETR variants target accuracy and convergence through sparse attention, spatial priors, structural redesign, assignment optimization, and pretraining.Source: A. Transformer Neck · p. 9
• DETR reformulates object detection as end-to-end set prediction using object queries and one-to-one bipartite matching.Source: 1) The Original Detectors · p. 9
• DETR’s matching-based training removes redundant predictions and eliminates non-maximum suppression from inference.Source: 1) The Original Detectors · p. 9
• DETR achieves competitive COCO performance but suffers from slow convergence and poor performance on small objects.Source: 1) The Original Detectors · p. 9
• Pix2seq treats generic detection as autoregressive language modeling that generates object classes and bounding boxes sequentially.Source: 1) The Original Detectors · p. 9

• Deformable DETR samples a small set of reference points through multiscale deformable attention to improve convergence and detection performance.Source: 2) Transformer with Sparse Attention · p. 9
• ACT dynamically clusters queries into prototypes, while PnP and Sparse DETR selectively condense or update tokens using attention-derived signals.Source: 2) Transformer with Sparse Attention · pp. 9–10
• DETR object queries weakly encode spatial information because they are randomly initialized and only implicitly related to bounding boxes.Source: 3) Transformer with Spatial Prior · p. 10
• SMCA explicitly supplies spatial priors by multiplying Gaussian-like query weight maps with cross-attention maps.Source: 3) Transformer with Spatial Prior · p. 10
• Two-stage methods initialize decoder embeddings from top-k encoder proposals or dense proposals before refining sparse predictions.Source: 3) Transformer with Spatial Prior · p. 10
• TSP removes the decoder and object queries, using an encoder-only design with fixed-size features of interest or proposals.Source: 4) Transformer with Redesigned Structure · p. 10
• UP-DETR pretrains detectors to localize randomly cropped image patches using query assignment and auxiliary reconstruction loss.Source: 6) Transformer Detector with… · p. 11

Backbone designs

• Hierarchical backbones such as PVT construct high-to-low resolution processes to learn multi-scale features for dense prediction.Source: B. Transformer Backbone · p. 11
• Locally enhanced backbones combine local and global modeling to capture short- and long-range dependencies while avoiding quadratic computational overhead.Source: B. Transformer Backbone · p. 11

Neck detectors

• Sparse attention and scoring networks reduce redundant feature interactions, lowering computational costs and accelerating convergence.Source: C. Discussion · p. 11
• Explicit spatial priors, multi-scale decoder updates, and improved bipartite matching support precise, refined, end-to-end detection.Source: C. Discussion · p. 11

Discussion

• Dense prediction has fewer Transformer backbones than classification, motivating cooperation with deep high-resolution networks.Source: C. Discussion · p. 11

Segmentation frameworks

• Segmentation frameworks include transfer learning, multi-task learning, cascade learning, independently supervised queries, and box-free mask prediction.Source: Figure 10 · p. 12

Object Queries

• QueryInst cascades box outputs and shared queries into mask prediction, preserving instance correspondences across stages and obtaining state-of-the-art COCO results.Source: 1) Object Queries · p. 12

Mask Embeddings

• Mask embeddings are supervised only by segmentation, while parallel query sets support separate tasks and box learning can provide auxiliary enhancement.Source: 2) Mask Embeddings · p. 12
• Max-DeepLab directly predicts masks without a box branch and achieves 51.3% PQ on COCO test-dev, but incurs heavy computational cost.Source: 2) Mask Embeddings · p. 13
• Maskformer reformulates semantic segmentation as mask prediction and reports superiority over per-pixel classification while unifying semantic and instance segmentation.Source: 2) Mask Embeddings · p. 13

Discussion

• Visual Transformers with mask embeddings surpass most prevailing models on segmentation and detection, while a large gap remains between box AP and segmentation AP.Source: C. Discussion · p. 13
• Mask prediction enables a box-free framework that can unify multiple segmentation tasks.Source: C. Discussion · p. 13

Cognition Mapping

• 3DETR extends DETR for 3D detection, while Group-Free 3D DETR selects candidate point samples as object queries and iteratively updates them.Source: B. Cognition Mapping · p. 14
• MonoDTR and MonoDETR use auxiliary depth supervision to estimate depth positional encodings that guide Transformer encoding and query initialization.Source: B. Cognition Mapping · p. 14
• PoinTr uses local point proxies and a geometry-aware encoder-decoder to move proxies toward incomplete regions for point-cloud completion.Source: C. Specific Processing · p. 14

Multi-sensory Fusion

• Transformers suit multi-sensory fusion through nonspecific embeddings and dynamically interactive attention across complementary sensor streams.Source: §VII · p. 15
• Homologous streams are fused through interactive or transfer mechanisms, including spatial concatenation and cross-attention across views or modalities.Source: A. Homologous Stream · p. 15, 1) Interactive Fusion · p. 15, 2) Transfer Fusion · p. 15

Visual-Linguistic Pre-Training

• Visual-linguistic pre-training methods combine visual and textual representations under supervised or self-supervised training schemes.Source: B. Heterologous Stream · p. 15
• Early VLP methods use object detectors and text encoders to extract object-centric visual tokens for joint distribution learning.Source: 1) Visual-Linguistic Pre-Training · p. 15
• ViLBERT applies vision-language cross-attention and pre-trains with masked language modeling, masked region classification, and image-text alignment on 3M image-text pairs.Source: 1) Visual-Linguistic Pre-Training · p. 15
• Single-stream models feed visual tokens and text embeddings into cross-modality self-attention for joint representation learning.Source: 1) Visual-Linguistic Pre-Training · pp. 15–16

Contrastive Learning

• CLIP jointly trains image and text encoders with contrastive loss on 400M Internet image-text pairs and supports zero-shot transfer with a linear classifier.Source: 1) Visual-Linguistic Pre-Training · p. 16

Visual Grounding

• Visual grounding methods learn concrete target supervision by locating objects from textual descriptions, including MDETR for phrase grounding.Source: 2) Visual Grounding · p. 16
• 3D grounding methods use language modeling or cross-attention to connect object features, point-cloud representations, and linguistic descriptions.Source: 2) Visual Grounding · pp. 16–17

Classification

• Hierarchical backbones reduce computational complexity and help avoid feature over-smoothing in deep classification layers.Early convolution captures low-level features, while convolutional projections and local attention improve locality.Source: A. Summary of Recent Improvements · p. 17

Detection

• Transformer necks reduce computation in detection, but decoder convergence remains slow and requires stronger spatial priors.Sparse attention and foreground-scoring networks reduce computational costs and accelerate convergence.Source: A. Summary of Recent Improvements · p. 17

Segmentation

• Encoder-decoder models unify segmentation subtasks through learnable mask embeddings and achieve state-of-the-art performance on multiple benchmarks.Cascading a hybrid task with box-based Transformers improves instance segmentation performance.Source: A. Summary of Recent Improvements · p. 17

3D visual recognition

• Visual Transformers efficiently model point-cloud features, aggregate surface points globally, and process multi-sensory 3D data.Local hierarchical designs with scoring networks support efficient feature extraction, while global modeling aggregates surface points.Source: A. Summary of Recent Improvements · p. 17

Visual-linguistic pretraining

• Visual-linguistic pretraining increasingly aligns data streams in latent space using large-scale noisy datasets and supports zero-shot downstream transfer.Mainstream approaches have moved away from pretrained detectors toward alignment or similarity objectives.Source: A. Summary of Recent Improvements · p. 17

Multi-sensory fusion

• Single-stream fusion spatially concatenates multiple data streams, while subsequent methods seek latent spaces with semantic consistency across streams.The survey emphasizes that visual Transformers still require deeper understanding despite substantial evolution.Source: A. Summary of Recent Improvements · p. 17, B. Discussion on Visual Transformers · p. 17

Learnable embeddings

• Learnable class tokens, object queries, and mask embeddings support encoder-only and encoder-decoder Transformer patterns.Learned positional encodings and decoder input embeddings are used in encoder-decoder structures.Source: 4) Learnable Embeddings for Different… · p. 18
• Later class tokens can reduce FLOPs and slightly improve performance, while similar strategies support segmentation efficiency.CaiT and Segmenter are cited as examples of later-token designs.Source: 4) Learnable Embeddings for Different… · p. 18

Future directions

• Multiple class tokens converge consistently under shared loss gradients, unlike dense prediction settings with different label assignments and target counts.The survey proposes set prediction as a natural direction for classification with multiple class tokens.Source: 1) Set Prediction · p. 18
• Self-supervised visual Transformer research seeks to bridge differences between contrastive convolutional approaches and masked auto-encoding paradigms.The passage situates this direction within the broader popularity of self-supervision in computer vision.Source: 2) Self-Supervised Learning · p. 18

Conclusion

• The survey reviews more than one hundred visual Transformer models across tasks, data streams, taxonomies, benchmarks, essential issues, and future directions.Covered data include images, point clouds, image-text pairs, and other multiple data streams.Source: D. Conclusion · p. 19

Development trend

• Early visual Transformers adapted either the vanilla NLP Transformer structure or attention-based computer-vision models.ViT and iGPT exemplify the former, while VTs and BoTNet exemplify the latter.Source: Appendix A · p. 19
• Later methods transferred hierarchical or deep CNN structures into Transformers using varied downsampling and deep-layer designs.T2T-ViT, PVT, CvT, and PiT differ in downsampling, while CaiT, Diverse Patch, DeepViT, and Refiner target deep Transformers.Source: Appendix A · p. 19
• The next development wave introduced locality through local attention or convolution, alongside structural combinations, scaling laws, and self-supervised learning.The passage describes these as major directions in recent supervised and self-supervised visual Transformers.Source: Appendix A · p. 19

DETR formulas

• DETR uses bipartite matching to assign predictions one-to-one with ground-truth objects before applying Hungarian loss.The loss combines negative log-likelihood for label predictions with box loss for matched pairs.Source: Appendix B · p. 19

Deformable DETR

• Deformable DETR samples a fixed number of key points at each feature scale and generates query-dependent attention maps over those samples.Offsets are produced for each query and head, while sampled key features are linearly projected.Source: Figure 14 · p. 19, Appendix C · p. 19

Comparisons

• The appendix compares visual Transformer backbones on COCO dense prediction and organizes visual-linguistic methods by pretraining data, objectives, architecture, and visual-token inputs.Table IV uses RetinaNet and Mask R-CNN with a 3× schedule and multi-scale inputs; Table V distinguishes single- and dual-stream methods and zero-shot transfer.Source: DENSE PREDICTION AND VISUAL-LINGUISTIC · pp. 19, 21, Table IV · p. 20, Table V · p. 20