PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Visual Transformers

Yang Liu, Yao Zhang, Yixin Wang, Feng Hou, Jin Yuan, Jiang Tian, Yang Zhang, Zhongchao Shi, Jianping Fan, Zhiqiang He

arXiv:2111.06091v4cs.CV

TL;DR

Visual Transformers address the need to understand rapidly proliferating Transformer-based approaches for computer vision across tasks and data streams. This survey reviews, categorizes, compares, and analyzes more than one hundred models, finding task-specific design patterns and broader opportunities for unifying visual tasks. It concludes with supported research directions, including bridging visual and sequential Transformers through high-level semantic embeddings.

  • Problem

    The rapid growth of Transformer-based vision models creates a need for systematic identification, categorization, and evaluation across vision tasks and data-stream types.

  • Method

    The survey reviews over one hundred visual Transformers, organizes them by task, data stream, motivation, and structure, and compares them under different datasets and restrictions.

  • Results

    The survey reports that visual Transformers have been applied across classification, detection, segmentation, and multiple data streams, with task-specific design patterns and frameworks that can unify related vision tasks.

  • Takeaways & Limitations

    The analysis highlights high-level semantic embeddings and query embeddings as promising ways to connect visual Transformers with sequential models and unify multiple visual tasks and data streams.

Abstract

from arXiv · show

Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern Convolution Neural Networks (CNNs). In this survey, we have reviewed over one hundred of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where a taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, three promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers.

I. INTRODUCTION

The survey responds to the rapid expansion of visual Transformers by systematically organizing and evaluating methods across core vision tasks and data-stream types. It also analyzes their internal connections, design trade-offs, and future research opportunities.

  • Motivation: Visual Transformers emerged from Transformer successes in NLP and subsequent efforts to model long-range dependencies in computer vision.Prior approaches added spatial or channel self-attention to CNNs, or replaced convolutions with global or local self-attention blocks.
  • Scope: The survey categorizes visual Transformers across classification, detection, segmentation, and image, point-cloud, and multi-stream data.Methods are grouped by vision task, data-stream type, motivation, and structural characteristics.
  • Survey design: It reviews over one hundred models, selecting representative methods for detailed descriptions while briefly introducing related works.The review also builds connections through progressive, contrastive, and multi-view analyses.
  • Evaluation: Because methods use different training schemes and hyper-parameters, the survey compares them laterally across datasets and restrictions.It also summarizes task-specific components such as hierarchical local convolution, sparse attention, and general-purpose mask prediction.
  • Analysis: The survey examines why visual Transformers work in vision, their correspondence with other neural networks, their double edges, and their learnable embeddings.It identifies query embeddings as a possible mechanism for unifying multiple visual tasks and data-stream types within an encoder-decoder backbone.

A. (Multi-Head) Attention Mechanism

The attention mechanism transforms inputs into queries, keys, and values, then aggregates values according to query-key similarity. Multi-head attention repeats this process in parallel feature subspaces and combines the resulting representations.

  • Scaled Dot-Product Attention: The transformation layer maps input sequences X and Y into query Q, key K, and value V vectors using linear matrices.The paper identifies n and d as input length and dimension, respectively.
  • Attention variants: When Y = X, the mechanism is self-attention; otherwise, the two inputs form a cross-attention mechanism.Self-attention is used in both encoder and decoder, while cross-attention connects the decoder to encoder outputs.
  • Scaled Dot-Product Attention: Attention computes query-key dot products, scales them by √d_k, applies softmax, and uses the normalized weights to aggregate value elements.This produces the final output vector from the corresponding values.
  • Multi-Head Attention: Multi-head self-attention projects inputs into multiple feature subspaces and processes independent attention heads in parallel.The resulting head vectors are concatenated and mapped to the final output.
  • Multi-Head Attention: Each of h heads uses d_model/h-dimensional vectors, enriching feature-subspace diversity without extra costs.The head outputs are integrated dependently after parallel processing.

B. Position-wise Feed-Forward Networks

Position-wise feed-forward networks follow multi-head self-attention in each Transformer encoder block. They apply the same position-wise operation across sequence elements while using layer-specific parameters.

  • Each encoder block feeds multi-head self-attention output into successive feed-forward networks with ReLU activation.
  • The position-wise feed-forward layer treats every sequence position equally while using different parameters between layers.It can be viewed as a point-wise convolution.

A. Original Visual Transformer

Original visual Transformers replace or augment convolution with self-attention, culminating in ViT’s patch-based vanilla Transformer and later hybrid and local-attention designs. These approaches target spatial modeling, locality, and computational efficiency in visual recognition.

  • SANet replaces spatial 3 × 3 convolutions in ResNet bottlenecks with locally spatial self-attention.
  • A fully vanilla Transformer with quadratic positional encoding can approximate convolution through multi-head self-attention with relative positional encoding.
  • ViT splits images into non-overlapping patches, projects them into embeddings, adds learnable positional encoding, and processes them with a Transformer.
  • Hybrid models such as BoTNet integrate multi-head self-attention into CNN bottlenecks and outperform similarly parameterized CNNs on ImageNet.
  • DeiT transfers convolutional inductive bias through teacher-student distillation, reaching 85.2% top-1 accuracy on ImageNet-1k without external data.
  • Swin uses shifted local windows to enable cross-window interaction while reducing one-layer attention complexity from O(2n^2C) to O(4M^2nC).

E. Hierarchical Transformer

Hierarchical Transformers address ViT’s fixed-resolution and coarse-patch limitations through multiscale representations, deeper architectures, and local or convolutional inductive biases. Comparative analyses identify training strategies, locality, convolutional stems, and hybrid designs as recurring performance factors.

  • T2T-ViT introduces hierarchical processing with overlapping unfolding for down-sampling, while PVT is developed to reduce its memory and computation costs.
  • Deep Transformers face attention collapse and increasingly indistinguishable patch representations as depth increases.
  • CaiT separates patch self-attention from later class-attention stages, while Patch Diversity losses regularize deep Transformers against patch over-smoothing.
  • Self-supervised visual Transformer methods include autoregressive pixel prediction, masked latent-space reconstruction, and discriminative teacher-student training.
  • Basic training strategies such as DeiT and LV-ViT are more universal across models, tasks, and input settings than many structure-specific improvements.
  • The survey identifies locality as indispensable, with VOLO dominant for classification and Swin dominant for dense prediction.
  • Convolutional patchify stems and early convolutional stages significantly boost Transformer accuracy, especially for large models.
  • The survey concludes that hybrid convolution-Transformer designs remain promising because shallow convolutions process locality and semantic features effectively.

2) Brief Discussion on Alternatives:

Transformer-based object detectors introduce alternative formulations of detection, including DETR’s end-to-end set prediction and Pix2seq’s autoregressive language-modeling view. DETR removes hand-designed representations and NMS but retains convergence and small-object challenges.

  • Subsequent DETR variants target accuracy and convergence through sparse attention, spatial priors, structural redesign, assignment optimization, and pretraining.
  • DETR reformulates object detection as end-to-end set prediction using object queries and one-to-one bipartite matching.
  • DETR’s matching-based training removes redundant predictions and eliminates non-maximum suppression from inference.
  • DETR achieves competitive COCO performance but suffers from slow convergence and poor performance on small objects.
  • Pix2seq treats generic detection as autoregressive language modeling that generates object classes and bounding boxes sequentially.

2) Transformer with Sparse Attention:

Sparse-attention and spatial-prior methods modify DETR to reduce dense interaction costs and improve convergence or detection quality. Other variants initialize queries from proposals, remove the decoder, or use pretraining and one-to-one assignment analysis.

  • Deformable DETR samples a small set of reference points through multiscale deformable attention to improve convergence and detection performance.
  • ACT dynamically clusters queries into prototypes, while PnP and Sparse DETR selectively condense or update tokens using attention-derived signals.
  • DETR object queries weakly encode spatial information because they are randomly initialized and only implicitly related to bounding boxes.
  • SMCA explicitly supplies spatial priors by multiplying Gaussian-like query weight maps with cross-attention maps.
  • Two-stage methods initialize decoder embeddings from top-k encoder proposals or dense proposals before refining sparse predictions.
  • TSP removes the decoder and object queries, using an encoder-only design with fixed-size features of interest or proposals.
  • UP-DETR pretrains detectors to localize randomly cropped image patches using query assignment and auxiliary reconstruction loss.

6) Transformer Detector with Pre-Training:

Transformer backbones and necks extend visual modeling to dense prediction, with designs targeting multi-scale features, local-to-global dependencies, spatial priors, and efficient attention. The survey notes that dense prediction remains less developed than classification and highlights complementary architectural trade-offs.

  • Backbone designs: Hierarchical backbones such as PVT construct high-to-low resolution processes to learn multi-scale features for dense prediction.
  • Backbone designs: Locally enhanced backbones combine local and global modeling to capture short- and long-range dependencies while avoiding quadratic computational overhead.
  • Neck detectors: Sparse attention and scoring networks reduce redundant feature interactions, lowering computational costs and accelerating convergence.
  • Neck detectors: Explicit spatial priors, multi-scale decoder updates, and improved bipartite matching support precise, refined, end-to-end detection.
  • Discussion: Dense prediction has fewer Transformer backbones than classification, motivating cooperation with deep high-resolution networks.
  • Segmentation frameworks: Segmentation frameworks include transfer learning, multi-task learning, cascade learning, independently supervised queries, and box-free mask prediction.

1) Object Queries:

Object-query Transformers extend detection-style queries to segmentation and related dense prediction tasks. Their variants use transfer, cascaded, auxiliary, and mask-prediction designs, with reported gains across semantic, instance, and panoptic segmentation.

  • Object Queries: QueryInst cascades box outputs and shared queries into mask prediction, preserving instance correspondences across stages and obtaining state-of-the-art COCO results.
  • Mask Embeddings: Mask embeddings are supervised only by segmentation, while parallel query sets support separate tasks and box learning can provide auxiliary enhancement.
  • Mask Embeddings: Max-DeepLab directly predicts masks without a box branch and achieves 51.3% PQ on COCO test-dev, but incurs heavy computational cost.
  • Mask Embeddings: Maskformer reformulates semantic segmentation as mask prediction and reports superiority over per-pixel classification while unifying semantic and instance segmentation.
  • Discussion: Visual Transformers with mask embeddings surpass most prevailing models on segmentation and detection, while a large gap remains between box AP and segmentation AP.
  • Discussion: Mask prediction enables a box-free framework that can unify multiple segmentation tasks.

B. Cognition Mapping

The survey maps Transformer methods from point-cloud representation learning to 3D cognition and processing, while also examining multi-sensory fusion. These methods exploit point-cloud geometry and Transformer attention for semantic representation, detection, completion, and cross-modal interaction.

  • Cognition Mapping: 3DETR extends DETR for 3D detection, while Group-Free 3D DETR selects candidate point samples as object queries and iteratively updates them.
  • Cognition Mapping: MonoDTR and MonoDETR use auxiliary depth supervision to estimate depth positional encodings that guide Transformer encoding and query initialization.
  • Cognition Mapping: PoinTr uses local point proxies and a geometry-aware encoder-decoder to move proxies toward incomplete regions for point-cloud completion.
  • Multi-sensory Fusion: Transformers suit multi-sensory fusion through nonspecific embeddings and dynamically interactive attention across complementary sensor streams.
  • Multi-sensory Fusion: Homologous streams are fused through interactive or transfer mechanisms, including spatial concatenation and cross-attention across views or modalities.

2) Transfer Fusion:

Transfer fusion treats cross-modal integration as transferring information from source data to target data through cross-attention. The survey covers visual-linguistic pre-training, contrastive image-text learning, grounding, and 3D language reasoning.

  • Visual-Linguistic Pre-Training: Visual-linguistic pre-training methods combine visual and textual representations under supervised or self-supervised training schemes.
  • Visual-Linguistic Pre-Training: Early VLP methods use object detectors and text encoders to extract object-centric visual tokens for joint distribution learning.
  • Visual-Linguistic Pre-Training: ViLBERT applies vision-language cross-attention and pre-trains with masked language modeling, masked region classification, and image-text alignment on 3M image-text pairs.
  • Visual-Linguistic Pre-Training: Single-stream models feed visual tokens and text embeddings into cross-modality self-attention for joint representation learning.
  • Contrastive Learning: CLIP jointly trains image and text encoders with contrastive loss on 400M Internet image-text pairs and supports zero-shot transfer with a linear classifier.
  • Visual Grounding: Visual grounding methods learn concrete target supervision by locating objects from textual descriptions, including MDETR for phrase grounding.
  • Visual Grounding: 3D grounding methods use language modeling or cross-attention to connect object features, point-cloud representations, and linguistic descriptions.

VIII. DISCUSSION AND CONCLUSION

The survey identifies task-specific performance improvements and discusses fundamental properties, unresolved issues, and future directions for visual Transformers across vision tasks and data streams.

  • Classification: Hierarchical backbones reduce computational complexity and help avoid feature over-smoothing in deep classification layers.Early convolution captures low-level features, while convolutional projections and local attention improve locality.
  • Detection: Transformer necks reduce computation in detection, but decoder convergence remains slow and requires stronger spatial priors.Sparse attention and foreground-scoring networks reduce computational costs and accelerate convergence.
  • Segmentation: Encoder-decoder models unify segmentation subtasks through learnable mask embeddings and achieve state-of-the-art performance on multiple benchmarks.Cascading a hybrid task with box-based Transformers improves instance segmentation performance.
  • 3D visual recognition: Visual Transformers efficiently model point-cloud features, aggregate surface points globally, and process multi-sensory 3D data.Local hierarchical designs with scoring networks support efficient feature extraction, while global modeling aggregates surface points.
  • Visual-linguistic pretraining: Visual-linguistic pretraining increasingly aligns data streams in latent space using large-scale noisy datasets and supports zero-shot downstream transfer.Mainstream approaches have moved away from pretrained detectors toward alignment or similarity objectives.
  • Multi-sensory fusion: Single-stream fusion spatially concatenates multiple data streams, while subsequent methods seek latent spaces with semantic consistency across streams.The survey emphasizes that visual Transformers still require deeper understanding despite substantial evolution.

3) Double Edges of Visual Transformers:

Visual Transformers combine global modeling and flexible inductive bias with efficiency, data, and low-level-feature trade-offs. The survey also examines learnable embeddings, future research directions, and the broader evidence base.

  • Learnable embeddings: Learnable class tokens, object queries, and mask embeddings support encoder-only and encoder-decoder Transformer patterns.Learned positional encodings and decoder input embeddings are used in encoder-decoder structures.
  • Learnable embeddings: Later class tokens can reduce FLOPs and slightly improve performance, while similar strategies support segmentation efficiency.CaiT and Segmenter are cited as examples of later-token designs.
  • Future directions: Multiple class tokens converge consistently under shared loss gradients, unlike dense prediction settings with different label assignments and target counts.The survey proposes set prediction as a natural direction for classification with multiple class tokens.
  • Future directions: Self-supervised visual Transformer research seeks to bridge differences between contrastive convolutional approaches and masked auto-encoding paradigms.The passage situates this direction within the broader popularity of self-supervision in computer vision.
  • Conclusion: The survey reviews more than one hundred visual Transformer models across tasks, data streams, taxonomies, benchmarks, essential issues, and future directions.Covered data include images, point clouds, image-text pairs, and other multiple data streams.

APPENDIX A OVERVIEW OF DEVELOPMENT TREND ON VISUAL

Visual Transformer development progressed from vanilla NLP-inspired and attention-based designs toward hierarchical, deep, local, convolutional, supervised, and self-supervised architectures. The appendix also details DETR-family formulas, dense-prediction comparisons, and visual-linguistic pretraining configurations.

  • Development trend: Early visual Transformers adapted either the vanilla NLP Transformer structure or attention-based computer-vision models.ViT and iGPT exemplify the former, while VTs and BoTNet exemplify the latter.
  • Development trend: Later methods transferred hierarchical or deep CNN structures into Transformers using varied downsampling and deep-layer designs.T2T-ViT, PVT, CvT, and PiT differ in downsampling, while CaiT, Diverse Patch, DeepViT, and Refiner target deep Transformers.
  • Development trend: The next development wave introduced locality through local attention or convolution, alongside structural combinations, scaling laws, and self-supervised learning.The passage describes these as major directions in recent supervised and self-supervised visual Transformers.
  • DETR formulas: DETR uses bipartite matching to assign predictions one-to-one with ground-truth objects before applying Hungarian loss.The loss combines negative log-likelihood for label predictions with box loss for matched pairs.
  • Deformable DETR: Deformable DETR samples a fixed number of key points at each feature scale and generates query-dependent attention maps over those samples.Offsets are produced for each query and head, while sampled key features are linearly projected.
  • Comparisons: The appendix compares visual Transformer backbones on COCO dense prediction and organizes visual-linguistic methods by pretraining data, objectives, architecture, and visual-token inputs.Table IV uses RetinaNet and Mask R-CNN with a 3× schedule and multi-scale inputs; Table V distinguishes single- and dual-stream methods and zero-shot transfer.
Loading 2111.06091v4…