PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Visual Transformer

Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, Dacheng Tao

arXiv:2012.12556v6cs.CVcs.AI

TL;DR

Vision transformers have rapidly expanded across computer-vision tasks, making the growing literature difficult to track. This survey categorizes recent models, analyzes their strengths and weaknesses, and concludes that they achieve strong performance across visual tasks while remaining incompletely explored and challenging to deploy efficiently.

  • Problem

    The rapid growth of vision-transformer research makes existing advances difficult to track, motivating a comprehensive survey for the computer-vision community.

  • Method

    The paper reviews vision transformers by task category, analyzes their advantages and disadvantages, and discusses self-attention, efficiency methods, challenges, and future directions.

  • Results

    The survey reports strong transformer performance across backbone, high/mid-level, low-level, and video-vision tasks, including state-of-the-art results on multiple image-recognition benchmarks.

  • Takeaways & Limitations

    Vision transformers show broad potential in computer vision, but their capabilities and applications remain not fully explored and require further research.

  • Takeaways & Limitations

    Efficient deployment remains difficult because transformer models are computationally expensive for resource-limited devices, with base ViT requiring 18 billion FLOPs per image.

Abstract

from arXiv · show

Transformer, first applied to the field of natural language processing, is a type of deep neural network mainly based on the self-attention mechanism. Thanks to its strong representation capabilities, researchers are looking at ways to apply transformer to computer vision tasks. In a variety of visual benchmarks, transformer-based models perform similar to or better than other types of networks such as convolutional and recurrent neural networks. Given its high performance and less need for vision-specific inductive bias, transformer is receiving more and more attention from the computer vision community. In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages. The main categories we explore include the backbone network, high/mid-level vision, low-level vision, and video processing. We also include efficient transformer methods for pushing transformer into real device-based applications. Furthermore, we also take a brief look at the self-attention mechanism in computer vision, as it is the base component in transformer. Toward the end of this paper, we discuss the challenges and provide several further research directions for vision transformers.

1 INTRODUCTION

Transformers, initially successful in NLP through their strong representation capacity, are increasingly being applied to computer vision as potential alternatives to CNNs. This survey organizes recent vision-transformer research across major vision tasks, efficient methods, self-attention, and future directions.

  • Background: Transformer models achieved significant improvements in NLP and strong performance across downstream language tasks through their representation capacity.The cited examples include the original attention-based transformer for machine translation and parsing and BERT for language representation.
  • Background: Inspired by NLP success, researchers applied transformers to computer vision, where sequence transformers achieved image-classification results comparable to CNNs.The passage presents transformers as a potential alternative to CNNs in vision applications.
  • Motivation: The survey responds to rapidly increasing transformer-based vision models by providing a comprehensive overview of recent advances and potential directions for improvement.The authors motivate the survey by noting that keeping pace with new progress is increasingly difficult.
  • Survey scope: The paper categorizes vision transformers into backbone, high/mid-level vision, low-level vision, video, and efficient-transformer methods.High- and mid-level vision are treated as one category because their distinction is becoming more obscure in DNN-based vision systems.
  • Survey scope: Representative high/mid-level applications include DETR and deformable DETR for object detection and Max-DeepLab for segmentation.The survey also discusses the standard transformer formulation and self-attention before reviewing these application areas.

2 FORMULATION OF TRANSFORMER

The original transformer uses encoder-decoder stacks built from attention and feed-forward sublayers, while self-attention computes input relationships through scaled score normalization and value weighting. Vision transformers commonly use the encoder as a feature extractor, providing global modeling alongside residual, normalization, positional, and multi-head mechanisms.

  • 2 FORMULATION OF TRANSFORMER: The original transformer comprises encoder and decoder modules made of repeated transformer blocks, with the encoder producing encodings and the decoder generating output sequences from contextual information.Each block includes a multi-head attention layer and a feed-forward neural network.
  • 2.1 Self-Attention: Self-attention transforms each input into query, key, and value vectors, packs them into Q, K, and V, and computes relationships among inputs.The formulation uses dq = dk = dv = dmodel = 512.
  • 2.1 Self-Attention: Self-attention computes scores as S = Q · K⊤, scales them as Sn = S/√dk, converts them with P = softmax(Sn), and obtains weighted values as Z = V·P.These steps respectively calculate pairwise scores, stabilize gradients, produce probabilities, and weight value vectors.
  • 2.1 Self-Attention: Attention scores determine how strongly each vector attends to other positions, while normalization improves gradient stability and larger probabilities increase focus on corresponding value vectors.The mechanism assigns attention according to pairwise scores and probability weights.
  • 2.1 Self-Attention: Decoder encoder-decoder attention derives K and V from the encoder and Q from the previous decoder layer, unlike encoder self-attention, which derives all three from the same input.Position information is added because self-attention is invariant to input order; positional encodings can be fixed, learned, or relative.
  • 2.1 Self-Attention: Multi-head attention uses distinct query, key, and value projections across representation subspaces so the model can attend to multiple important positions simultaneously.With h heads, each head uses dq′ = dk′ = dv′ = dmodel/h = 64.
  • 2.2 Other Key Concepts in Transformer: Each encoder and decoder sublayer applies a two-layer feed-forward network with a nonlinear activation, residual connections, and subsequent layer normalization.The FFN hidden dimensionality is dh = 2048, and GELU is an example activation.
  • 2.2 Other Key Concepts in Transformer: For computer vision, transformers typically use the encoder as a feature extractor that captures long-distance and global characteristics more directly than locally focused CNNs.The passage also contrasts transformer parallelism with the sequential hidden-state computation of RNNs.

3 VISION TRANSFORMER

This section reviews transformer-based models across major computer vision tasks and summarizes self-attention applications and model compression for efficient transformers.

  • 3 VISION TRANSFORMER: The review covers image classification, high/mid-level vision, low-level vision, and video processing.It organizes applications of transformer-based models across these computer vision areas.
  • 3 VISION TRANSFORMER: It also summarizes self-attention applications and model compression methods for efficient transformer deployment.Self-attention is reviewed as a mechanism, while compression methods address transformer efficiency.

3.1 Backbone for Representation Learning

Vision transformers provide a transformer-based backbone for image representation learning, with ViT tokenizing image patches and using standard transformer encoding. Their performance depends strongly on data scale and architectural improvements that restore locality and efficiency.

  • ViT: ViT applies a pure transformer directly to sequences of image patches for image classification.It follows the original transformer design as closely as possible.
  • ViT: Images are reshaped into flattened patches, projected into dimension d, and processed as a sequence of length n = hw/p^2.Here h and w denote image resolution, p patch resolution, and c the number of channels.
  • ViT: ViT adds a learnable class embedding and 1D positional embeddings, using the class embedding state as the image representation.The model uses the standard transformer encoder, with classification heads attached during pretraining and fine-tuning.
  • ViT: 88.36% ImageNet accuracy and 77.16% VTAB accuracy were achieved by ViT after JFT-300M pretraining, whereas mid-sized-data training produced results below comparable ResNets.Large-scale pretraining enabled strong performance on recognition benchmarks despite weaker inductive biases than CNNs.
  • Variants of ViT: ViT variants improve locality, self-attention, and architecture, including sub-patch modeling, modified attention, and pyramid-like network designs.TNT uses an inner transformer for sub-patches; other methods address attention diversity, locality, noisy tokens, and local patterns.
  • Hybrid Backbones: Combining CNNs and transformers achieves better performance, indicating complementary strengths between convolution and attention in visual recognition.Convolution supplies local information that vision transformers can lack, while attention captures long-range dependencies.

3.2 High/Mid-level Vision

Vision transformers are applied across high- and mid-level tasks including detection, segmentation, pose estimation, pedestrian detection, and tracking. The section emphasizes transformer-based detection designs, their benefits and limitations, and broader challenges involving representation, efficiency, and task adaptation.

  • Scope: High/mid-level vision applications include object, lane, pedestrian, and video tracking detection, segmentation, and pose estimation.The section reviews transformer methods across these task categories.
  • Object Detection: Transformer-based object detectors are broadly divided into set prediction methods and transformer-based backbone methods.Other approaches enhance detector components such as feature fusion modules and prediction heads.
  • Object Detection: DETR reformulates detection as fully end-to-end set prediction, removing hand-crafted components such as anchor generation and NMS post-processing.It uses CNN features, positional encodings, an encoder-decoder transformer, and learned object queries.
  • Object Detection: DETR achieves comparable accuracy and speed with Faster R-CNN on COCO, but vanilla DETR requires longer training and performs poorly on small objects.Deformable DETR addresses these challenges with deformable attention and significantly improves detection performance.
  • Object Detection: Adaptive Clustering Transformer reduces pre-trained DETR computation cost by clustering query features with LSH and broadcasting attention outputs from selected prototypes.ACT replaces DETR self-attention without requiring re-training.
  • Other High/Mid-level Tasks: Vision transformers show potential for segmentation and use encoder-decoder architectures in tracking to replace explicit cross-correlation with global contextual dependencies.The section also identifies input embedding, position encoding, and prediction loss as key issues for high-level adoption.

3.3 Low-level Vision

Low-level vision applies transformers to image-producing tasks such as generation, super-resolution, and denoising, which are more challenging than label- or box-producing vision tasks. Approaches include autoregressive and GAN-based generation, patch-based texture transfer, and transformer frameworks for multiple image-processing tasks.

  • Low-level Vision: Low-level vision is challenging because its outputs are images, unlike the labels or boxes produced by classification, segmentation, and detection.
  • Image Generation: TransGAN replaces CNN architectures with transformers for image generation and gradually increases feature-map resolution to avoid the difficulty of pixel-wise high-resolution generation.
  • Image Generation: Image Transformer performs autoregressive image translation and generation with an encoder-decoder that embeds 0–255 pixel values into 256 × d-dimensional vectors.
  • Image Super-Resolution: TTSR uses transformer-based patch matching, hard attention, and soft attention to transfer relevant high-resolution textures from reference images to low-resolution images.
  • Image Processing: IPT uses large pre-training datasets and achieves state-of-the-art performance across super-resolution, denoising, and deraining.
  • Low-level Vision: Transformer image-processing systems encode images into token or patch sequences for the encoder, while generation systems use GAN-based models to learn decoders that generate patches.

3.4 Video Processing

Transformers are applied to video processing because they can model spatial and temporal information simultaneously across frame synthesis, action recognition, and video retrieval. However, efficiently and accurately modeling both relationships remains an open problem because video data are complex and redundant.

  • Video Processing: Transformers have been applied to video tasks including frame synthesis, action recognition, and video retrieval by exploiting spatial and temporal information.Video processing benefits from modeling both dimensions of video data.
  • Video Action Recognition: Action transformers model relationships between a human of interest and surrounding context for recognizing and localizing human actions in videos.Context includes other people and objects, which plays a critical role in action recognition.
  • Video Action Recognition: Actor-transformer architectures support group activity recognition by using static and dynamic representations generated by 2D and 3D networks.The transformer output is the predicted activity.
  • Frame/Video Synthesis: ConvTransformer performs frame and video synthesis with feature embedding, position encoding, an encoder, a query decoder, and a synthesis feed-forward network.Compared with LSTM-based works, its architecture achieves superior results while being more parallelizable.
  • Challenges: Despite improvements from simultaneous spatial-temporal modeling, efficiently and accurately modeling both relationships remains an open problem because video data have high complexity and redundancy.Transformers improve video processing by mining spatial and temporal information simultaneously.

3.5 Multi-Modal Tasks

Transformer-based models are being adapted to multi-modal tasks by jointly processing or generating text, images, video, and audio. Existing approaches demonstrate unified architectures across modalities and tasks, while motivating research on training effectiveness and extendability.

  • VideoBERT preprocesses videos with a CNN module, then trains a transformer encoder on representation tokens for video-text tasks such as video captioning.
  • CLIP jointly trains text and image encoders to predict corresponding text-image pairs, using natural language supervision to learn efficient image representations.Its text encoder is a standard transformer with masked self-attention.
  • DALL-E is a multimodal transformer with 12 billion model parameters autoregressively trained on 3.3 million text-image pairs to synthesize images from text.Training first compresses 256×256 RGB images into 32×32 image tokens with a discrete variational autoencoder, then models joint image-text token distributions.
  • UniT uses separate image and text transformer encoders and a decoder that combines their outputs to support multi-modal multi-task learning across several domains.Its tasks include object detection, natural language understanding, and vision-language reasoning.
  • The surveyed models illustrate transformer architectures’ potential to unify data and tasks across modalities for general-purpose intelligence, while highlighting training and extendability as future directions.

3.6 Efficient Transformer

Efficient-transformer research addresses the memory and compute demands that hinder deployment on resource-limited devices such as mobile phones. Reviewed strategies include pruning, low-rank decomposition, knowledge distillation, quantization, compact architecture design, and reducing self-attention complexity.

  • Motivation: Efficient-transformer methods compress or accelerate models to enable implementation on resource-limited devices.The review covers network pruning, low-rank decomposition, knowledge distillation, network quantization, and compact architecture design.
  • Pruning: Pruning removes redundant attention heads, feed-forward layers, or transformer layers to reduce model width or depth and accelerate inference.Attention heads can be removed at test time without impact in some settings, while layer-wise dropping targets sequential depth reduction.
  • Decomposition: Low-rank decomposition approximates large transformer matrices with multiple smaller matrices to improve inference efficiency.This approach replaces standard matrix multiplication with a decomposition based on the low-rank assumption.
  • Knowledge distillation: Knowledge distillation transfers information from large teacher networks to thinner and shallower student networks for easier deployment.Transfer can use outputs or intermediate features, with objectives tailored to transformer layers such as embeddings and self-attention.
  • Quantization and compact computation: Quantization reduces the bits used to represent weights or intermediate features, while compact designs and sparse approximations target cheaper transformer computation.Compact architectures modify self-attention or substitute its layers, and sparse graph methods can achieve O(N) complexity for long sequences.

4 CONCLUSIONS AND DISCUSSIONS

Vision transformers show competitive performance across diverse computer-vision tasks, but their architecture, generalization, interpretability, robustness, and efficiency remain open challenges. Future work should pursue more effective and efficient models, broader multitask capabilities, and clearer understanding of when transformers outperform CNNs.

  • Conclusions: Transformer-based methods achieve excellent performance across backbone, high/mid-level vision, low-level vision, and video-processing tasks.
  • Open challenges: Vision transformers still require improved computer-vision-specific architectures and application to tasks beyond those covered by existing work.
  • Open challenges: Large datasets and data quality strongly influence transformer generalization and robustness, while vulnerability remains an open problem.
  • Open challenges: The reasons transformers work well for visual tasks remain difficult to explain because transformers lack CNN inductive biases such as translation equivariance and locality.
  • Open challenges: 18 billion FLOPs are required by base ViT per image, versus about 600 million FLOPs for lightweight GhostNet at similar performance.
  • Future directions: Future research should develop high-performance, low-resource-cost transformers and models that handle multiple tasks or domains in one system.IPT processes super-resolution, image denoising, and deraining, while Perceiver and Perceiver IO span images, audio, multimodal inputs, and point clouds.

APPENDIX · A1. General Formulation of Self-attention

Self-attention computes each output position by relating it to all input positions, aggregating position-specific representations, and normalizing the resulting responses. In computer vision, this formulation corresponds closely to non-local filtering and can be expressed through embedding-based pairwise functions or query-key-value representations.

  • A1. General Formulation of Self-attention: Self-attention estimates relationships between each position and all positions, then gathers the corresponding input embeddings to produce responses.The input signal may be an image, sequence, video, or feature, with X ∈ R^n×d and n = h × w for images.
  • A1. General Formulation of Self-attention: The computer-vision self-attention module is, to some extent, the same as preceding non-local filtering operations.This connects the machine-translation self-attention formulation to computer-vision formulations.
  • A1. General Formulation of Self-attention: For position i, the pairwise function f(·) measures relationships with positions j, while g(·) represents the input at position j and C(x_i) normalizes the response.The input and output positions are represented by x_i ∈ R^1×d and y_i ∈ R^1×d, respectively.
  • A1. General Formulation of Self-attention: The pairwise function can use different choices, including a Gaussian-function extension that computes similarity in an embedding space.The formulation therefore separates relationship estimation from representation aggregation.
  • A1. General Formulation of Self-attention: With linear embeddings, θ(X), ϕ(X), and g(X) are parameterized by W_θ, W_ϕ, and W_g, with output dimensions d_k, d_k, and d_v.The supplied formulation defines θ(X) = XW_θ, ϕ(X) = XW_ϕ, and g(X) = XW_g.
  • A1. General Formulation of Self-attention: Under the normalization convention, 1/C(x_i) f(x_i, x_j) becomes the softmax output along dimension j.The resulting formulation uses the row w_θ,i of W_θ for a given index i.
  • A1. General Formulation of Self-attention: The formulation can be expressed with query, key, and value representations Q = XW_q, K = XW_k, and V = XW_v by identifying W_q = W_θ, W_k = W_ϕ, and W_v = W_g.The supplied passage states that this substitution reformulates the preceding equation in query-key-value form.
  • A1. General Formulation of Self-attention: Initializing W_o as zero allows the self-attention module to be inserted into an existing model without breaking its initial behavior.The final computer-vision output signal wraps the self-attention response generated through the preceding formulation.

A2. Revisiting Transformers for NLP

Transformer-based models address limitations of sequential RNN language models and have become versatile across pretraining, generation, domain-specific NLP, and multimodal applications. Their broad success motivates extending transformers beyond NLP, including to computer vision.

  • A2. Revisiting Transformers for NLP: RNN language models process information sequentially, limiting training acceleration and parallelization and hindering longer-sequence processing or larger models.The passage contrasts these constraints with transformer development in 2017.
  • A2. Revisiting Transformers for NLP: BERT and variants such as SpanBERT and RoBERTa use multi-layer transformer encoders and pretrain with masked language modeling on BookCorpus and English Wikipedia.BERT also learns [SEP] and [CLS] embeddings during pre-training.
  • A2. Revisiting Transformers for NLP: GPT’s unidirectional pre-training supports strong natural-language-generation performance, while GPT-3 contains 175 billion parameters and processes downstream NLP tasks without fine-tuning.GPT-3 was pre-trained on 45 TB of compressed plaintext data.
  • A2. Revisiting Transformers for NLP: Transformer-based models extend beyond general NLP to biomedical, scientific, clinical, other NLP-related, and multimodal tasks.Examples include BioBERT, SciBERT trained on 114M scientific articles, and ClinicalBERT.
  • A2. Revisiting Transformers for NLP: Their performance across NLP tasks demonstrates structural superiority and versatility, supporting transformer applications in AI fields beyond NLP and motivating computer-vision research.The survey next examines transformer applications across computer-vision tasks that emerged over the past two years.

A3. Self-attention for Computer Vision

Self-attention is a pivotal transformer component and a building block used in CNNs to capture long-range interactions and enhance high-level semantic features. Vision applications span classification, segmentation, detection, pose estimation, and depth estimation through diverse attention mechanisms.

  • Overview: Self-attention captures long-range interactions and enhances high-level semantic features, complementing CNN architectures with limited large-receptive-field scaling.The section identifies self-attention as both central to transformers and a reusable CNN building block.
  • Image Classification: Image classification uses hard attention to select relevant regions and soft attention to generate non-rigid feature maps.These two streams apply attention to image regions or locations and can reduce computational complexity.
  • Semantic Segmentation: Self-attention in semantic segmentation models relationships between contextual pixels, with DANet attending across spatial and channel dimensions and A2Net grouping pixels into regions.PSANet, OCNet, DANet, and CFNet pioneered self-attention for semantic segmentation.
  • Object Detection: Attention-based detection replaces or augments convolution, models global context, exchanges information across positions, and exploits cross-scale feature interactions.Feature Pyramid Transformer addresses cross-scale interactions, while other methods process object representations through self-attention.
  • Other Vision Tasks: Attention mechanisms also enhance multi-resolution pose estimation, self-supervised monocular depth estimation, and scene-dependent depth prediction.Proposed modules include resolution-wise attention, attention-based feature fusion and aggregation, and bidirectional attention.
Loading 2012.12556v6…