PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformers in Vision: A Survey

Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, Mubarak Shah

arXiv:2101.01169v5cs.CVcs.AIcs.LG

TL;DR

Transformer models have spurred broad interest in computer vision, but visual data requires specialized designs and training schemes. This survey organizes Transformer approaches across vision tasks, compares their strengths and shortcomings, and analyzes open research directions.

  • Problem

    Computer vision lacks a comprehensive organization of emerging Transformer approaches despite growing interest in adapting them to structured visual data.

  • Method

    The survey develops a taxonomy of Transformer network designs and reviews their concepts, applications, strengths, shortcomings, and experimental value across computer vision.

  • Results

    The survey provides an in-depth analysis of Transformer and self-attention models spanning recognition, detection, segmentation, video, multimodal, generative, low-level, and 3D vision tasks.

  • Takeaways & Limitations

    The survey serves as a comprehensive reference for visual Transformers and identifies open research directions and possible future work.

  • Takeaways & Limitations

    Transformer vision models face high computational costs for high-resolution images, with efficient alternatives often trading off complexity and accuracy or requiring specialized hardware.

Abstract

from arXiv · show

Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g., Long short-term memory (LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities (e.g., images, videos, text and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and huge datasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction to fundamental concepts behind the success of Transformers i.e., self-attention, large-scale pre-training, and bidirectional encoding. We then cover extensive applications of transformers in vision including popular recognition tasks (e.g., image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks (e.g., visual-question answering, visual reasoning, and visual grounding), video processing (e.g., activity recognition, video forecasting), low-level vision (e.g., image super-resolution, image enhancement, and colorization) and 3D analysis (e.g., point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works.

1 INTRODUCTION

Breakthroughs in Transformer-based NLP have prompted computer vision researchers to adapt these models to visual and multimodal tasks. This survey organizes visual Transformer methods, emphasizing their network-design space and the strengths and shortcomings of existing approaches.

  • Motivation: NLP breakthroughs from Transformer networks sparked interest in adapting them for computer vision and multimodal learning tasks.
  • Motivation: Visual data’s spatial and temporal coherence requires novel Transformer network designs and training schemes.
  • Applications: Transformer variants have been applied to recognition, detection, segmentation, super-resolution, video understanding, generation, text-image synthesis, and visual question answering.
  • Foundations: Transformers use self-attention to model relationships across complete sequences and learn long-range dependencies.
  • Survey scope: The survey provides a holistic overview, develops a taxonomy of visual Transformer network designs, and highlights existing methods’ major strengths and shortcomings.

2 FOUNDATIONS

Transformer foundations center on self-attention, which models global dependencies through query-key attention and weighted aggregation, and large-scale pre-training followed by downstream fine-tuning. Their flexibility arises from permutation invariance and support for irregular inputs, while self-supervision enables learning from vast unannotated data and bidirectional encoding incorporates both left and right context.

  • Self-Attention: Self-attention updates each sequence element by aggregating global information, using softmax-normalized query-key dot products as weights over values.This explicitly models interactions among all sequence entities rather than only local relationships.
  • Self-Attention: Unlike convolution, self-attention dynamically calculates filters, is permutation-invariant, and can process inputs with changing cardinality or irregular structure.With positional encodings, it is theoretically more flexible and can model convolutional behavior.
  • Pre-training: Transformer training generally pre-trains on large supervised or self-supervised datasets before adapting the learned weights to downstream tasks with smaller datasets.Vision Transformer ViT-L shows a 13% ImageNet test accuracy drop when trained only on ImageNet compared with pre-training on JFT’s 300 million images.
  • Pre-training: Self-supervised pre-training learns representations from non-annotated data by solving automatically generated pretext tasks without expensive manual annotations.This paradigm supports scalability and generalization, including training networks above a trillion parameters.
  • Bidirectional Encoding: Bidirectional encoding improves textual representations by jointly incorporating the context to the right and left of each word, unlike the original Transformer’s left-only context.BERT introduced this self-supervised bidirectional encoding approach.

3 SELF-ATTENTION & TRANSFORMERS IN VISION · 3.1 Single-head Self-Attention

Single-head self-attention frameworks in vision apply global or local attention within CNNs, or use matrix factorization for efficiency. They capture long-range or adaptive feature interactions, but global attention can be computationally intensive, motivating restricted or decomposed designs and hybrid convolutional approaches.

  • 3 SELF-ATTENTION & TRANSFORMERS IN VISION: Single-head self-attention models generally apply global or local attention within CNN architectures or use matrix factorization to improve efficiency.
  • 3.1.1 Self-Attention in CNNs: Non-local operations compute each position as a weighted sum of features at all positions, capturing long-range dependencies across space and time.
  • 3.1.1 Self-Attention in CNNs: Global self-attention models full-image context but non-local blocks require O(N^2) complexity, making them memory- and compute-intensive.
  • 3.1.1 Self-Attention in CNNs: Criss-cross attention reduces the computational burden of dense global attention, while reporting state-of-the-art results on semantic and instance segmentation benchmarks.
  • 3.1.1 Self-Attention in CNNs: Local relation networks adapt weight aggregation to pixel or feature similarities within a fixed local window, such as a 7x7 neighborhood.
  • 3.1.1 Self-Attention in CNNs: Two-dimensional relative position encoding lets self-attention maintain translation equivariance, although its best performance is achieved when combined with convolutional operations.
  • 3.1.2 Self-Attention as Stand-alone Primitive: Stand-alone self-attention addresses convolution’s limited receptive field, while global attention remains computationally intensive and is preferred for down-sampled images, patches, or augmented convolutional features.
  • 3.1.2 Self-Attention as Stand-alone Primitive: Self-attention networks separate feature aggregation from transformation, beat ResNet baselines on ImageNet with fewer parameters, and show robustness to adversarial perturbations and unseen transformations.

3.2 Multi-head Self-Attention (Transformers)

Multi-head self-attention vision models began with ViT, which applies minimally modified Transformers to flattened image patches, and expanded through distillation, token refinement, efficient attention, and hierarchical designs. These developments address dataset requirements, computational complexity, multiscale representation, and the integration of convolutional inductive biases.

  • Uniform-scale ViTs: ViT replaces standard convolutions with a Transformer over flattened image patches, pre-trained on JFT’s 300 million images before fine-tuning for recognition.The first token’s learned feature is used for classification.
  • Uniform-scale ViTs: DeiT enables Transformer training on ImageNet’s 1.2 million examples using augmentation, regularization, and CNN-teacher native distillation.The teacher model is RegNetY-16GF, and distillation uses an appended distillation token with a dedicated distillation loss.
  • Uniform-scale ViTs: Token-efficient and deeper ViT variants refine spatial aggregation, local attention, token supervision, and attention re-attendance to improve representation or scaling.T2T ViT recursively merges neighboring tokens, Transformer in Transformer uses patch- and sub-patch-level attention, and DeepViT re-attends attention maps.
  • Efficient attention: XCiT replaces token attention with feature-channel cross-covariance attention, yielding linear complexity and enabling large-resolution image processing across classification and dense prediction.Its complexity depends on feature dimension rather than token count.
  • Hierarchical and hybrid ViTs: Hierarchical and hybrid ViTs progressively reduce tokens while increasing feature dimensions, use local-window or efficient attention, and combine convolution with self-attention for multiscale and low-level structure.Examples include PVT, PVTv2, SegFormer, Swin Transformer, CvT, LocalViT, LeViT, and CoAtNet; CoAtNet reaches 86% ImageNet top-1 accuracy without extra data.

3.3 Transformers for Object Detection

Transformer-based object detection spans hybrid designs with Transformer backbones or decoders and fully Transformer-based pipelines. DETR formulates detection as set prediction while reducing hand-crafted components, but faces small-object, convergence, and computational challenges that deformable attention addresses.

  • 3.3 Transformers for Object Detection: Object detection uses Transformer backbones with R-CNN heads, CNN backbones with Transformer decoders, or purely Transformer-based end-to-end designs.These three configurations organize the main applications of Transformer modules to detection.
  • DETR: DETR treats detection as set prediction, using bipartite matching to associate predicted and ground-truth boxes and modeling relationships among objects.Its encoder flattens CNN spatial features into a sequence, while the decoder predicts all boxes in parallel.
  • DETR: DETR removes reliance on hand-crafted components such as region proposal networks and nonmaximum suppression, enabling an end-to-end trainable pipeline.This relaxes dependence on prior knowledge and careful engineering in structured detection tasks.
  • DETR: DETR struggles with small objects and suffers from slow convergence and high computational cost because self-attention scales quadratically with spatial feature size.Attention requires many training epochs to learn meaningfully sparse locations, contributing to slow convergence.
  • Deformable Attention: Deformable attention mitigates DETR’s limitations by attending only to a sparse set of feature-map elements regardless of spatial size and supporting cross-scale aggregation.The module is inspired by deformable convolutions and processes feature maps more selectively.
  • YOLOS: YOLOS adapts ViT for detection by replacing the class token with learnable object-query tokens and applying bipartite matching in a minimally biased sequence-to-sequence architecture.This demonstrates that ViTs can support object detection with minimal image-related 2D inductive biases.

3.4 Transformers for Segmentation

Transformers support segmentation by modeling rich pixel interactions and long-range context through self-attention, while cross-modal variants align visual features with language for referring image segmentation. Recent architectures also use ViT-based encoder-decoder designs for dense prediction.

  • Image segmentation: Self-attention models rich interactions between pixels, making it suitable for dense prediction tasks such as image segmentation.The section discusses axial self-attention as one approach for segmentation.
  • Related approaches: The surveyed segmentation methods include cross-modal approaches for language-guided regions and ViT-based architectures.These approaches complement self-attention mechanisms discussed for dense prediction.
  • Panoptic segmentation: Panoptic segmentation jointly assigns semantic labels and instance identities to pixels, but applying self-attention to its large dense inputs is prohibitively expensive.Global context is useful for this complex visual understanding task, motivating efficient attention strategies.
  • Referring image segmentation: Cross-modal self-attention encodes long-range dependencies between linguistic and visual features to segment image entities referred to by language descriptions.It concatenates image features with each word embedding and spatial coordinate features before generating image attention for each word.
  • Transformer architectures: Recent transformer-based segmentation models replace CNN-centered designs with encoder-decoder architectures, including SETR and SegFormer.SETR uses a ViT encoder with progressive upsampling and multi-level feature aggregation, whereas SegFormer combines a hierarchical pyramid ViT encoder without position encoding with an MLP decoder.

3.5 Transformers for Image and Scene Generation

Transformer-based generative models support autoregressive image synthesis, unsupervised representation learning, efficient visual generation, 3D scene layout generation, and text-to-image synthesis. Across these settings, self-attention models pixel, object, or multimodal relationships while enabling high-quality outputs and downstream utility.

  • Autoregressive image generation: Image Transformer sequentially predicts each pixel from previously generated pixels by factorizing the image distribution into pixel-wise conditional distributions, addressing PixelCNN’s limited receptive field.Its self-attention mechanism models long-term relationships among image pixels.
  • Unsupervised representation learning: iGPT trains GPT-2 on flattened image sequences and generates plausible images without external supervision, while learning strong features for downstream vision tasks.It uses no image-specific design knowledge such as two-dimensional position embeddings.
  • Efficient image generation: Transformers’ high compute cost on high-dimensional sequences can be reduced by combining CNN-inspired local connectivity and spatial invariance with a learned visual-pattern dictionary.Esser et al. use a Generative Adversarial approach to learn the dictionary before applying a Transformer to longer-range relationships.
  • Efficient image generation: TransGAN constructs a convolution-free GAN whose generator and discriminator are both Transformer-based, with the generator organized into multiple stages for memory efficiency.Both networks are based on the encoder of the original Transformer model.
  • 3D scene generation: SceneFormer generates realistic 3D room object configurations from empty room shapes by modeling inter-object relationships with self-attention, without appearance information.It applies a Transformer to object sequences to predict the next suitable object in a scene.
  • Text-to-image generation: DALL·E generates high-fidelity images from text and controls multiple objects, attributes, viewpoints, internal structures, and combinations of unrelated objects.It also performs image-to-image translation.

3.6 Transformers for Low-level Vision

Transformers have been applied to low-level vision tasks including restoration, super-resolution, denoising, deraining, and colorization. These methods address restoration-specific challenges through task-sharing, reference-based texture transfer, efficient attention, and conditional attention mechanisms.

  • Low-level vision applications: Transformer-based methods target image super-resolution, denoising, deraining, and colorization while preserving fine image details such as edges and texture.Image restoration requires pixel-to-pixel correspondence between input and output images.
  • Efficient image restoration: Efficient restoration Transformers reduce attention costs through local windows, spatial reduction, or encoder-decoder designs.These strategies address the infeasibility of single-scale feature processing and the high computational burden of image restoration.
  • Image restoration: Image Processing Transformer (IPT) uses multiheads, multitails, and a shared encoder-decoder Transformer body for multiple restoration tasks.When fine-tuned for a specific low-level vision task, pretrained IPT provides significant gains over state-of-the-art methods.
  • Super-resolution: TTSR searches reference images for relevant regions and transfers rich textures to improve reference-based image super-resolution.It is designed to alleviate the trade-off between perceptual reproduction and accurate reproduction using paired low-resolution, high-resolution, and reference images.
  • Image colorization: Colorization Transformer applies conditional attention, with row-wise and column-wise attention, to generate grayscale image colorizations at 256×256 resolution.Because colorization is a one-to-many task, the model is probabilistic and capable of producing multiple colorized output samples.

3.7 Transformers for Multi-Modal Tasks

Transformers support diverse vision-language tasks through large-scale vision-language pre-training that learns cross-modality relationships. Architectures range from single- and dual-stream models to contrastive image-text encoders, with reported gains in cross-modal and novel-task evaluations.

  • Vision-language applications: Vision-language Transformers address VQA, visual commonsense reasoning, cross-modal retrieval, image captioning, and related tasks through pre-training on large-scale multimodal datasets.These methods aim to learn generic representations that encode cross-modality relationships.
  • Dual-stream architectures: ViLBERT introduced a dual-stream BERT extension, applying self-supervised pre-training on 3.3M weakly captioned images before downstream fine-tuning.Its proxy tasks include image-text relationship prediction and masked image-region and text-semantic prediction.
  • Architectural comparisons: Evaluations found that a two-stream model can outperform a single-stream shared-parameter model, while PEMT identified mid-level fusion as optimal among compared strategies.PEMT compares early, mid-level, and late fusion mechanisms for audio-visual representation learning.
  • Dual-stream architectures: LXMERT models object relationships with a dual-stream architecture, uses five pre-training tasks, and demonstrates impressive improvements on the novel NLVR task.Its cross-attention blocks fuse information across visual and language streams, while the task set includes masked prediction, cross-modality matching, and VQA.
  • Contrastive learning: CLIP jointly trains image and text encoders to maximize valid pair similarity and minimize invalid pair similarity, using 400 million image-text pairs.It considers ResNet-50 and ViT image encoders and learns a multimodal embedding space.

3.8 Video Understanding

Transformer-based video understanding methods address the short-range dependency limits of conventional audio-video and CNN-based approaches by modeling long temporal, multimodal, spatial, and inter-frame relationships. Applications include representation learning, classification, captioning, action localization, and unified video instance segmentation.

  • Long-range dependency modeling: Transformer methods target the short-range dependency limits of approaches trained on videos lasting only a few seconds.Long-range dependency modeling is desirable for unimodal and multimodal tasks such as activity recognition.
  • Multimodal representation learning: VideoBERT predicts masked visual and linguistic tokens to learn multimodal representations that capture high-level semantics and long-range temporal dependencies.Its representations support action classification, zero-shot classification, and video captioning.
  • Unified video understanding: Unified Transformer architectures jointly perform dense video event detection and captioning, while VisTR frames video instance segmentation as sequence-to-sequence prediction integrating detection and tracking.VisTR uses encoder-decoder Transformers and produces instance predictions consistently across frames.
  • Video classification: Video Transformer Network applies a Longformer encoder to frame-wise CNN features, learning temporal relationships with O(n) complexity for arbitrarily long sequences.This design addresses the limited intervals used by traditional 3D CNN video-classification methods.
  • Action classification and localization: A Transformer aggregates person-specific contextual cues for action classification and localization, but global-information integration can be sub-optimal for precise action localization.Adding optical flow or audio would further increase compute complexity, indicating a trade-off between global and local context.
  • Skeleton-based action recognition: A two-stream Transformer models skeleton-based actions through spatial self-attention over body joints and temporal self-attention over long-range inter-frame dependencies.The two dedicated attention modules separately capture within-frame joint relations and across-frame temporal relations.

3.9 Transformers in Low-shot Learning

Transformers support low-shot learning by adapting representations to each task and modeling relationships between query and support examples. Applications include task-specific embeddings, patch-wise spatial matching, and dynamic integration of domain-specific representations.

  • Query-support matching: Patch-wise spatial self-attention between query and support images achieves higher absolute performance than image-level association.This comparison concerns few-shot learning methods that model relationships between query and support examples.
  • Query-support matching: Cross-Transformer relates query and support images by finding spatially similar regions and using their corresponding features for class decisions.Its queries derive from query-image grid features, while support-image grid features provide the comparison representations.
  • Task-specific adaptation: FEAT uses a Transformer-based set-to-set function to adapt few-shot embeddings to target classes during inference, achieving the best performance among evaluated mappings.Compared alternatives include graph convolutional networks, bidirectional LSTMs, and DeepSets.
  • Cross-domain representation: The URT layer dynamically re-weights representations from domain-specific backbones, effectively handling few-shot tasks across varied data distributions.It integrates visual representations learned by models trained on different domains within a meta-dataset.

3.10 Transformers for Clustering

Transformers for clustering address the challenge of discovering structure by grouping similar data points, while preserving inter-point relationships in set-based inputs. Set Transformers operate directly on sets to overcome limitations of individually processed setpoints.

  • Clustering groups similar data points to discover structure for visualization, interpretation, anomaly detection, and open-set categorization.
  • Set-prediction neural networks process setpoints individually, which can lose information about inter-point relationships.
  • Set Transformers operate on set inputs for clustering.

3.11 Transformers for 3D Analysis

Transformers are well suited to 3D analysis because self-attention can model permutation-invariant point sets and non-local relationships in point clouds, meshes, and body joints. Recent methods adapt Transformer attention and positional mechanisms for point-cloud classification, segmentation, and single-image 3D human pose and mesh reconstruction.

  • Point-cloud analysis: Transformers suit point clouds because self-attention is a set-operator invariant to variable point counts and permutations.This motivates Transformer-based 3D representations that encode relationships among irregular point sets.
  • Point-cloud analysis: Point Transformer applies local self-attention with vectorized channel-wise attention and positional encoding to 3D point neighborhoods.Its architecture also includes transition down/up blocks for changing the number of points in an encoding-decoding pipeline.
  • Point-cloud analysis: Point Transformer architectures show promising results on 3D classification and segmentation tasks.These results follow their use of Transformer mechanisms tailored to irregular, permutation-invariant point-cloud data.
  • Point-cloud analysis: Point Cloud Transformer adapts the conventional Transformer with 3D coordinate-based positional encoding, offset attention, and neighbor embeddings for local structure.Unlike Point Transformer, it does not use vector attention.
  • Human pose and mesh reconstruction: METRO jointly models vertex-to-vertex and vertex-to-body-joint relationships to reconstruct 3D human pose and meshes from a single 2D image.Self-attention captures non-local interactions among mesh vertices and body joints, including distant parts such as hands and feet.

4 OPEN CHALLENGES & FUTURE DIRECTIONS

Transformer models face practical barriers from their high data and computational demands, especially for high-resolution vision, while lacking visual inductive biases and requiring improved interpretability. Future directions include scalable training, efficient self-attention, architectures tailored to visual inputs, and better attribution methods.

  • Core challenges: The main practical bottlenecks are the need for large training datasets, high computational costs, and difficulty visualizing and interpreting Transformer models.These challenges are identified as limiting Transformer applicability in practical settings.
  • Scalability: Scaling compute, model size, and training data improves performance, but only large models benefit from additional training data and compute.The scalability study spans five million to two billion parameters, 30 million to three billion images, and 1–10000 TPU core-days.
  • Efficient vision Transformers: Self-attention costs O(n^2) with n image patches, hindering high-resolution tasks such as object detection and segmentation.This cost is especially problematic when n = H × W for low-level vision inputs.
  • Efficient vision Transformers: Existing efficiency methods—including pooling, local windows, axial attention, low-rank projections, kernelization, and clustering—trade off complexity and accuracy or require special hardware.The survey calls for efficient self-attention applicable to high-resolution images on resource-limited systems without compromising accuracy.
  • Visual inductive biases: Because Transformers lack inherent visual inductive biases, they must learn concepts such as translation invariance and weight sharing from training data, while current architectures remain better tailored to language.The survey highlights the need for further intuitions and more efficient designs for visual inputs.
  • Interpretability: Attention scores do not necessarily provide optimal explanations, motivating relevance propagation methods that preserve a constant total relevancy throughout the network.The proposed attribution design handles both positive and negative attributions.

5 CONCLUSION

The survey examines self-attention approaches, focusing on Transformer and bidirectional encoding architectures built on self-attention. It covers foundational concepts before analyzing competing approaches in computer vision.

  • Attention has enabled efficient and accurate computer vision systems while offering insights into deep neural network function.
  • The survey reviews self-attention approaches, especially Transformer and bidirectional encoding architectures.
  • It first presents fundamental concepts of self-attention architectures and then analyzes competing approaches.
Loading 2101.01169v5…