PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Visual Transformer

Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, Dacheng Tao

arXiv:2012.12556v6cs.CVcs.AI

TL;DR

Rapid growth in vision-transformer research makes it difficult to keep pace with new work. This survey organizes recent methods by task and reports excellent performance across a wide range of visual tasks.

  • Problem

    Rapid growth in vision-transformer research makes keeping pace with new work difficult, motivating a comprehensive overview of recent advances and future directions.

    Source: §1 · p. 1

  • Method

    The paper surveys vision transformers by application scenario, covering backbone, high/mid-level, low-level, and video tasks, as well as methods for efficient implementation.

    Source: §1 · p. 1, §3.6 · p. 14

  • Results

    Surveyed methods show excellent performance across a wide range of visual tasks, including backbone, high/mid-level, low-level, and video processing.

    Source: §4 · p. 16

  • Takeaways & Limitations

    The survey concludes that vision transformers’ potential remains incompletely explored and identifies unresolved challenges and future prospects.

    Source: §4 · p. 16

    Efficient vision transformers remain an open problem because existing models are computationally expensive and compression methods remain complex and may not suit computer vision.

    Source: §4.1 · p. 16

Abstract

from arXiv · show

Transformer, first applied to the field of natural language processing, is a type of deep neural network mainly based on the self-attention mechanism. Thanks to its strong representation capabilities, researchers are looking at ways to apply transformer to computer vision tasks. In a variety of visual benchmarks, transformer-based models perform similar to or better than other types of networks such as convolutional and recurrent neural networks. Given its high performance and less need for vision-specific inductive bias, transformer is receiving more and more attention from the computer vision community. In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages. The main categories we explore include the backbone network, high/mid-level vision, low-level vision, and video processing. We also include efficient transformer methods for pushing transformer into real device-based applications. Furthermore, we also take a brief look at the self-attention mechanism in computer vision, as it is the base component in transformer. Toward the end of this paper, we discuss the challenges and provide several further research directions for vision transformers.

1 INTRODUCTION

The paper surveys recent vision transformers, organizing models by application and reviewing their progress, limitations, and potential research directions.

1 INTRODUCTION

  • Transformers use self-attention to extract features, following successes in NLP that motivated their application to computer vision.The paper frames transformers alongside MLP/FC networks, CNNs, and RNNs, which have typically served different task types,,,,,. It notes that transformer-based models achieved significant breakthroughs in NLP [9], [10], [11].Source: §1 · p. 1
  • Vision transformers have expanded from image classification to detection, segmentation, image processing, and video understanding.ViT applies a pure transformer to image-patch sequences and achieved state-of-the-art performance on multiple image-recognition benchmarks [15]. The survey also cites DETR [16], deformable DETR [17], and Max-DeepLab [25] as examples across detection and segmentation.Source: §1 · pp. 1–2
  • The survey covers transformer models for backbone networks, high/mid-level vision, low-level vision, video processing, and efficient deployment.It treats high- and mid-level vision together because their distinction is becoming less clear in DNN-based systems. It notes that low-level transformer applications remain few, while video’s sequential structure suits transformers, which are beginning to perform on par with CNNs and RNNs.Source: §1 · pp. 1–2
  • The survey reviews standard transformers and self-attention, summarizes representative vision models and efficient methods, and concludes with challenges and research directions.It places NLP transformer methods and computer-vision self-attention in supplemental material, and limits coverage to representative works because of space constraints.Source: §1 · p. 2

2 FORMULATION OF TRANSFORMER

Transformer [9] is formulated as an encoder–decoder built from repeated blocks, with attention and supporting components described before the section discusses its use in computer vision.

2 FORMULATION OF TRANSFORMER

  • Transformer [9] uses an encoder to produce input encodings and a decoder to generate output sequences, with each block combining multi-head attention, a feed-forward network, a shortcut connection, and layer normalization.The decoder uses contextual information from the encoder’s output.Source: §2 · p. 2

2.1 Self-Attention

  • Self-attention computes pairwise query–key scores, scales them by √dk for gradient stability, converts them to probabilities with softmax, and uses those probabilities to weight value vectors.Higher-probability values receive more focus in subsequent layers.Source: §2.1 · p. 2
  • Decoder encoder–decoder attention derives keys and values from the encoder, while its queries come from the decoder’s previous layer.This distinguishes it from encoder self-attention, where queries, keys, and values come from the same input.Source: §2.1 · p. 3
  • Because self-attention lacks positional information, positional encodings are added to input embeddings; sinusoidal encodings support relative-position attention and extrapolation to longer sequences.Learned and relative positional encodings are also used in various models,, [10], [15].Source: §2.1 · p. 3
  • Multi-head attention uses separate query, key, and value projections to attend across different representation subspaces and focus on multiple positions simultaneously.With h heads, each query, key, and value vector has dimension dmodel/h.Source: §2.1 · p. 3

2.2 Other Key Concepts in Transformer

  • Each encoder and decoder sub-layer applies a two-linear-layer feed-forward network with a nonlinear activation, while residual connections and subsequent layer normalization support information flow.The described hidden-layer dimension is dh = 2048, and GELU is given as an example activation [51].Source: §2.2 · p. 4
  • The decoder’s final linear layer maps its vector to vocabulary-sized logits, and softmax converts them into word probabilities.Pre-LN places normalization inside the residual connection before attention or FFN; batch normalization usually performs worse on transformers,, [15],,.Source: §2.2 · p. 4
  • For computer vision, most transformers use the encoder as a feature extractor, capturing long-distance characteristics unlike CNNs’ local focus and enabling parallel computation unlike sequential RNN hidden-state updates.The passage notes that self-attention and fully connected layer outputs can be computed in parallel and accelerated.Source: §2.2 · p. 4

3 VISION TRANSFORMER

The section reviews transformer-based applications in computer vision, spanning visual tasks, self-attention, and model compression for efficiency.

3 VISION TRANSFORMER

  • The review covers image classification and high- and mid-level vision applications of transformer-based models.It also includes low-level vision and video processing.Source: §3 · p. 4
  • The section also summarizes applications of the self-attention mechanism in computer vision.Source: §3 · p. 4
  • Model compression methods for efficient transformers are included in the review.Source: §3 · p. 4

3.1 Backbone for Representation Learning

Vision-transformer backbones range from pure transformers to CNN–transformer hybrids, with self-supervised learning offering another route to visual representation learning. The survey describes their mechanisms, performance conditions, limitations, and avenues for improving their architectures and training.

3.1 Backbone for Representation Learning

  • Backbone approaches include pure transformers and CNN–transformer combinations, with Wu et al. [59] using ResNet features before tokenizing them for transformer-based classification.The survey summarizes backbone results in Table 2 and Figure 6 and notes that self-supervised learning is also explored.Source: §3.1 · p. 4, Figure 4 · p. 4

3.1.1 Pure Transformer

  • 88.36% ImageNet accuracy and 77.16% VTAB accuracy followed ViT pretraining on JFT-300M; DeiT-B reached 83.1% ImageNet top-1 without external data and 84.40% with token-based distillation [15],.ViT uses flattened image patches, learned patch embeddings, a class embedding, and positional embeddings; it performs modestly on mid-sized datasets because it lacks CNN inductive biases, but large-scale pretraining enables strong results.Source: §3.1.1 · pp. 4–5
  • Locality-enhancing variants combine local and global information through sub-patch modeling, local attention, shifted windows, regional tokens, or feature aggregation,, [62], [63],, [65], [66], [67],.Other approaches alter attention through cross-head communication, k-NN selection, convolutional augmentation, or channel-wise attention; pyramid, two-stream, U-net, and searched architectures also depart from ViT’s uniform stack, [70], [71], [72],,,, [81], [82],,,.Source: §3.1.1 · p. 5

3.1.2 Transformer with Convolution

  • CNN–transformer hybrids introduce locality through convolutional encodings, convolutional FFN layers, or convolutional stems, while BoTNet replaces late ResNet spatial convolutions with global attention [86], [95], [97], [98], [99],,,.BoTNet improves instance segmentation and object detection baselines with minimal latency overhead; convolutional stems can also speed ViT convergence and support AdamW or SGD without significant accuracy loss.Source: §3.1.2 · pp. 5–6

3.1.3 Self-supervised Representation Learning

  • iGPT uses autoregressive or BERT-style pixel prediction for self-supervised pretraining before classification fine-tuning, while later masked-image-modeling methods MAE and SimMIM achieve competitive downstream performance, [32],.iGPT clusters pixels into sequences and uses an encoder-decoder, unlike ViT’s local patches, encoder-only architecture, and supervised classification training.Source: §3.1.3 · pp. 6–7
  • MoCo v3 uses two augmented image crops and a query–key contrastive objective, but self-supervised ViT training remains vulnerable to instability even when patch-projection training is skipped [31],.Skipping patch projection can improve stability, but does not solve it; a learning rate that is too large can still destabilize training.Source: §3.1.3 · p. 7

3.1.4 Discussions

  • CNN–transformer combinations achieve better performance through complementary local and global connections, as shown in Figure 6.The survey argues that backbone improvements can benefit vision more broadly, while large-scale self-supervised pretraining remains an open goal.Source: Figure 6 · p. 6, §3.1.4 · p. 7

3.2 High/Mid-level Vision

Transformers have been explored across detection, segmentation, pose estimation, and other high/mid-level vision tasks. The survey highlights end-to-end designs and attention mechanisms while noting that key design questions and opportunities for further research remain.

3.2.1 Generic Object Detection

  • 10× less training cost and 1.6× faster inference speed distinguish Deformable DETR from DETR, using deformable attention around reference points to reduce computation and support multi-scale features.DETR [16] casts detection as end-to-end set prediction, removing hand-crafted anchor generation and NMS; it achieves accuracy and speed comparable to Faster R-CNN [13].Source: §3.2.1 · pp. 7–8

3.2.2 Segmentation

  • Segmentation methods span panoptic, instance, semantic, and medical tasks, using mask prediction, sequence matching, patch embeddings, and U-shaped transformer architectures.Max-DeepLab [25] directly predicts nonoverlapping masks and labels end-to-end; VisTR [34] segments instance sequences across frames, while Cell-DETR achieves state-of-the-art cell instance segmentation.Source: §3.2.2 · pp. 9–10

3.2.3 Pose Estimation

  • Pose-estimation transformers model global relationships among hand and human joints or mesh vertices from point clouds, images, or initial pose estimates.METRO [37] progressively reduces feature dimensions to predict 3D coordinates, while TransPose uses attention to capture long-range keypoint relationships and explain their dependencies.Source: §3.2.3 · p. 10

3.2.4 Other Tasks

  • 2.82% higher accuracy and 3.65× higher FPS with 5-times fewer parameters are reported for LSTR versus PolyLaneNet.Other work adapts DETR for crowded pedestrian scenes with dense queries and rectified attention [147], while a transformer encoder improves lane proposal-point detection using context [149].Source: §3.2.4 · p. 10
  • 74.5% and 64.5% MOTA on MOT17 and MOT20 are reported for TransTrack [158], which uses learned queries to detect new objects and track existing ones.For scene graphs, transformers operate on extracted object embeddings, while Texema uses pretrained T5 to create structured graphs from text and improve relational reasoning [153].Source: §3.2.4 · pp. 10–11
  • Transformer-based re-identification and point-cloud methods adapt representations to their input structures, with TransReID reporting state-of-the-art person and vehicle benchmarks.TransReID uses overlapping patches, jigsaw patch and side-information modules; Point Transformer uses permutation-invariant attention for point-set processing and shows strong 3D semantic-segmentation performance.Source: §3.2.4 · p. 11

3.2.5 Discussions

  • High-level vision transformers remain at a preliminary stage, with input embeddings, positional encodings, prediction losses, and self-attention design identified as key issues.Open directions include large-scale pretraining, reusable models fine-tuned across tasks, task-specific prior knowledge, and whether CNN or PointNet feature extraction is needed [17],,.Source: §3.2.5 · p. 11

3.3 Low-level Vision

Low-level vision uses transformers for image-output tasks such as generation and processing, which are more challenging than high-level tasks that output labels or boxes. Approaches span autoregressive and GAN-based generation as well as patch-based image processing.

3.3.1 Image Generation

  • TransGAN progressively increases feature-map resolution with a multi-scale discriminator, while ViTGAN [165] adds training-stabilization techniques and first shows transformer GANs can match state-of-the-art CNN GANs.TransGAN also uses grid self-attention, data augmentation, relative position encoding, and modified normalization; ViTGAN uses Euclidean-distance attention, self-modulated layer normalization, and implicit neural representations.Source: §3.3.1 · p. 11
  • Image Transformer generates pixels autoregressively and uses local self-attention over nearby generated pixels to reduce the cost of high-resolution generation while matching CNN-based models on generation and translation.It supports image-conditioned tasks such as super-resolution and inpainting, as well as unconditional and class-conditional generation.Source: §3.3.1 · pp. 11–12
  • Taming Transformer [38] combines VQGAN’s context-rich discrete image vectors with an autoregressive transformer that models long-range interactions, achieving state-of-the-art results across image-synthesis tasks.VQGAN uses a discriminator and perceptual loss to improve visual quality.Source: §3.3.1 · p. 12
  • DALL·E uses a two-stage discrete-VAE and autoregressive-transformer framework to generate images from captions, with CLIP [41] ranking samples; diffusion models later advanced image generation.The survey names DALLE2 [168] and Stable Diffusion as diffusion-model examples.Source: §3.3.1 · p. 12

3.3.2 Image Processing

  • TTSR uses reference-image patches to transfer relevant high-resolution textures into low-resolution images, enabling transformer-based reference-guided super-resolution.It selects high-resolution features with hard attention and transfers them to low-resolution features with soft attention.Source: §3.3.2 · p. 12
  • 2 dB in image denoising tasks is an example of IPT’s reported improvement; the model uses task-specific heads, tails, and embeddings, and ImageNet pre-training for multiple image-processing tasks.IPT is fine-tuned for super-resolution, denoising, and deraining after pre-training on synthetically degraded images; self-supervision is also used to improve generalization.Source: Figure 10 · p. 12
  • SceneFormer represents a 3D indoor scene as a sequence of objects and predicts their locations, categories, and sizes, outperforming conventional CNN-based methods in user studies.The transformer decoder predicts the object sequence and its attributes.Source: §3.3.2 · pp. 12–13
  • Image processing typically encodes images as token or patch sequences for transformer encoders and decoders, while generation predicts image patches or encoded tokens; the survey identifies task-specific architecture design as future work.The survey treats iGPT primarily as an image-classification attempt because its reported focus is fine-tuning for classification.Source: §3.3.2 · p. 13

3.4 Video Processing

Video transformers use spatial and temporal information across tasks such as synthesis, action recognition, and retrieval; modeling both dimensions helps video understanding, but efficient and accurate relationship modeling remains an open problem.

3.4.1 High-level Video Processing

  • High-level video transformers support action and group-activity recognition, retrieval, object detection, and multi-task learning on untrimmed videos.Action models use context-aware attention, temporal modules, and static/dynamic actor features; retrieval models capture long-range semantics or cross-modal cues. MEGA, spatiotemporal aggregation, and stacked fusion with a class conversion matrix [183] address detection and multi-task video.Source: §3.4.1 · p. 13

3.4.2 Low-level Video Processing

  • Frame and video synthesis use transformer architectures for parallel frame generation and recurrent generation of human actions from novel views, while inpainting fills missing regions across frames.ConvTransformer [171] is reported to outperform LSTM-based works with a more parallelizable architecture; video inpainting uses all input frames in parallel and spatial-temporal adversarial loss [28].Source: §3.4.2 · p. 13

3.4.3 Discussions

  • Transformers improve video processing by mining spatial and temporal information simultaneously, but efficiently and accurately modeling their relationships remains an open problem.Video's extra temporal dimension and data redundancy contribute to the challenge.Source: §3.4.3 · pp. 13–14

3.5 Multi-Modal Tasks

The section surveys transformer approaches to multimodal learning, including image-text transfer, text-conditioned image generation, and models that handle multiple tasks and modalities.

3.5 Multi-Modal Tasks

  • VideoBERT learns video-text representations from CNN-preprocessed video tokens; VisualBERT [186] and VL-BERT unify visual and text inputs for VQA and VCR, while SpeechBERT explores audio-text encoding for SQA.Source: §3.5 · p. 14
  • 76.2% top-1 accuracy on ImageNet-1K without ImageNet training labels: CLIP [41] demonstrates zero-shot transfer and reports transfer experiments across 30 CV benchmarks.CLIP jointly trains text and image encoders on 400 million image-text pairs, maximizing cosine similarity for matched embeddings and minimizing it for mismatches.Source: §3.5 · p. 14
  • DALL-E generates images matching text or extends existing images; CogView accepts Chinese text and is claimed to outperform DALL-E and earlier GAN methods without CLIP reranking.DALL-E uses a discrete variational autoencoder to compress images into tokens, then trains an autoregressive transformer on joint image-text tokens; CogView uses a similar VQ-VAE tokenizer.Source: §3.5 · p. 14
  • UniT jointly trains image and text encoders, a modality-conditioned decoder, and task-specific heads across object detection, language understanding, and vision-language reasoning, selecting a task each iteration.The paper reports satisfactory performance on every task with a compact set of model parameters.Source: §3.5 · p. 14
  • The review concludes that multimodal transformers can unify data and tasks across modalities, and identifies effective training and extensibility as future research directions, including GPT-4.Source: §3.5 · p. 14

3.6 Efficient Transformer

Because transformer models’ memory and computing demands hinder deployment on resource-limited devices, this section reviews methods for compression and acceleration, including pruning, decomposition, distillation, quantization, and compact architecture design.

3.6.1 Pruning and Decomposition

  • Pruning removes redundant heads, FFN/head subnetworks, image patches [192], projection dimensions [193], or layers, while decomposition replaces large matrices with smaller ones.Some layers may need only one attention head; layer dropping can be encouraged during training before whole layers are removed at test time.Source: §3.6.1 · pp. 14–15

3.6.2 Knowledge Distillation

  • Knowledge distillation trains thinner, shallower students from teacher outputs or features, using attention, layer-specific, and vision patch-manifold transfer methods [207],,,, [211],, [213].Examples include matching student embedding outputs with MSE losses and transferring knowledge through relationships between images and their patches [213].Source: §3.6.2 · p. 15

3.6.3 Quantization

  • Quantization reduces bits for weights or intermediate features,, with examples including binary inputs, 4-bit weights, k-means methods, and post-training vision-transformer quantization.For translation, Prato et al. claim their fully quantized 8-bit transformer is the first not to lose translation quality.Source: §3.6.3 · p. 15

3.6.4 Compact Architecture Design

  • Compact designs simplify attention with span-based dynamic convolution or decomposition-based “hamburger” layers, and use manual or neural-architecture search for efficient vision and hybrid models,.Linear attention approximations and sparse-graph approximations target O(N) sequence complexity.Source: §3.6.4 · pp. 15–16

4 CONCLUSIONS AND DISCUSSIONS

Vision transformers show strong performance across visual tasks, but their potential remains underexplored and important challenges remain. The survey discusses these challenges and outlines future research directions.

4.1 Challenges

  • Vision transformers still need computer-vision-specific architectures and broader application beyond the tasks studied so far.ViT [15] follows the standard NLP transformer [9], while an architecture designed specifically for computer vision remains to be explored.Source: §4.1 · p. 16
  • ViT [15] performs strongly on CIFAR and VTAB [236] classification, but its direct use for object detection has not outperformed CNNs [115].Transformers’ generalization is affected by data quality and their reliance on massive training datasets; robustness remains an open problem despite prior studies, [239],.Source: §4.1 · p. 16
  • Explaining why transformers work well on visual tasks remains difficult because they lack CNN inductive biases such as translation equivariance and locality.The survey notes intuitive explanations, including large-scale training overcoming inductive bias [15], positional embeddings, and over-parameterization, as a possible avenue for interpretability.Source: §4.1 · p. 16
  • 18 billion FLOPs are required for the base ViT [15] to process an image, versus about 600 million FLOPs for GhostNet, at similar performance.Existing compression methods remain complex, and methods designed for NLP may not suit computer vision, leaving efficient models needed for resource-limited devices.Source: §4.1 · p. 16

4.2 Future Prospects

  • Future work should balance transformer performance with resource cost, which affect real-world applicability and device deployment,.The survey identifies effectiveness and efficiency as correlated concerns.Source: §4.2 · p. 16
  • A unified transformer that handles many visual and other tasks is an exciting direction beyond today’s mostly single-task models.Examples include GPT-3 [11], IPT [19] for multiple low-level vision tasks, and Perceiver and Perceiver IO [251] across several domains.Source: §4.2 · p. 16
  • The future choice between CNNs and transformers remains open, with CNNs performing well on small datasets and transformers better on large datasets [15].The survey also asks whether massive-data training could let a simple transformer using only fully connected layers achieve satisfactory results.Source: §4.2 · pp. 16–17

APPENDIX · A1. General Formulation of Self-attention

This appendix frames self-attention as position-wise aggregation over all input positions and connects its normalized linear-embedding form to query, key, and value attention.

A1. General Formulation of Self-attention

  • Self-attention [9] computes each position’s response by scoring all positions and gathering their embeddings, a form of non-local filtering,.The paper notes that the machine-translation module [9] is, to some extent, the same as the preceding non-local filtering operations proposed for computer vision.Source: A1. General Formulation of… · p. 17
  • For input X, each output position y_i aggregates representations g(x_j) weighted by pairwise relationships f(x_i, x_j) and normalized by C(x_i).The indices span all input positions j; positions may represent space, time, or spacetime.Source: A1. General Formulation of… · p. 17
  • The pairwise function f can use an extension of the Gaussian function to measure similarity in an embedding space.The embeddings θ and ϕ may be any embedding layers.Source: A1. General Formulation of… · p. 17
  • With linear embeddings for θ, ϕ, and g, the normalization factor sums f(x_i, x_j) over j, yielding softmax weights along the position dimension.The embedding matrices map from d channels to d_k for θ and ϕ, and to d_v for g.Source: A1. General Formulation of… · p. 17
  • Setting W_q = W_θ, W_k = W_ϕ, and W_v = W_g expresses this formulation using the query, key, and value representations Q, K, and V.The output signal is Y ∈ R^{n×c}.Source: A1. General Formulation of… · p. 17
  • Wrapping the output with W_o initialized to zero lets the self-attention module be inserted into an existing model without changing its initial behavior.The final computer-vision output is formed from Y generated by the preceding formulation.Source: A1. General Formulation of… · p. 17

A2. Revisiting Transformers for NLP

Transformer replaced sequential recurrent processing with attention-based language modeling that supports parallel training and global dependencies. Its success led to pretrained models and applications across NLP domains and multimodal tasks.

A2. Revisiting Transformers for NLP

  • Vaswani et al. [9] introduced an encoder-decoder transformer using multi-head self-attention and feed-forward networks to model global sequence dependencies while enabling parallel training.Unlike RNNs, which process information sequentially, the transformer architecture supports training on larger datasets.Source: A2. Revisiting Transformers for NLP · p. 17
  • BERT [10] uses an encoder pretrained with masked language modeling and next sentence prediction, then fine-tuned for downstream tasks; it achieved state-of-the-art results on 11 NLP tasks.Sequence-level classification uses the first token representation, while token-level tasks classify all tokens.Source: A2. Revisiting Transformers for NLP · p. 17
  • GPT and GPT-2 use decoder-based left-to-right pretraining, while GPT-3 [11] has 175 billion parameters and performs downstream tasks without fine-tuning.GPT-3 was pretrained on 45 TB of compressed plaintext and performed strongly on language understanding and generation datasets.Source: A2. Revisiting Transformers for NLP · pp. 17–18
  • Transformer-based models have also been applied to other NLP-related domains and multimodal tasks, illustrating versatility beyond general NLP and motivating their use in computer vision.The survey turns next to transformer applications across computer vision tasks.Source: A2. Revisiting Transformers for NLP · p. 18

BioNLP Domain

  • BioBERT and SciBERT [266] apply transformers to biomedical and scientific text, while ClinicalBERT [267] uses clinical-note representations whose attention maps can help explain predictions.SciBERT was trained on 114M scientific articles.Source: A2. Revisiting Transformers for NLP · p. 18

A3. Self-attention for Computer Vision

Self-attention is a key transformer component and also a building block in CNNs, where it captures long-range interactions and strengthens high-level features for challenging vision tasks.

Image Classification

  • Classification attention divides into hard attention over image regions and soft attention over feature maps, supporting region selection, medical-image cropping, channel reweighting, and feature enhancement,,,,,,, [275],.SENet reweights convolutional channels with soft self-attention, while AG-CNN crops a sub-region using an attention heat map.Source: A3. Self-attention for Computer Vision · p. 18

Semantic Segmentation

  • Semantic segmentation methods use self-attention to model contextual pixel relationships, with spatial-and-channel attention, region aggregation, graph reasoning, and efficient approximations to reduce the cost of pixel-similarity computation,,,,,,,, [291],, [293],,.DANet applies attention across spatial and channel dimensions, while A2Net groups pixels into regions; CGNL, CCNet, and ISSA approximate or factorize dense affinities.Source: A3. Self-attention for Computer Vision · p. 18

Object Detection

  • For object detection, self-attention enables fully attentional detectors, global-context modeling, bidirectional information exchange, cross-scale interactions, and joint processing or bridging of object representations,,,,,.The fully attentional detector outperforms RetinaNet on COCO; Feature Pyramid Transformer addresses cross-scale interactions, and RelationNet++ uses a master representation as query and auxiliary representations as keys.Source: A3. Self-attention for Computer Vision · pp. 18–19

Other Vision Tasks

  • For other vision tasks, attention-based feature enhancement and fusion support human keypoint estimation, while self-attention and bidirectional attention aggregate context for monocular depth estimation,,,,.The depth-estimation methods use attention to model generalized context, scene-dependent context, or forward and backward information.Source: A3. Self-attention for Computer Vision · p. 19

Related papers

Loading 2012.12556v6…