PaperlayerView exact revision on arXiv

Source-linked AI summary

Attention Mechanisms in Computer Vision: A Survey

Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng-Tao Jiang, Tai-Jiang Mu, Song-Hai Zhang, Ralph R. Martin, Ming-Ming Cheng, Shi-Min Hu

arXiv:2111.07624v1cs.CV

TL;DR

Attention mechanisms address how computer-vision systems can focus on salient information by adaptively weighting features. This survey synthesizes and classifies visual-attention methods by operating domain, reviews their development and applications, and identifies future research needs. It concludes that attention models form an independent and potentially powerful area of computer vision, while their formal characterization and interpretability remain open challenges.

  • Problem

    Computer-vision attention methods lack a unified review and categorization, while their necessary and sufficient definition and precise interpretability remain unresolved.

  • Method

    The paper systematically reviews attention mechanisms, classifies them by data domain, and analyzes their motivations, formulations, functions, and development.

  • Results

    The survey shows that attention methods span multiple visual tasks and can be treated as an independent topic organized by domain rather than application task.

  • Takeaways & Limitations

    The domain-based synthesis provides a framework for understanding visual-attention methods and identifies directions for future research.

  • Takeaways & Limitations

    No necessary and sufficient condition for all attention mechanisms is established, and attention maps alone provide only an intuitive rather than precise understanding.

Abstract

from arXiv · show

Humans can naturally and effectively find salient regions in complex scenes. Motivated by this observation, attention mechanisms were introduced into computer vision with the aim of imitating this aspect of the human visual system. Such an attention mechanism can be regarded as a dynamic weight adjustment process based on features of the input image. Attention mechanisms have achieved great success in many visual tasks, including image classification, object detection, semantic segmentation, video understanding, image generation, 3D vision, multi-modal tasks and self-supervised learning. In this survey, we provide a comprehensive review of various attention mechanisms in computer vision and categorize them according to approach, such as channel attention, spatial attention, temporal attention and branch attention; a related repository https://github.com/MenghaoGuo/Awesome-Vision-Attentions is dedicated to collecting related work. We also suggest future directions for attention mechanism research.

1 INTRODUCTION

The survey introduces attention as adaptive feature weighting inspired by human visual selection, then classifies methods by data domain and traces four developmental phases. It reviews attention methods systematically and proposes future research directions.

  • 1 INTRODUCTION: Attention mechanisms adaptively weight input features to emphasize important regions and improve computer-vision systems.The motivation comes from the human visual system’s selective processing of complex scenes.
  • 1 INTRODUCTION: The field developed through four phases: RNN-based attention, explicit region prediction, implicit attention, and self-attention.RAM, STN, SENet, and non-local or transformer-style methods represent these phases respectively.
  • 1 INTRODUCTION: Attention methods are grouped into four basic domains—channel, spatial, temporal, and branch—and two hybrids: channel & spatial and spatial & temporal.The classification links methods by what, where, when, or which branch receives attention.
  • 1 INTRODUCTION: The survey’s contributions include a systematic review, data-domain categorization independent of application, and suggestions for future visual-attention research.The paper treats attention methods as a topic in their own right rather than only as task-specific components.

2 OTHER SURVEYS

The survey distinguishes its scope from prior surveys by covering attention mechanisms in computer vision broadly rather than focusing mainly on natural language processing, visual transformers, or task-specific applications.

  • 2 OTHER SURVEYS: Unlike NLP-focused surveys, this paper surveys attention mechanisms specifically in computer vision.Its scope centers on visual rather than natural-language applications.
  • 2 OTHER SURVEYS: Unlike visual-transformer surveys, it reviews attention mechanisms in vision generally, not only self-attention methods.The comparison places transformer-based methods within a broader attention taxonomy.
  • 2 OTHER SURVEYS: The survey organizes attention categories and representative works as a broader review of attention methods in their own right.The organization is presented independently of particular application tasks.

3 ATTENTION METHODS IN COMPUTER VISION

The main methods section establishes a general attention formulation and then reviews attention categories through their motivations, formulations, functions, development, and representative works.

  • 3 ATTENTION METHODS IN COMPUTER VISION: The survey first defines a general attention mechanism based on the recognition process of the human visual system.This provides a common formulation for the methods reviewed later.
  • 3 ATTENTION METHODS IN COMPUTER VISION: It then reviews the categories introduced in its data-domain taxonomy, dedicating a subsection to each category.Representative works are tabulated for every category.
  • 3 ATTENTION METHODS IN COMPUTER VISION: Each category is examined through its motivation, formulation, function, and development over time.The review combines conceptual explanation with representative-work summaries.

3.1 General form

The general form models attention as generating a selection signal over an input and then processing that input according to the selected discriminative information. The formulation is intended to encompass existing attention mechanisms, including self-attention and SE attention.

  • 3.1 General form: The formulation represents visual attention as focusing on discriminative regions and processing those regions efficiently.It is motivated by selective human scene processing.
  • 3.1 General form: g(x) generates attention for input x, while f(g(x), x) processes x using the resulting attention.Together, these functions encode selection followed by information processing.
  • 3.1 General form: The authors state that almost all existing attention mechanisms can be written in this formulation, using self-attention and SE attention as examples.The examples instantiate the attention-generation and input-processing components.
  • 3.1 General form: The section uses the formulation to introduce and specify the various attention mechanisms reviewed in the survey.The equation framework serves as a common language for later categories.

3.2 Channel Attention

Channel attention adaptively recalibrates feature-map channels to determine what to emphasize, while successive methods improve global-information modeling, channel interactions, and computational efficiency. The survey organizes these mechanisms by their attention process and reviews representative designs and applications.

  • Overview: Channel attention adaptively recalibrates channel weights, treating channels as object-selection units that determine what the network should emphasize.The survey describes channel attention through the attention process g(x) and subsequent feature processing f(g(x), x).
  • SENet: SENet uses global average pooling to collect global information, then fully connected and nonlinear layers to produce an attention vector that scales each channel.The squeeze module gathers spatial information, while the excitation module models channel-wise relationships with ReLU and sigmoid activations.
  • SENet: SENet’s simple pooling limits complex global-information modeling, while fully connected excitation layers increase model complexity and motivate later improvements.Subsequent methods target the squeeze module, excitation module, or both.
  • GSoP-Net: GSoP-Net uses second-order pooling and covariance modeling to capture channel correlations and improve global-information collection over SE blocks, at additional computational cost.Its covariance matrix explicitly relates channel pairs, and the survey notes that GSoP blocks are typically inserted after several residual blocks.
  • SRM: SRM combines mean and standard-deviation pooling with a lightweight channel-wise fully connected layer to improve global-information capture while reducing excitation cost.The resulting block can be added after each residual unit like an SE block.
  • Later developments: Other channel-attention designs include GCT’s explicit channel interaction, ECA’s adaptive local cross-channel convolution, and frequency-based or contextual mechanisms for richer representations.The reviewed examples include DCT-based multi-spectral attention, context encoding with SE-loss, and bilinear pooling for higher-order feature interactions.

3.3 Spatial Attention

Spatial attention adaptively selects important image regions, using recurrent, transformation-based, implicit-mask, or self-attention mechanisms. Across examples, it reduces computation, improves representation, supports interpretability, and models global context, though self-attention has quadratic complexity.

  • Overview: Spatial attention adaptively selects image regions by determining where the model should focus.Representative approaches include recurrent attention, spatial transformers, implicit soft masks, and self-attention.
  • Recurrent attention: RAM uses glimpse sensing, a glimpse network, and an RNN to sequentially predict attended locations from multi-resolution image patches.The model receives a coordinate and image, extracts centered patches, and predicts the next center coordinate and action.
  • Recurrent attention: RAM reduces calculations, especially for large inputs, while improving image classification results.Its recurrent policy concentrates computation on key regions rather than processing the entire image uniformly.
  • Recurrent attention: Deep recurrent visual attention processes small glimpses sequentially, lowering computational cost and handling images of different sizes.The model also improves robustness and alleviates over-fitting through recurrent attention.
  • Applications: Attention-based image captioning improves interpretability by showing what and where the model focuses while strengthening representation capability.Hard attention treats weights as location probabilities, whereas soft attention treats them as relative importance for the next word.
  • Self-attention: Self-attention models global information, while ViT shows that pure attention can outperform CNNs on large datasets such as JFT-300 and ImageNet-21K.CNNs’ localized operations limit global scene understanding, motivating self-attention and transformer-based alternatives.

3.4 Temporal Attention

Temporal attention dynamically selects when to focus in video, combining local and long-range frame dependencies. GLTR captures multi-scale temporal cues but has quadratic self-attention complexity, while TAM uses adaptive kernels at lower computational cost.

  • Overview: Temporal attention dynamically selects relevant times and is mainly used for video processing.Representative methods target short-term and long-term cross-frame feature dependencies.
  • GLTR: GLTR combines a dilated temporal pyramid for local context with temporal self-attention for global temporal interaction.Its dilated convolutions use progressively increasing dilation rates to cover multiple temporal ranges.
  • GLTR: Short-term context helps distinguish visually similar regions, while long-term information addresses occlusions and noise.GLTR combines both forms of context to enhance representation capability and suppress noise.
  • GLTR: GLTR can learn a global video descriptor with CNN backbones, but its self-attention has quadratic time complexity.This complexity limits the method’s application.
  • TAM: TAM uses local and global branches with adaptive kernels to capture complex temporal relationships more efficiently than GLTR.The local branch produces location-sensitive temporal maps, while the global branch generates channel-wise kernels from global temporal information.
  • TAM: TAM captures complex temporal structures at low computational cost and can be added to existing 2D CNNs.Its lightweight design supports flexible integration into conventional image-based convolutional backbones.

3.5 Branch Attention

Branch attention dynamically selects or fuses branches in multi-branch networks. Highway networks route information adaptively, SK convolution selects receptive-field scales, and CondConv or dynamic convolution increase representational capacity with low overhead.

  • Overview: Branch attention dynamically selects which branch to emphasize in a multi-branch structure.Its reviewed purposes include routing information, fusing branches, selecting receptive fields, and combining convolution kernels.
  • Highway networks: Highway networks use adaptive transform gates and skip connections to route information across layers and train very deep networks.Unlike fixed skip connections, the gating mechanism adapts to the input and can be incorporated into CNNs.
  • SK convolution: SK convolution splits features across different kernel sizes, fuses branch information, and selects outputs using a gate vector.Different kernel sizes provide multiple receptive-field scales, while the gate controls information flow from the branches.
  • SK convolution: SK convolution adaptively adjusts receptive-field sizes according to the input with notable improvement at little computational cost.Its lightweight design allows replacement of large-kernel convolutions in CNN backbones.
  • CondConv: CondConv adaptively combines multiple convolution kernels, providing an efficient multi-branch operator that increases network capability.Combining kernels before convolution is equivalent to a mixture of experts but has much lower computational cost.
  • Dynamic convolution: Dynamic convolution uses attention-weighted parallel kernels to improve representational power with negligible additional computational cost.The weighted kernels are aggregated before being applied to the input and can replace ordinary convolution.

3.6 Channel & Spatial Attention

Channel-and-spatial attention jointly selects informative channels and regions. The surveyed modules use parallel or sequential designs to improve representation and dense prediction efficiently, while retaining limitations such as restricted long-range context or receptive fields.

  • Overview: Channel-and-spatial attention adaptively selects important objects and regions across channel and spatial dimensions.The category combines channel attention’s feature selection with spatial attention’s region selection.
  • Residual attention: Residual attention networks use trunk and mask branches, with a bottom-up top-down mask softly weighting trunk features.The structure models spatial and cross-channel dependencies and can be trained end-to-end in deep networks.
  • CBAM: CBAM sequentially infers channel and spatial attention maps, combining cross-channel relationships with spatial relationships at negligible additional cost.Its channel module uses multiple pooling operations, while its spatial module uses a convolution to generate the spatial map.
  • Limitations: CBAM’s spatial sub-module may suffer from a limited receptive field because it generates the spatial map with a convolution.This leaves room for improving spatial-context modeling within lightweight channel-and-spatial attention.
  • BAM: BAM uses parallel spatial and channel streams, with dilated convolutions enlarging the spatial receptive field while a bottleneck structure saves computation.The two attention maps are resized and summed before modulating the input through a residual formulation.
  • Limitations: BAM improves representation with little extra cost but does not capture long-range context or encode cross-domain relationships effectively.Its dilated convolutions enlarge the receptive field, yet the cited limitation remains for broader contextual modeling.
  • scSE: scSE combines spatial and channel excitation to enhance features and capture pixel-wise spatial information, consistently improving semantic segmentation at negligible extra cost.Spatial excitation complements channel excitation by focusing weights on important regions.

3.6.5 Triplet Attention

Triplet attention addresses the limitation of independently computed channel and spatial attention by modeling interactions across pairs of feature-map domains. The survey also contrasts related mechanisms that encode 3D weights, positional information, global context, or pairwise structural relations.

  • Triplet Attention: Unlike CBAM and BAM, triplet attention explicitly models cross-domain interactions rather than computing spatial and channel attention independently.The survey describes this design as capturing richer discriminative feature representations while remaining lightweight.
  • Triplet Attention: Triplet attention uses three rotated branches to capture cross-domain interactions among height, width, and channel dimensions.Each branch applies axis rotation, Z-pooling, and a k × k convolution to model relationships between two domains.
  • SimAM: SimAM directly estimates 3D attention weights with a simple parameter-free module based on neuroscience theory.Its design avoids manual fine-tuning of the network structure and uses a sigmoid followed by element-wise multiplication for the output.
  • Coordinate Attention: Coordinate attention embeds horizontal and vertical positional information into channel attention through directional pooling and shared convolutional transformation.It is described as combining a larger receptive field with cross-channel modeling in a lightweight module suitable for mobile-network building blocks.
  • DANet: DANet uses parallel position and channel attention modules to capture global spatial dependencies and enhance useful channels while suppressing noise.The survey notes that this explicit modeling improves scene-segmentation representations but is computationally costly for large feature maps.
  • Relation-Aware Global Attention: Relation-aware global attention generates attention scores from global pairwise relations and can jointly model spatial and cross-channel structure.RGA-S and RGA-C are presented as flexible modules for CNN backbones and can be used sequentially.

3.6.10 Self-Calibrated Convolutions

Self-calibrated convolution enlarges receptive fields by combining pooled, transformed, and upsampled features with standard convolution. The section also surveys pooling and attention modules that integrate spatial, channel, semantic, global, or local information.

  • Self-Calibrated Convolutions: Self-calibrated convolution splits input channels, uses average pooling and convolution to enlarge the receptive field, upsamples the result, and applies element-wise multiplication.The resulting module is used with standard convolution to improve adaptability.
  • Self-Calibrated Convolutions: Self-calibrated convolution achieves excellent results in image classification, instance segmentation, object detection, and keypoint detection.The survey presents these as downstream tasks where the module improves performance.
  • Strip Pooling: Strip pooling encodes long-range context through separate horizontal and vertical pooling branches whose outputs are fused into an attention map.Each direction uses one-dimensional convolution to model relationships across rows and channels or their vertical counterparts.
  • Mixed Pooling: The mixed pooling module extends strip pooling by considering spatial and channel relationships, and SPNet achieves state-of-the-art results on several complex semantic-segmentation benchmarks.Both modules address the locality of convolutional neural networks.
  • SCA-CNN: SCA-CNN uses an encoder-decoder with spatial and channel-wise attention guided by the LSTM hidden state for image captioning.The spatial branch emphasizes semantically useful regions, while channel attention aggregates global information before computing channel weights.
  • SCA-CNN: SCA-CNN uses semantic information to determine both where and what the model should focus on during sentence generation.The attention model is presented as offering interpretive information beyond a generic attention mechanism.

3.7 Spatial & Temporal Attention

Spatial and temporal attention mechanisms select important regions and frames in video, either separately, jointly, serially, or through learned relations. The survey highlights their use for action recognition and person re-identification, alongside benefits and computational strategies.

  • STA-LSTM: Joint spatial-temporal attention adaptively identifies discriminative joints and keyframes for human action recognition.The LSTM-based method contains separate spatial and temporal attention subnetworks and reports excellent action-recognition results.
  • RSTAN: RSTAN applies spatial and temporal attention serially to produce spatiotemporal feature representations for action recognition.Its spatial module forms global frame features, after which temporal attention estimates the importance of each frame feature.
  • RSTAN: RSTAN identifies prediction-related regions across spatial and temporal domains and enhances the representation power of a 2D CNN.The mechanism uses the current-step RNN context to guide attention.
  • STA: STA assigns attention scores to spatial regions across frames without extra parameters, jointly using spatial and temporal information for video person re-identification.Its non-parametric design supports variable-length input sequences and reduces occlusion influence.
  • STGCN: STGCN models video patches as graph nodes and uses pairwise-attention-weighted adjacency matrices with temporal and structural graph-convolution branches.The method captures spatiotemporal patch relationships, addresses occlusion, and can be combined with any CNN backbone.

4 FUTURE DIRECTIONS

The survey identifies open directions involving formal definitions, task-general attention blocks, interpretability, pretraining, optimization, and deployment. It emphasizes that current methods remain task-specific, difficult to characterize precisely, and challenging to deploy efficiently on edge devices.

  • Necessary and sufficient condition for attention: Equation 1 is necessary but not sufficient to characterize all attention mechanisms, leaving necessary-and-sufficient conditions unresolved.The survey gives GoogleNet as an example that conforms to the formula without being an attention mechanism.
  • General attention blocks: Current attention mechanisms often require task-specific design, motivating general blocks that can select among channel, spatial, and other attention types.The survey suggests branch attention as one possible soft-selection mechanism.
  • Characterisation and interpretability: Attention maps provide intuitive but not precise understanding, so safety-critical applications need better characterization of model behavior and failure modes.The survey specifically names medical diagnosis and automated driving as demanding stricter interpretability requirements.
  • Pretraining: Attention-based pretrained models have demonstrated suitability for visual tasks and can transfer pretrained weights across tasks because they adapt to varying inputs.The survey cites MoCoV3, DINO, BEiT, and MAE as examples.
  • Optimization: Different attention-based and convolutional models may benefit from different optimization methods, motivating further optimizer research for attention mechanisms.The survey contrasts SGD and Adam for CNNs with AdamW and SAM for visual transformers.
  • Deployment: Complex and varied attention models are difficult to optimize on edge devices despite reported performance advantages over CNNs.The survey calls for simple, efficient, and effective attention models suitable for broad deployment.

5 CONCLUSIONS

The survey systematically reviews and classifies computer-vision attention mechanisms by their domain of operation, then identifies future research directions.

  • The survey organizes attention methods by their domain of operation rather than by application task.
  • It argues that attention models constitute an independent topic within computer vision.
  • The survey outlines potential future research directions and aims to support both practical use and deeper understanding of relationships among attention mechanisms.
Loading 2111.07624v1…