Source-linked AI summary
$A^2$-Nets: Double Attention Networks
Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, Jiashi Feng
TL;DR
CNNs are limited in efficiently modeling long-range image and video relations because convolution operators focus on local features. The paper proposes a double attention block that gathers global features through second-order attention pooling and distributes them adaptively. Experiments on image and video recognition benchmarks report effective performance with efficient global feature modeling.
Problem
CNNs inefficiently model long-range interdependencies because convolution operators focus on local features and typically require stacked layers to capture global information.
Method
A2-Net uses a lightweight double attention block that gathers global features into a compact set through second-order attention pooling and adaptively distributes them to locations.
Results
The method is evaluated with extensive ablations and state-of-the-art comparisons on ImageNet-1k, Kinetics, and UCF-101 for image and video recognition.
Takeaways & Limitations
The proposed block provides a generic, easily inserted mechanism for capturing and distributing global information with little computational overhead.
Takeaways & Limitations
The stated setup defines “space” as complete feature maps or spatio-temporal video features and stores all values in 32-bit float.
Abstract
from arXiv · showhide
Learning to capture long-range relations is fundamental to image/video recognition. Existing CNN models generally rely on increasing depth to model such relations which is highly inefficient. In this work, we propose the "double attention block", a novel component that aggregates and propagates informative global features from the entire spatio-temporal space of input images/videos, enabling subsequent convolution layers to access features from the entire space efficiently. The component is designed with a double attention mechanism in two steps, where the first step gathers features from the entire space into a compact set through second-order attention pooling and the second step adaptively selects and distributes features to each location via another attention. The proposed double attention block is easy to adopt and can be plugged into existing deep neural networks conveniently. We conduct extensive ablation studies and experiments on both image and video recognition tasks for evaluating its performance. On the image recognition task, a ResNet-50 equipped with our double attention blocks outperforms a much larger ResNet-152 architecture on ImageNet-1k dataset with over 40% less the number of parameters and less FLOPs. On the action recognition task, our proposed model achieves the state-of-the-art results on the Kinetics and UCF-101 datasets with significantly higher efficiency than recent works.
1 Introduction
CNNs capture local features efficiently but model long-range interdependencies poorly without added depth and cost. The paper introduces a double attention block that gathers global features and distributes them adaptively, then evaluates it across image and video recognition benchmarks.
- Motivation: CNN convolution operators focus on local neighborhoods, making long-range interdependencies inefficient to model.Stacking layers increases computation, memory use, over-fitting risks, and optimization difficulty.
- Related work: Recent alternatives partially address the limitations but are described as either non-flexible or computationally expensive.The comparison concerns recent works including Squeeze-and-Excitation and Non-local approaches.
- Approach: The double attention block gathers long-range features into a compact set and adaptively assigns them to locations.Its first attention operation captures second-order feature statistics, while the second performs adaptive feature assignment.
- Approach: The proposed A2-block is designed as an efficient component with low computational and memory footprint for long-range feature modeling.It is intended to let subsequent convolution layers sense the entire spatio-temporal space without a large receptive field.
- Evaluation: The paper studies A2-Net through extensive ablations and comparisons on ImageNet-1k, Kinetics, and UCF-101.The evaluations cover both image recognition and video action recognition tasks.
2 Method
The double-attention block gathers global features from the entire spatio-temporal input and distributes location-specific selections back to each position. It uses second-order attention pooling and adaptive feature distribution, with equivalent matrix associations offering different computational and memory costs.
- Core formulation: The block gathers features from the entire input space into a compact set, then distributes them to each location conditioned on its local feature.This enables subsequent convolution layers to access global information with small kernels.
- Feature gathering: Second-order attention pooling adaptively aggregates local features and can represent global texture, lighting, objects, or object parts through dense or sparse attention.The implementation uses transformed feature maps A = φ(X; Wφ) and B = softmax(θ(X; Wθ)).
- Feature distribution: Unlike SENet's single globally averaged feature, the second attention step selects an adaptive subset of gathered visual primitives for each location.The selected features are intended to complement the local feature and capture more complex relations.
- Block implementation: The double-attention operation combines gathering and distribution using feature arrays generated by separate convolution layers, followed by matrix multiplications and an output projection.The projected output is encoded back into the input through element-wise addition.
- Efficiency: The two matrix-multiplication associations are mathematically equivalent but differ in computational complexity and memory consumption.Left association has second-multiplication complexity O(mndhw), whereas right association has O(m(dhw)^2).
- Relation to prior work: The right-associated form resembles non-local networks but uses a different pairwise relation function based on two softmax-normalized projections.The paper contrasts this design with Embedded Gaussian relations and notes practical cost and memory concerns for such methods.
3 Experiments
Experiments evaluate A2-Net through ablations and comparisons across image and video recognition benchmarks. The results show improved accuracy and efficiency, including gains over deeper or nonlocal alternatives.
- Ablation studies: A single A2-block consistently improves performance over baseline ResNet-26 and deeper ResNet-29 models with little additional cost.The gain is more significant when the block is placed in top layers.
- Ablation studies: Adding more A2-blocks monotonically improves accuracy while using fewer FLOPs than the competing nonlocal blocks.Distributing blocks across different stages can produce larger accuracy gains than placing them all in one stage.
- Image recognition: A ResNet-50 with five A2-blocks outperforms ResNet-152 while using 6.5 GFLOPs and 33.0 M parameters.The A2-equipped ResNet-50 is reported as over 40% more efficient than ResNet-152 and more accurate than SENet.
- Video recognition: On Kinetics, A2-Net achieves higher accuracy than I3D and R(2+1)D while using fewer sampled frames.The comparison uses RGB input, and the model is built from ImageNet-pretrained ResNet-50 with five A2-blocks.
- Video recognition: On UCF-101, A2-Net achieves leading performance with significantly lower computational cost and transfers features effectively to the smaller dataset.UCF-101 results report averaged Top-1 video accuracy across three train/test splits.
4 Conclusions
The paper concludes that double attention addresses the locality of convolution by gathering global information and distributing it adaptively. Its lightweight design is effective across image and video recognition benchmarks, while future work targets compact mobile-friendly architectures.
- Conclusion: Double attention captures global information and distributes it to every location through a two-step attention mechanism.The method is formulated as a lightweight block that can be inserted into existing CNNs with little computational overhead.
- Conclusion: Experiments on ImageNet-1k, Kinetics, and UCF-101 confirm A2-Net's effectiveness for both 2D image and 3D video recognition.The conclusion reports results across both recognition settings.
- Future work: Future work will integrate double attention into compact network architectures for smaller, mobile-friendly models.This direction aims to leverage the method's expressiveness in resource-constrained architectures.