Source-linked AI summary
Global Attention Mechanism: Retain Information to Enhance Channel-Spatial Interactions
Yichao Liu, Zongru Shao, Nico Hoffmann
TL;DR
Prior attention mechanisms lose global spatial-channel interactions through information reduction and dimension separation. GAM retains information across dimensions using redesigned channel and spatial attention, and stably improves CNN performance across architectures and depths, while increasing network parameters.
Problem
Prior attention mechanisms use limited receptive fields and lose global spatial-channel interactions through information reduction and dimension separation.
Method
GAM redesigns sequential channel-spatial attention with 3D permutation and a two-layer MLP for channel attention alongside a convolutional spatial attention submodule.
Results
GAM stably improves CNN performance across different architectures and depths, including ResNet and MobileNet evaluations on CIFAR-100 and ImageNet-1K.
Takeaways & Limitations
GAM magnifies salient cross-dimension receptive regions and provides a feasible attention mechanism for CNNs with different architectures and depths.
Takeaways & Limitations
GAM increases the number of network parameters, motivating future parameter-reduction methods for large networks such as ResNet50 and ResNet101.
Abstract
from arXiv · showhide
A variety of attention mechanisms have been studied to improve the performance of various computer vision tasks. However, the prior methods overlooked the significance of retaining the information on both channel and spatial aspects to enhance the cross-dimension interactions. Therefore, we propose a global attention mechanism that boosts the performance of deep neural networks by reducing information reduction and magnifying the global interactive representations. We introduce 3D-permutation with multilayer-perceptron for channel attention alongside a convolutional spatial attention submodule. The evaluation of the proposed mechanism for the image classification task on CIFAR-100 and ImageNet-1K indicates that our method stably outperforms several recent attention mechanisms with both ResNet and lightweight MobileNet.
1 Introduction
Prior attention mechanisms improved vision tasks but often reduced information and separated channel and spatial dimensions, losing global cross-dimension interactions. GAM addresses this gap by retaining information to magnify such interactions.
- Attention mechanisms improved performance across multiple computer-vision applications.
- Prior methods used limited receptive fields because of information reduction and dimension separation.
- GAM investigates attention across spatial-channel dimensions by reserving information for global cross-dimension interactions.
2 Related Works
Earlier attention modules addressed channel and spatial dimensions separately, sequentially, or in parallel, but did not fully capture their interactions. GAM instead targets features across all three dimensions simultaneously.
- SENet applies channel attention but is less efficient at suppressing unimportant pixels.
- CBAM and BAM combine channel and spatial attention but ignore channel-spatial interactions and consequently lose cross-dimension information.
- TAM uses pairwise attention among channel, spatial width, and spatial height, but processes only two dimensions at a time.
- GAM captures significant features across all three dimensions to magnify cross-dimension interactions.
3 Global Attention Mechanism (GAM)
GAM redesigns sequential channel-spatial attention to reduce information loss and magnify global dimension-interactive features. Its channel branch uses 3D permutation and an MLP, while its spatial branch uses convolutional fusion without pooling.
- GAM processes an input feature map F1 ∈ R^C×H×W into intermediate state F2 and output F3 using channel and spatial attention maps.
- GAM’s overview, channel branch, and spatial branch are presented in Figures 1, 2, and 3, respectively.
- Channel attention: The channel submodule uses 3D permutation to retain information across dimensions and a two-layer MLP to magnify channel-spatial dependencies.
- Spatial attention: The spatial submodule fuses spatial information with two convolutional layers and removes max-pooling to retain feature maps.
4 Experiment
GAM is evaluated on CIFAR-100 and ImageNet-1K with ResNet and MobileNet V2, showing stable performance improvements across architectures and ablations supporting both attention submodules.
- Classification benchmarking: GAM is benchmarked on CIFAR-100 and ImageNet-1K using ResNet and MobileNet V2 under matched evaluation conditions.The comparisons include SE, BAM, CBAM, TAM, and ABN.
- Classification benchmarking: GAM outperforms SE, BAM, and CBAM on CIFAR-100.The CIFAR-100 experiments evaluate GAM with and without group convolution.
- Classification benchmarking: GAM stably enhances performance across different neural architectures on ImageNet-1K.For ResNet18, it outperforms ABN with fewer parameters and better efficiency.
- Ablation studies: Both spatial and channel attention contribute to performance gains, while their combination produces a further improvement.The ablation study switches each attention type on and off separately before evaluating their combination.
- Ablation studies: GAM outperforms CBAM with and without max-pooling on ResNet18.This ablation tests whether max-pooling affects spatial attention performance.
5 Conclusion
The paper concludes that GAM improves CNN performance across architectures and depths by magnifying salient cross-dimension receptive regions. The evaluation provides a proof of concept, while detailed scaling capability and parameter reduction remain future directions.
- Conclusion: GAM magnifies salient cross-dimension receptive regions.This is the stated central design objective of the proposed mechanism.
- Scope and future work: The evaluation is presented as a proof of concept, and detailed scaling capability is identified as future work.The authors note that full ImageNet would better represent production applications but is expensive for large-model training.
- Scope and future work: GAM gains performance with increased network parameters, motivating future work on parameter reduction for large networks.The authors also plan to explore other cross-dimension attention mechanisms using parameter-reduction techniques.