Source-linked AI summary
NAM: Normalization-based Attention Module
Yichao Liu, Zongru Shao, Yueyang Teng, Nico Hoffmann
TL;DR
Attention mechanisms had not considered using weight-contribution factors to suppress less salient features for model compression. NAM redesigns channel and spatial attention around batch-normalization scaling factors and an l1-based sparsity penalty, and the reported ResNet and MobileNet comparisons show higher accuracy or efficiency gains than competing mechanisms.
Problem
Prior attention mechanisms suppress less salient features but do not consider the contributing factors of weights, which could further suppress insignificant channels or pixels.
Method
NAM redesigns CBAM-style channel and spatial attention using batch-normalization scaling factors and adds an l1-norm regularization term to suppress less salient weights.
Results
NAM’s channel or spatial variants outperform four other mechanisms on ResNet50/CIFAR-100, while combined NAM outperforms three others on MobileNet/ImageNet with similar computation complexity.
Takeaways & Limitations
Experiments indicate that NAM provides efficiency gains on both ResNet and MobileNet.
Takeaways & Limitations
The authors have not yet completed detailed analyses of integration variations and hyperparameter tuning, or evaluated NAM on other architectures and applications.
Abstract
from arXiv · showhide
Recognizing less salient features is the key for model compression. However, it has not been investigated in the revolutionary attention mechanisms. In this work, we propose a novel normalization-based attention module (NAM), which suppresses less salient weights. It applies a weight sparsity penalty to the attention modules, thus, making them more computational efficient while retaining similar performance. A comparison with three other attention mechanisms on both Resnet and Mobilenet indicates that our method results in higher accuracy. Code for this paper can be publicly accessed at https://github.com/Christian-lyc/NAM.
1 Introduction
Attention mechanisms suppress less salient pixels or channels, but prior approaches do not consider the contribution of weights to attention. NAM addresses this by using batch-normalization scaling factors to build an efficient attention mechanism.
- Prior attention methods suppress less salient pixels or channels by exploiting mutual information across feature dimensions.
- NAM uses batch-normalization scaling factors, whose standard deviation represents weight importance, to suppress insignificant channels or pixels.
- NAM avoids the fully connected and convolutional layers used in SE, BAM, and CBAM.
2 Related work
Related attention modules differ in how they combine spatial, channel, and cross-dimension information. SENet, BAM, CBAM, and TAM represent successive designs for modeling these feature relationships.
- SENet integrates spatial information into channel responses using two multi-layer perceptron layers.
- BAM uses parallel spatial and channel submodules that can be embedded into each bottleneck block.
- CBAM applies channel and spatial attention submodules sequentially.
- TAM accounts for dimension correlation to address the omission of cross-dimension interactions.
3 Methodology
NAM is a lightweight attention module integrated like CBAM but redesigned around batch-normalization scaling factors for channel and spatial importance. An l1-based regularization term suppresses less salient weights.
- NAM adopts CBAM’s module integration and embeds the redesigned attention module at the end of each network block.In residual networks, it is placed at the end of residual structures.
- Channel attention uses a batch-normalization scaling factor to measure channel variance and indicate channel importance.
- The channel submodule normalizes each channel scaling factor into weights Wγ = γ_i / Σ_j γ_j and produces output features M_c.
- Spatial attention applies batch-normalization scaling to pixels, termed pixel normalization, with weights Wλ = λ_i / Σ_j λ_j.
- The loss adds an l1-norm regularization term weighted by p to suppress less salient attention weights.The penalty balances the regularization terms on γ and λ.
4 Experiment
NAM is evaluated against SE, BAM, CBAM, and TAM on ResNet/CIFAR-100 and MobileNet/ImageNet. The reported comparisons indicate that NAM’s attention variants outperform competing mechanisms, including at similar computation complexity.
- On ResNet50 with CIFAR-100, NAM using channel-only or spatial-only attention outperforms the other four attention mechanisms.The experiment uses the same preprocessing and training configurations as CBAM, with p = 0.0001.
- On MobileNet with ImageNet, combined channel and spatial NAM outperforms the other three mechanisms with similar computation complexity.The experiment sets p = 0.001 and otherwise follows CBAM configurations.
- The experiments compare NAM with SE, BAM, CBAM, and TAM across both ResNet and MobileNet settings.
5 Conclusion
NAM suppresses less salient features and provides efficiency gains on both ResNet and MobileNet. The authors identify further optimization and broader evaluation as future work.
- NAM improves efficiency by suppressing less salient features.
- Experiments indicate efficiency gains on both ResNet and MobileNet.
- Future work will examine integration variations, hyper-parameter tuning, compression techniques, and other architectures and applications.
A.1 Comparison of CBAM and NAM regarding the number of parameters
NAM uses fewer parameters than CBAM overall, with substantial reductions in channel attention and only a small spatial-attention increase on ResNet50.
- NAM has fewer parameters than CBAM overall.
- NAM significantly reduces parameters in the channel attention module compared with CBAM.
- NAM causes an insignificant parameter increase in the spatial attention module relative to CBAM.