Source-linked AI summary
BAM: Bottleneck Attention Module
Jongchan Park, Sanghyun Woo, Joon-Young Lee, In So Kweon
TL;DR
Existing work explored attention in deep networks, but this paper investigates a lightweight, general-purpose way to improve representational power. BAM uses separate channel and spatial pathways at network bottlenecks, and experiments report consistent gains across classification and detection benchmarks with negligible overhead.
Problem
The paper investigates how attention can improve the representational power of general deep neural networks efficiently rather than serving only task-specific purposes.
Method
BAM generates 3D attention maps through separate channel and spatial pathways, placing modules at bottlenecks for end-to-end integration with CNNs.
Results
BAM consistently improves classification and detection across CIFAR-100, ImageNet-1K, VOC 2007, and MS COCO with negligible parameter and computational overhead.
Takeaways & Limitations
BAM provides a broadly applicable, lightweight way to enhance network capacity across varied architectures and tasks.
Abstract
from arXiv · showhide
Recent advances in deep neural networks have been developed via architecture search for stronger representational power. In this work, we focus on the effect of attention in general deep neural networks. We propose a simple and effective attention module, named Bottleneck Attention Module (BAM), that can be integrated with any feed-forward convolutional neural networks. Our module infers an attention map along two separate pathways, channel and spatial. We place our module at each bottleneck of models where the downsampling of feature maps occurs. Our module constructs a hierarchical attention at bottlenecks with a number of parameters and it is trainable in an end-to-end manner jointly with any feed-forward models. We validate our BAM through extensive experiments on CIFAR-100, ImageNet-1K, VOC 2007 and MS COCO benchmarks. Our experiments show consistent improvement in classification and detection performances with various models, demonstrating the wide applicability of BAM. The code and models will be publicly available.
1 Introduction
BAM targets representational power by adding a lightweight, general attention module to convolutional networks. It uses separate channel and spatial pathways, is placed at bottlenecks, and improves performance across classification and detection benchmarks.
- Its two attention pathways learn what feature channels and where spatial locations to emphasize in a 3D feature map.
- Placing multiple BAMs at bottlenecks builds hierarchical attention while keeping parameter and computational overhead negligible.
- BAM improves classification on CIFAR-100 and ImageNet-1K and detection on VOC 2007 and MS COCO across varied baseline architectures.
- BAM is a simple attention module designed for integration with any CNN without architectural extras.
- The paper validates BAM’s design through extensive ablation studies alongside experiments using multiple benchmarks and baseline architectures.
Related Work
BAM extends attention beyond task-specific uses by combining channel and spatial attention in a lightweight module placed at network bottlenecks. Its design uses hierarchical bottleneck placement, contextual spatial processing, and efficient channel modeling.
- Prior attention mechanisms: Earlier attention work often targets task-specific or multimodal settings, whereas BAM investigates attention as an efficient way to improve general network representational power.Self-attention methods addressed general classification, but some incurred substantial computational and parameter overhead.
- BAM design: BAM exploits both channel and spatial attention axes and places the module at bottlenecks where feature maps are downsampled.Multiple bottleneck modules can form hierarchical attention across network stages.
- BAM design: Given F, BAM computes a 3D attention map M(F) from separate channel and spatial branches, then refines the feature map through residual attention.The refined output uses element-wise multiplication between F and M(F), together with residual addition.
- Channel attention branch: The channel branch globally averages each channel and uses a one-hidden-layer MLP with reduction ratio r to estimate inter-channel attention.Batch normalization adjusts the channel-branch scale, while the hidden activation size is reduced to C/r.
- Spatial attention branch: The spatial branch compresses channels with 1×1 convolutions and applies 3×3 dilated convolutions to aggregate contextual information over larger receptive fields.The branch produces a spatial attention map over H×W locations and uses the same reduction ratio r.
- Combining attention branches: The two branch outputs are resized to C×H×W and combined by element-wise summation to form the final attention map.The paper reports choosing summation for efficient gradient flow and empirically finding it best among the compared combining operations.
Experiments
Experiments evaluate BAM across classification and detection benchmarks, ablations, insertion locations, and comparisons with baseline modules. BAM consistently improves performance while adding little parameter or computational overhead.
- Benchmark evaluation: BAM is evaluated on CIFAR-100, ImageNet-1K, VOC 2007, and MS COCO using reproduced baseline architectures.The experiments cover image classification and object detection across multiple models.
- Ablation studies: Larger spatial-attention dilation values improve performance until saturation at dilation value 4.The dilation value controls receptive-field size in the spatial-attention branch.
- Ablation studies: Combining channel and spatial attention produces substantially stronger gains than using either branch alone.The ablation identifies both pathways as important for inferring the final attention map.
- Ablation studies: Bottleneck placement provides a better overhead/accuracy trade-off than placing BAM inside each convolution block in most models.The bottleneck location generally yields lower overhead and better accuracy, except for PreResNet 110.
- Classification results: BAM improves classification results across CIFAR-100 and ImageNet-1K models, with negligible parameter and computational overhead.On CIFAR-100, ResNet50 with BAM reaches 20.00% error using half the parameters of ResNet101, while ResNeXt29 8x64d with BAM reaches 16.71% error using half the parameters of ResNeXt29 16x64d.
- Detection results: BAM improves object-detection accuracy for strong VOC 2007 baselines and is also validated on MS COCO with ResNet101-Faster-RCNN.The VOC gains have negligible parameter overhead, including with a lightweight backbone.
5 Conclusion
BAM enhances network representation by refining intermediate features through efficient attention at bottlenecks. Experiments across four benchmarks found BAM outperformed baseline models and exhibited hierarchical reasoning.
- BAM enhances network representation by learning what and where to focus or suppress through two separate pathways.The module refines intermediate features efficiently.
- Placing attention modules at network bottlenecks targets critical points of information flow.The placement is inspired by the human visual system.
- BAM outperformed baseline models across CIFAR-100, ImageNet-1K, VOC2007, and MS COCO benchmarks.The evaluation used various state-of-the-art models.
- Visualizations of intermediate feature maps revealed a hierarchical reasoning process resembling human perception.
- The authors suggest that adaptive feature refinement at bottlenecks may help other vision tasks.