Source-linked AI summary
Efficient Multi-Scale Attention Module with Cross-Spatial Learning
Daliang Ouyang, Su He, Guozhong Zhang, Mingzhu Luo, Huaiyong Guo, Jian Zhan, Zhijie Huang
TL;DR
Attention modules improve visual representations, but channel dimensionality reduction can impair deep-feature extraction. EMA retains per-channel information through grouped multi-scale parallel branches and cross-spatial interaction, achieving strong accuracy with limited computational overhead across classification and detection benchmarks.
Problem
Channel dimensionality reduction can bring side effects when attention mechanisms extract deep visual representations, motivating efficient attention without this reduction.
Method
EMA reshapes partial channels into batch dimensions, groups channels into sub-features, uses parallel 1x1 and 3x3 branches, and fuses their maps through cross-spatial interaction.
Results
EMA achieves 74.4% ImageNet Top-1 validation accuracy with 306M multiply-adds, while experiments report consistent improvements across classification and object detection.
Takeaways & Limitations
EMA is presented as a flexible, lightweight attention module that can be integrated into different computer-vision tasks and modern CNN architectures.
Takeaways & Limitations
Triplet-style attention can add parameters and calculations, while simple averaging of branch weights may limit deep-feature discriminability.
Abstract
from arXiv · showhide
Remarkable effectiveness of the channel or spatial attention mechanisms for producing more discernible feature representation are illustrated in various computer vision tasks. However, modeling the cross-channel relationships with channel dimensionality reduction may bring side effect in extracting deep visual representations. In this paper, a novel efficient multi-scale attention (EMA) module is proposed. Focusing on retaining the information on per channel and decreasing the computational overhead, we reshape the partly channels into the batch dimensions and group the channel dimensions into multiple sub-features which make the spatial semantic features well-distributed inside each feature group. Specifically, apart from encoding the global information to re-calibrate the channel-wise weight in each parallel branch, the output features of the two parallel branches are further aggregated by a cross-dimension interaction for capturing pixel-level pairwise relationship. We conduct extensive ablation studies and experiments on image classification and object detection tasks with popular benchmarks (e.g., CIFAR-100, ImageNet-1k, MS COCO and VisDrone2019) for evaluating its performance.
1. Introduction
Deep CNNs improve feature representation but increase memory, computation, and latency. EMA addresses attention-related efficiency concerns by avoiding channel dimensionality reduction, using grouped parallel branches, and applying cross-spatial learning.
- Deep CNNs enhance feature representation but require more memory and computation as convolutional depth increases.
- Attention mechanisms provide flexible, plug-in structures that strengthen discriminative feature representation in CNN backbones.
- Channel dimensionality reduction lowers computational burden but may cause side effects when extracting deep visual representations.
- EMA avoids dimensionality reduction and uses grouped parallel subnetworks with 1x1 and 3x3 convolutional kernels.
- EMA combines channel or spatial attention prediction with cross-dimensional interaction and is evaluated on classification and object-detection benchmarks.
2. Related Work
Prior work uses feature grouping, parallel structures, and multi-scale convolution to manage computation and capture spatial dependencies. EMA extends these ideas by replacing simple attention-map averaging with cross-spatial learning that models pixel-level relationships.
- Feature grouping: Grouped convolution distributes computation across groups, while later grouped architectures use sub-features to represent diverse spatial patterns.
- Multi-stream networks: Parallel substructures reduce sequential processing and support modeling long-range dependencies for pixel-wise tasks.
- EMA illustration: Figure 2 identifies g as divided groups and X Avg Pool and Y Avg Pool as horizontal and vertical one-dimensional global pooling.
- Attention aggregation: Triplet attention aggregates parallel-branch weights by simple averaging, which is described as unfavorable for improving deep-feature discriminability.
- Multi-scale convolution: Multi-scale convolution uses different kernel sizes and branches to collect and aggregate spatial information at multiple scales.
- Cross-spatial learning: EMA fuses parallel attention maps through matrix dot products to capture pixel-level pairwise relationships and highlight global context.
3. Efficient Multi-Scale Attention
The section revisits Coordinate Attention and develops EMA using parallel, grouped processing without channel dimensionality reduction. EMA combines multi-scale spatial attention to capture positional information, global context, and pixel-level relationships while preserving input size.
- Revisit Coordinate Attention (CA): Coordinate Attention uses global average pooling along spatial directions to embed positional information into channel attention maps.Its parallel one-dimensional routes encode long-range dependencies while preserving precise positional information.
- Revisit Coordinate Attention (CA): CA neglects interactions among entirely spatial positions, while its 1x1 convolution has limited receptive field for local cross-channel and contextual modeling.These limitations motivate EMA's cross-spatial and multi-scale design.
- Multi-Scale Attention (EMA) Module: EMA adopts parallel subnetworks and grouped features to learn channel descriptions without dimensionality reduction while avoiding deeper sequential processing.The design uses a shared 1x1 convolution branch from CA and parallel processing to reduce sequential depth.
- Multi-Scale Attention (EMA) Module: EMA aggregates two spatial attention maps with a Sigmoid function to capture pixel-level pairwise relationships and highlight global context.The resulting output retains the same size as the input feature map, supporting efficient integration into modern architectures.
- Multi-Scale Attention (EMA) Module: EMA combines long-range dependencies and precise positional information by fusing context from different scales.This cross-spatial information aggregation produces pixel-level attention for high-level feature maps.
4. Experiments
Experiments evaluate EMA across classification and detection benchmarks using standard CNN backbones and matched training settings. EMA generally improves accuracy while adding relatively little computational or parameter overhead.
- Experiments cover CIFAR-100, ImageNet-1k, MS COCO, and VisDrone2019 using ResNet, MobileNetV2, and YOLOv5 backbones.
- Feature-map visualization indicates that EMA focuses on more relevant regions with more object details than other attention methods.
- 74.4% ImageNet Top-1 accuracy is achieved by EMA with 306M multiply-adds, while using 3.55M parameters versus CA’s 3.95M.
- 57.8% mAP (0.5) and 38.4% mAP (0.5:0.95) are achieved by EMA on MS COCO with only 0.03M more FLOPs than YOLOv5s.
- 0.31% mAP (0.5) and 0.4% mAP (0.5:0.95) improvements over YOLOv5x are reported for EMA on VisDrone2019.
5. Ablation Study
The ablation study examines EMA’s cross-spatial learning and group size using ResNet50. EMA_32 provides stronger classification accuracy than EMA_16 at similar computational cost.
- The ablation compares EMA_no without cross-spatial learning, EMA_16 with group size 16, and EMA_32 with group size 32.
- Group size 16 results in relatively higher FLOPs and network parameters than group size 32 because channel reshaping reduces model parameters.
6. Conclusion
The paper presents EMA as a flexible, lightweight attention module that avoids some dimensionality reduction while targeting strong representation and computational efficiency. The authors identify broader applications and stacking in other CNN structures as future work.
- EMA combines attention mechanisms through a generic approach that avoids some form of dimensionality reduction via a universal convolution.
- The authors characterize EMA as flexible and lightweight for use across different computer vision tasks.
- Semantic segmentation and stacking EMA into other deep CNN structures are proposed as future work.