PaperlayerView exact revision on arXiv

Source-linked AI summary

SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

Qing-Long Zhang Yu-Bin Yang

arXiv:2102.00240v1cs.CVcs.AI

TL;DR

Combining spatial and channel attention can improve computer-vision performance but increases computational overhead. The paper proposes Shuffle Attention, which groups channels into sub-features and uses Shuffle Units to jointly model spatial and channel dependencies. Across ImageNet-1k and MS COCO tasks, SA achieves higher performance with lower model complexity than state-of-the-art attention approaches.

  • Problem

    Fusing spatial and channel attention can improve performance but inevitably increases computational overhead.

  • Method

    Shuffle Attention groups channel dimensions into sub-features, processes them in parallel with Shuffle Units, aggregates them, and applies channel shuffle.

  • Results

    SA achieves lower model complexity while achieving outstanding performance across ImageNet-1k and MS COCO experiments.

  • Takeaways & Limitations

    SA is an extremely lightweight plug-and-play block that can significantly improve the performance of various deep CNN architectures.

Abstract

from arXiv · show

Attention mechanisms, which enable a neural network to accurately focus on all the relevant elements of the input, have become an essential component to improve the performance of deep neural networks. There are mainly two attention mechanisms widely used in computer vision studies, \textit{spatial attention} and \textit{channel attention}, which aim to capture the pixel-level pairwise relationship and channel dependency, respectively. Although fusing them together may achieve better performance than their individual implementations, it will inevitably increase the computational overhead. In this paper, we propose an efficient Shuffle Attention (SA) module to address this issue, which adopts Shuffle Units to combine two types of attention mechanisms effectively. Specifically, SA first groups channel dimensions into multiple sub-features before processing them in parallel. Then, for each sub-feature, SA utilizes a Shuffle Unit to depict feature dependencies in both spatial and channel dimensions. After that, all sub-features are aggregated and a "channel shuffle" operator is adopted to enable information communication between different sub-features. The proposed SA module is efficient yet effective, e.g., the parameters and computations of SA against the backbone ResNet50 are 300 vs. 25.56M and 2.76e-3 GFLOPs vs. 4.12 GFLOPs, respectively, and the performance boost is more than 1.34% in terms of Top-1 accuracy. Extensive experimental results on common-used benchmarks, including ImageNet-1k for classification, MS COCO for object detection, and instance segmentation, demonstrate that the proposed SA outperforms the current SOTA methods significantly by achieving higher accuracy while having lower model complexity. The code and models are available at https://github.com/wofmanaf/SA-Net.

1. INTRODUCTION

The paper identifies inefficiency in combining spatial and channel attention and proposes Shuffle Attention, which processes grouped sub-features with Shuffle Units to improve efficiency and accuracy.

  • Attention mechanisms improve representations by emphasizing essential features and suppressing unnecessary ones across computer vision tasks.The paper cites applications including image classification, object detection, and instance segmentation.
  • Combining spatial and channel attention can improve performance but generally suffers from convergence difficulty or inefficiency.
  • Shuffle Attention groups channel dimensions into sub-features and processes them in parallel with Shuffle Units.Each sub-feature integrates channel and spatial attention, followed by aggregation and channel shuffling for information communication.
  • ImageNet-1k experiments show higher accuracy than current state-of-the-art attention methods with fewer parameters.
  • Channel shuffling enables information communication between the aggregated sub-features.

2. RELATED WORK

Related work develops multi-branch and grouped-feature architectures for efficient representation learning, while attention methods model informative feature relationships across channels and spatial positions.

  • Multi-branch architectures: Multi-branch CNN architectures use split-transform-merge designs to improve accuracy and speed while easing the difficulty of training very deep networks.
  • Multi-branch architectures: ShuffleNets combine channel split and channel shuffle operations to trade off speed and accuracy.
  • Grouped Features: Grouped-feature methods organize channels or neurons into groups to learn spatial relationships or richer feature representations.
  • Attention mechanisms: Attention methods allocate greater emphasis to informative feature expressions while suppressing less useful ones.
  • Attention mechanisms: Self-attention models contextual information as a weighted sum across image positions, while SE and ECA-Net model channel relationships with different complexity profiles.

3. SHUFFLE ATTENTION

The Shuffle Attention module groups channels into sub-features, processes channel and spatial attention in parallel with Shuffle Units, then aggregates and shuffles channels for cross-group communication. Experiments validate improved feature representation and relevant-region focus while retaining lightweight integration into CNNs.

  • Feature Grouping and Attention: SA divides each input feature map into channel groups and splits every sub-feature into parallel channel-attention and spatial-attention branches.Channel attention uses global average pooling and learnable scaling and shifting, while spatial attention uses group normalization and a compact gating feature.
  • Aggregation: The two attention branches are concatenated, all sub-features are aggregated, and channel shuffle enables information flow across groups.The final output preserves the input size, facilitating integration with modern architectures.
  • Complexity and Integration: SA introduces 3C/G parameters per module, typically with G equal to 32 or 64, making its parameter overhead small relative to the full network.The module can also be implemented in a few lines of PyTorch or TensorFlow and replaces SENet blocks in SA-Net.
  • Validation: Top-1 accuracy statistically increases after SA, while channel shuffle adds an average gain of approximately 0.4% in each group.The comparison trains SA-Net50B without channel shuffle and SA-Net50 with channel shuffle on ImageNet-1k.
  • Visualization and Interpretation: Across network depths, feature-group activation distributions differ more between classes in later layers, while GradCAM visualizations show greater focus on relevant regions and object details than ResNet50.The visual comparison uses SA-Net50 at SA 5 3 and ResNet50 at layer4.2 on ImageNet-1k validation images.

4. EXPERIMENTS

Experiments evaluate SA-Net for classification, object detection, and instance segmentation, emphasizing accuracy alongside computational efficiency. Across these benchmarks, SA generally improves performance while maintaining low model complexity.

  • Classification on ImageNet-1k: SA-Net matches the original ResNet’s model complexity while gaining 1.34% Top-1 and 0.89% Top-5 accuracy on ResNet-50.With ResNet-101, the gains are 0.76% Top-1 and 0.59% Top-5 accuracy.
  • Classification on ImageNet-1k: SA is evaluated using network parameters, GFLOPs, and Top-1/Top-5 accuracy on ImageNet-1k.Table 1 compares different attention methods using these efficiency and effectiveness metrics.
  • Ablation Study: Removing Group Norm substantially reduces performance, while removing channel shuffle causes a smaller performance drop.The ablation attributes the latter effect to reduced information communication among groups.
  • Object Detection on MS COCO: Object detection experiments compare attention methods on COCO val2017 using bounding-box Average Precision.Table 3 reports object detection results and marks the best records and improvements.
  • Instance Segmentation on MS COCO: SA module improves Mask R-CNN instance segmentation over ResNet and other attention modules while using less model complexity.The gains are particularly larger for small objects, supporting generalization across computer vision tasks.

5. CONCLUSION

The conclusion presents SA as a lightweight attention module that combines spatial and channel dependencies through grouped sub-features and channel shuffling. Experiments support its use as a plug-and-play block, while future work extends the design to additional CNN architectures.

  • 5. CONCLUSION: SA groups channel dimensions into sub-features and uses Shuffle Units to capture spatial and channel dependencies in parallel.The sub-features are later aggregated and connected through channel shuffling.
  • 5. CONCLUSION: Channel shuffle enables information communication between the aggregated sub-features.This operation follows aggregation in the SA pipeline.
  • 5. CONCLUSION: SA is presented as an extremely lightweight plug-and-play block that improves performance across various deep CNN architectures.The conclusion describes this claim as supported by experimental results.
  • 5. CONCLUSION: Future work will explore SA’s spatial and channel attention modules in ShuffleNet, SKNet, and MobileNetV3 architectures.These architectures are identified as planned extension targets.
Loading 2102.00240v1…