PaperlayerView exact revision on arXiv

Source-linked AI summary

ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks

Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, Qinghua Hu

arXiv:1910.03151v4cs.CV

TL;DR

Existing channel-attention methods often improve CNN accuracy by increasing complexity, so ECA-Net learns attention without dimensionality reduction using efficient local cross-channel interaction. It adds 80 parameters and 4.7e-4 GFLOPs to ResNet-50 while improving Top-1 accuracy by 2.28%.

  • Problem

    Existing channel-attention methods improve accuracy at the cost of higher model complexity, motivating efficient channel-attention learning.

  • Method

    ECA avoids dimensionality reduction and uses 1D convolution with adaptively selected kernel size to capture local cross-channel interaction.

  • Results

    80 additional parameters and 4.7e-4 GFLOPs improve ResNet-50 Top-1 accuracy by 2.28%, while ECA-Net performs competitively across classification, detection, and segmentation.

  • Takeaways & Limitations

    ECA-Net is an extremely lightweight plug-and-play channel-attention block that improves various CNN architectures with low model complexity.

Abstract

from arXiv · show

Recently, channel attention mechanism has demonstrated to offer great potential in improving the performance of deep convolutional neural networks (CNNs). However, most existing methods dedicate to developing more sophisticated attention modules for achieving better performance, which inevitably increase model complexity. To overcome the paradox of performance and complexity trade-off, this paper proposes an Efficient Channel Attention (ECA) module, which only involves a handful of parameters while bringing clear performance gain. By dissecting the channel attention module in SENet, we empirically show avoiding dimensionality reduction is important for learning channel attention, and appropriate cross-channel interaction can preserve performance while significantly decreasing model complexity. Therefore, we propose a local cross-channel interaction strategy without dimensionality reduction, which can be efficiently implemented via $1D$ convolution. Furthermore, we develop a method to adaptively select kernel size of $1D$ convolution, determining coverage of local cross-channel interaction. The proposed ECA module is efficient yet effective, e.g., the parameters and computations of our modules against backbone of ResNet50 are 80 vs. 24.37M and 4.7e-4 GFLOPs vs. 3.86 GFLOPs, respectively, and the performance boost is more than 2% in terms of Top-1 accuracy. We extensively evaluate our ECA module on image classification, object detection and instance segmentation with backbones of ResNets and MobileNetV2. The experimental results show our module is more efficient while performing favorably against its counterparts.

1. Introduction

ECA-Net addresses the performance–complexity trade-off in channel attention by avoiding dimensionality reduction and efficiently modeling local cross-channel interactions. It adds very little computation while improving ResNet-50 Top-1 accuracy by 2.28%.

  • Motivation: Channel attention improves deep CNN performance, but increasingly sophisticated modules often raise model complexity and computational burden.Recent methods capture richer channel dependencies or combine channel and spatial attention, commonly at higher complexity.
  • Motivation: The SE block uses two fully connected layers with non-linearity and a Sigmoid function to generate channel weights, with dimensionality reduction controlling complexity.The analysis motivates avoiding dimensionality reduction while retaining appropriate cross-channel interaction.
  • Method: ECA avoids dimensionality reduction and captures local cross-channel interaction by considering each channel with its k neighbors after global average pooling.The module uses a fast 1D convolution, with kernel size k adaptively determined from channel dimension C.
  • Results: 80 additional parameters and 4.7e-4 GFLOPs enable ECA-Net50 to improve ResNet-50 by 2.28% in Top-1 accuracy.The ResNet-50 baseline has 24.37M parameters and 3.86 GFLOPs.
  • Evaluation: ECA-Net is evaluated on ImageNet-1K and MS COCO across multiple tasks and deep CNN architectures.The paper positions ECA as an extremely lightweight module that increases little model complexity while improving performance.

2. Related Work

Prior channel-attention research began with SE-Net and developed along feature aggregation and combined channel–spatial attention. ECA is positioned among efficient convolutions, targeting local cross-channel interaction because common efficient convolutions show limited effectiveness in attention modules.

  • SE-Net first introduced an effective mechanism for learning channel attention and achieved promising performance.
  • Subsequent attention modules primarily pursued enhanced feature aggregation or combined channel and spatial attention.Examples include CBAM, GSoP, GE, and scSE; GCNet integrates a simplified non-local network with an SE block.
  • Although group and depth-wise separable convolutions use fewer parameters, they show little effectiveness in attention modules.
  • ECA targets local cross-channel interaction, sharing similarities with channel local convolutions.

3. Proposed Method

ECA-Net diagnoses dimensionality reduction and cross-channel interaction in SENet, then proposes efficient channel attention without dimensionality reduction. It uses shared-parameter 1D convolution with adaptively selected kernel size to capture local channel dependencies efficiently.

  • Dimensionality Reduction: Avoiding dimensionality reduction improves channel-attention learning because direct channel-to-weight correspondence matters.Experiments show SE variants without dimensionality reduction outperform the original SE block, including a single fully connected layer outperforming the reduced two-layer design.
  • Cross-Channel Interaction: Cross-channel interaction benefits channel attention, but full channel connectivity requires C×C parameters and high model complexity.SE-Var3 outperforms the non-interacting SE-Var2, while its full matrix introduces many parameters.
  • Cross-Channel Interaction: Group convolutions capture local interactions with C^2/G parameters but gain little because they completely discard dependencies between groups.SE-GC provides no improvement over SE-Var2 across varying group counts.
  • ECA Module: The band-matrix ECA-NS limits each channel to k neighbors, using k×C parameters while retaining dependencies across groups.This design outperforms SE-GC and is more efficient than the block-diagonal alternative.
  • Adaptive Kernel Selection: ECA adaptively selects kernel size from channel dimension using a nonlinear mapping, assigning longer interaction ranges to higher-dimensional channels.The method uses the nearest odd kernel size and sets γ = 2 and b = 1 throughout the experiments.

4. Experiments · 4.1. Implementation Details

The experiments evaluate ECA across image classification, object detection, and instance segmentation, first studying kernel size and ImageNet comparisons, then testing ECA-Net on MS COCO detectors. ImageNet uses four CNN backbones, while MS COCO uses three detectors with ResNet-FPN backbones under specified training settings.

  • 4. Experiments: The experiments cover ImageNet classification and MS COCO object detection and instance segmentation.The study first assesses ECA kernel size and compares attention methods on ImageNet, then evaluates ECA-Net with Faster R-CNN, Mask R-CNN, and RetinaNet on MS COCO.
  • 4.1. Implementation Details: Four CNN backbones—ResNet-50, ResNet-101, ResNet-512, and MobileNetV2—are used for ImageNet evaluation.ResNet training follows the same data augmentation and hyper-parameter settings as prior work, with random 224×224 crops and horizontal flipping.
  • 4.1. Implementation Details: MS COCO evaluation uses Faster R-CNN, Mask R-CNN, and RetinaNet with ResNet-50 and ResNet-101 plus FPN backbones.All detectors are implemented with the MMDetection toolkit using default settings.
  • 4.1. Implementation Details: 800-pixel shorter-side resizing, SGD, weight decay of 1e-4, momentum of 0.9, and mini-batch size 8 define the reported detector training setup.The mini-batch uses 4 GPUs with 2 images per GPU.
  • 4.1. Implementation Details: Table 3 compares attention methods by network parameters (#.Param.), FLOPs, FPS, and Top-1/Top-5 accuracy.The table notes unavailable running-time comparisons for A2-Nets and AA-Net, and different training settings for AA-Net.
  • 4.1. Implementation Details: The reported hardware includes an Xeon Silver 4112 CPU@2.60GHz.This hardware specification appears in the implementation details.

4.2. Image Classification on ImageNet-1K

On ImageNet-1K, ECA’s performance depends on the 1D-convolution kernel size, with adaptive selection generally improving stability and avoiding manual tuning. Across CNN backbones, ECA is evaluated against attention modules and state-of-the-art architectures for both efficiency and accuracy.

  • Kernel-size analysis: Fixed kernel sizes achieve their best results at k = 9 for ResNet-50 and k = 5 for ResNet-101, showing backbone-dependent optima.ResNet-101 exhibits larger accuracy fluctuations (∼0.5%) than ResNet-50 (∼0.15%) under fixed kernel sizes.
  • Kernel-size analysis: Adaptive kernel selection usually outperforms fixed choices while avoiding manual cross-validation of k, yielding better and more stable results.The adaptive size is determined by Eq. (12).
  • Kernel-size analysis: ECA modules with various kernel sizes consistently outperform the SE block, supporting channel attention without dimensionality reduction.Figure 4 compares fixed and adaptive ECA settings with SENet as the baseline.
  • ResNet-50 comparison: Using ResNet-50 on ImageNet, ECA is compared with SENet, CBAM, A2-Nets, AA-Net, GSoP-Net, and GCNet on efficiency and Top-1/Top-5 accuracy.Efficiency includes parameters, FLOPs, and training/inference speed.
  • Comparison with state-of-the-art CNNs: ECA-Net101 outperforms ResNet-200, indicating improved deep-CNN performance with much less computational cost.The section also compares ECA-Net50 and ECA-Net101 with Inception-v3, ResNeXt, and DenseNet.

4.3. Object Detection on MS COCO

ECA-Net generalizes effectively to MS COCO object detection across Faster R-CNN, Mask R-CNN, and RetinaNet. It consistently improves over ResNet and SE blocks, while also matching or surpassing other attention modules in reported comparisons.

  • Experimental setup: ECA-Net is evaluated on MS COCO with Faster R-CNN, Mask R-CNN, and RetinaNet, comparing against ResNet and SENet backbones pretrained on ImageNet and fine-tuned on COCO.The evaluation covers both two-stage and one-stage detectors.
  • Faster R-CNN: 0.3% and 0.7% AP gains over SE blocks are achieved with Faster R-CNN using ResNet-50 and ResNet-101, respectively.Both SE and ECA improve detection performance over the corresponding ResNet baselines.
  • Mask R-CNN: 1.8% and 1.9% AP improvements over original ResNet are obtained with Mask R-CNN using 50- and 101-layer backbones, respectively.ECA also gains 0.3% and 0.6% over SE with ResNet-50 and ResNet-101, respectively, and is superior to NL while comparable to GC with lower model complexity using ResNet-50.
  • RetinaNet: 1.8% and 1.4% AP gains over original ResNet are reported for RetinaNet with 50- and 101-layer networks, respectively.ECA-Net improves over SE-Net by 0.2% with ResNet-50 and 0.4% with ResNet-101.

4.4. Instance Segmentation on MS COCO

On MS COCO instance segmentation with Mask R-CNN, ECA improves over the original ResNet and outperforms SE with lower model complexity. With ResNet-50, ECA surpasses NL and is comparable to GC, supporting its generalization across tasks.

  • 4.4. Instance Segmentation on MS COCO: ECA achieves a notable instance-segmentation gain over the original ResNet while using less model complexity than the SE block.The evaluation uses Mask R-CNN on MS COCO.
  • 4.4. Instance Segmentation on MS COCO: With ResNet-50 as backbone, ECA is superior to NL while having lower model complexity.This comparison is reported for Mask R-CNN instance segmentation on MS COCO.
  • 4.4. Instance Segmentation on MS COCO: With ResNet-50 as backbone, ECA performs comparably to the GC block.The result is part of the MS COCO instance-segmentation comparison.
  • 4.4. Instance Segmentation on MS COCO: These results indicate that ECA has good generalization ability across various tasks.The generalization conclusion follows the instance-segmentation comparisons.

5. Conclusion

The paper proposes ECA, an efficient channel-attention module for deep CNNs that targets low model complexity. ECA uses fast 1D convolution with adaptively determined kernel size and serves as a lightweight plug-and-play block for improving various deep CNN architectures.

  • 5. Conclusion: ECA targets effective channel attention for deep CNNs while maintaining low model complexity.The module is designed as an efficient alternative for learning channel attention.
  • 5. Conclusion: ECA generates channel attention through a fast 1D convolution.This mechanism supports the module’s efficient implementation.
  • 5. Conclusion: ECA adaptively determines the 1D-convolution kernel size using a non-linear mapping of channel dimension.The adaptive kernel size controls the module’s channel-interaction scope.
  • 5. Conclusion: ECA is an extremely lightweight plug-and-play block that improves the performance of various deep CNN architectures.The conclusion characterizes ECA as broadly applicable across deep CNN architectures.

Appendix I: Comparison of Different Methods using ResNet-18 and ResNet-34 on ImageNet

On ImageNet, ECA-Net is compared with ResNet, SENet, and CBAM using ResNet-18 and ResNet-34, improving the original backbones in Top-1 accuracy while using less model complexity than competing attention methods.

  • Experimental comparison: The appendix compares ResNet, SENet, CBAM, and ECA-Net on ImageNet with ResNet-18 and ResNet-34 backbones.ResNet, SENet, and CBAM results are duplicated from [33], while ECA-Net uses [33]'s hyper-parameter settings.
  • Experimental comparison: Over 0.38% Top-1 accuracy improvement is achieved on ResNet-18, while the improvement reaches over 0.9% on ResNet-34 versus the original ResNet backbones.The passage reports these gains directly from Table 7.
  • Experimental comparison: ECA-Net achieves better performance than SENet and CBAM using less model complexity.The supplied passage states this comparison but does not provide the corresponding complexity values here.

Appendix II: Stacking More 1D Convolutions in ECA Module

The appendix evaluates stacking additional 1D convolutions in ECA, finding negligible benefit from one extra convolution and performance degradation with more, so the final module uses one.

  • Appendix II: Stacking More 1D Convolutions in ECA Module: One extra 1D convolution brings only ~0.1% gains while slightly increasing complexity.The passage reports this as a trivial improvement relative to the added complexity.
  • Appendix II: Stacking More 1D Convolutions in ECA Module: More than one additional 1D convolution degrades performance, potentially because gradient backpropagation becomes more difficult.This proposed cause is presented as an explanation for the observed degradation.
  • Appendix II: Stacking More 1D Convolutions in ECA Module: The final ECA module therefore contains only one 1D convolution.The design choice follows the observed trade-off between stacking depth, performance, and complexity.

Appendix III: Visualization of Weights Learned by ECA Modules and SE Blocks

Using ResNet-50 and four sampled ImageNet classes, the appendix visualizes channel weights learned by ECA modules and SE blocks. The weights are similar across classes in earlier layers, while ECA learns more class-specific and discriminative weights than SE.

  • Visualization setup: ResNet-50 channel weights were visualized for ECA modules and SE blocks using four randomly sampled ImageNet classes.The classes were hammerhead shark, ambulance, medicine chest, and butternut squash, with example images shown in Figure 5.
  • Observed weight patterns: In earlier layers from conv 2 1 to conv 3 4, both ECA and SE produce very similar channel-weight distributions across classes.These layers are described as capturing basic elements such as boundaries and corners, which are nearly similar across classes.
  • Observed weight patterns: ECA modules learn more class-specific channel weights than SE blocks.The appendix states that this demonstrates better discriminative ability for weights learned by ECA modules.
Loading 1910.03151v4…