PaperlayerView exact revision on arXiv

Source-linked AI summary

Squeeze-and-Excitation Networks

Jie Hu, Li Shen, Samuel Albanie, Gang Sun, Enhua Wu

arXiv:1709.01507v4cs.CV

TL;DR

CNNs primarily model channel dependencies implicitly and locally, limiting explicit control over feature importance. This paper introduces Squeeze-and-Excitation blocks for global channel-wise recalibration, and SENets achieve state-of-the-art performance, including first place in ILSVRC 2017 with 2.251% top-5 error.

  • Problem

    Convolutions model channel relationships implicitly and locally, motivating explicit modelling of channel interdependencies to improve feature representations.

  • Method

    The SE block aggregates global spatial information into channel descriptors and uses self-gating to selectively emphasize informative features and suppress less useful ones.

  • Results

    SENets achieve state-of-the-art performance across multiple datasets and tasks, including first place in ILSVRC 2017 with 2.251% top-5 error.

  • Takeaways & Limitations

    SE blocks provide a lightweight way to enhance existing state-of-the-art CNN architectures through dynamic channel-wise feature recalibration.

Abstract

from arXiv · show

The central building block of convolutional neural networks (CNNs) is the convolution operator, which enables networks to construct informative features by fusing both spatial and channel-wise information within local receptive fields at each layer. A broad range of prior research has investigated the spatial component of this relationship, seeking to strengthen the representational power of a CNN by enhancing the quality of spatial encodings throughout its feature hierarchy. In this work, we focus instead on the channel relationship and propose a novel architectural unit, which we term the "Squeeze-and-Excitation" (SE) block, that adaptively recalibrates channel-wise feature responses by explicitly modelling interdependencies between channels. We show that these blocks can be stacked together to form SENet architectures that generalise extremely effectively across different datasets. We further demonstrate that SE blocks bring significant improvements in performance for existing state-of-the-art CNNs at slight additional computational cost. Squeeze-and-Excitation Networks formed the foundation of our ILSVRC 2017 classification submission which won first place and reduced the top-5 error to 2.251%, surpassing the winning entry of 2016 by a relative improvement of ~25%. Models and code are available at https://github.com/hujie-frank/SENet.

1 INTRODUCTION

The paper shifts attention from spatial dependencies to channel relationships in CNNs, introducing Squeeze-and-Excitation blocks that explicitly model channel interdependencies for feature recalibration. SE blocks can be stacked into SENets or integrated into existing architectures, improving performance with slight added complexity and strong ImageNet results.

  • Channel relationship: SE blocks explicitly model interdependencies between convolutional feature channels to improve network representations through feature recalibration.The proposed mechanism enables the network to learn to recalibrate feature responses.
  • SE block mechanism: The squeeze operation aggregates feature maps across H × W to produce a channel descriptor representing the global distribution of features.This descriptor forms the input to the SE block’s recalibration process.
  • Architecture and depth: SENets are formed by stacking SE blocks, which can also replace original blocks at different depths in existing architectures.Their role varies with depth: earlier blocks strengthen shared low-level features, while later blocks become increasingly class-specific.
  • Architectural integration: SE blocks are simple, computationally lightweight, and impose only a slight increase in model complexity while enhancing existing state-of-the-art architectures.This makes them suitable as drop-in replacements rather than requiring entirely new CNN designs.
  • Evaluation: 2.251% top-5 error was achieved by the best SENet ensemble on the ILSVRC 2017 classification test set, representing roughly a 25% relative improvement.The authors also report benefits beyond ImageNet, indicating that the approach is not restricted to a specific dataset or task.

2 RELATED WORK

Related work improves CNNs through deeper architectures, redesigned computational elements, automated architecture search, and attention or gating mechanisms. SE blocks also serve as effective atomic building blocks for architecture-search algorithms.

  • Deeper architectures: VGGNets and Inception models showed that increasing network depth can significantly improve the quality of learned representations.Batch Normalization stabilised learning in deep networks and produced smoother optimisation, while later connection reformulations sought further improvements.
  • Computational elements: Grouped convolutions increase the cardinality of learned transformations, while multi-branch convolutions enable more flexible operator compositions.Multi-branch convolutions can be viewed as a natural extension of grouping operators.
  • Algorithmic Architecture Search: Architecture-search research replaces manual design with methods including evolutionary topology search, hyperparameter optimisation, path-based topology selection, direct architecture prediction, and reinforcement learning.Evolutionary search is often computationally demanding but has achieved notable success; reinforcement learning has produced particularly strong results.
  • Algorithmic Architecture Search: SE blocks can function as atomic building blocks for architecture-search algorithms and were demonstrated to be highly effective in that role.This capability was reported in concurrent work.
  • Attention and gating mechanisms: Attention biases computational resources toward the most informative signal components and has proved useful in sequence learning, image localisation, captioning, and lip reading.The related work presents attention as a general computational-resource allocation mechanism across these tasks.

3 SQUEEZE-AND-EXCITATION BLOCKS

SE blocks address the implicit, local, and spatially entangled modelling of channel relationships in convolutions by globally summarising channels and adaptively recalibrating their responses. Their flexible transformation design supports integration into convolutional, Inception, and residual architectures.

  • Motivation: Convolutional channel relationships are implicit, local, and entangled with the spatial correlations captured by filters.The block is motivated by the limitation that local receptive fields cannot directly exploit broader contextual information.
  • Squeeze: Global average pooling squeezes each feature map into a channel descriptor containing spatially aggregated channel-wise statistics.The descriptor z ∈ R^C is obtained by shrinking U across its H × W spatial dimensions.
  • Excitation: The excitation operator uses a bottleneck of two fully connected layers, ReLU, and reduction ratio r to learn nonlinear, non-mutually-exclusive channel dependencies.This design permits multiple channels to be emphasised while limiting model complexity and aiding generalisation.
  • Excitation: The excitation output maps the input-specific descriptor to channel weights, introducing channel dynamics beyond convolutional local receptive fields.The resulting operation can be regarded as a channel self-attention function conditioned on the input.
  • Architectural integration: SE blocks can be inserted after convolutional nonlinearities and applied to entire Inception modules or non-identity branches of residual modules.These integrations produce SE-VGGNet, SE-Inception, and SE-ResNet-style architectures, with alternative inclusion strategies assessed through ablations.

4 MODEL AND COMPUTATIONAL COMPLEXITY

SE-ResNet-50 adds only slight computational and runtime overhead to ResNet-50 while improving accuracy. Its extra parameters mainly arise from excitation layers, and removing the costly final stage reduces the parameter increase with minimal performance loss.

  • Computational cost: ∼3.87 GFLOPs for SE-ResNet-50 represents a 0.26% relative increase over ResNet-50 at reduction ratio r=16.ResNet-50 requires ∼3.86 GFLOPs for a 224 × 224 input image.
  • Accuracy trade-off: SE-ResNet-50 surpasses ResNet-50 in accuracy and approaches the accuracy of the deeper ResNet-101 network despite its slight additional computational burden.This comparison accompanies the 0.26% relative GFLOP increase at reduction ratio r=16.
  • Runtime overhead: 209 ms versus 190 ms for ResNet-50 measures the single-pass forward-and-backward runtime with a minibatch of 256 images on 8 NVIDIA Titan X GPUs.The authors describe this as a reasonable overhead and note that library optimization could reduce it.
  • Parameter cost: ∼2.5 million additional parameters beyond ∼25 million for ResNet-50 correspond to a ∼10% increase in SE-ResNet-50.The added parameters come solely from the two fully connected layers in the excitation mechanism.
  • Parameter cost: <0.1% top-5 error on ImageNet is the performance cost of removing the comparatively costly final SE stage.This removal reduces the relative parameter increase to ∼4%.

5 EXPERIMENTS

Experiments across ImageNet, CIFAR, Places365, mobile architectures, and object detection show that SE blocks consistently improve CNN performance with little added computational cost. The strongest reported results include gains across network depths and architectures, scene classification, and the ILSVRC 2017 submission.

  • Network depth: SE-ResNet-50 achieves 6.62% single-crop top-5 validation error, improving over ResNet-50’s 7.48% by 0.86%.The result approaches ResNet-101’s 6.52% top-5 error with an extremely small increase in computational complexity.
  • Integration with modern architectures: SE blocks improve performance when integrated with ResNeXt and Inception-ResNet-v2, including SE-ResNeXt-50’s 5.49% top-5 error versus 5.90% for ResNeXt-50.SE-ResNeXt-50 also outperforms deeper ResNeXt-101 at 5.57% top-5 error, while the SE Inception-ResNet counterpart reaches 4.79% versus 5.21% for its baseline.
  • Optimization and mobile setting: SE blocks yield steady optimization improvements across baseline architectures and also improve non-residual and mobile CNNs at minimal computational cost.The experiments cover VGG-16, BN-Inception, MobileNet, and ShuffleNet, with gains sustained throughout training.
  • Additional datasets: SE-ResNet-152 reaches 11.01% top-5 validation error on Places365, improving over ResNet-152’s 11.61% and Places-365-CNN’s 11.48%.This supports generalization to scene classification, which involves more complex data associations and greater appearance variation.
  • ILSVRC submission: The ILSVRC submission achieved first place with a 2.251% top-5 test error using a small ensemble of SENets and multi-scale, multi-crop fusion.SENet-154 achieved 18.68% top-1 and 4.47% top-5 error with 224 × 224 centre-crop evaluation, reported as the strongest result.

6 ABLATION STUDY

Ablations on ImageNet with ResNet-50 show that SE performance is robust to several design choices, while the excitation non-linearity and placement impose important constraints. Average pooling, careful excitation construction, stage-wise integration, and pre-aggregation placement support effective SE blocks, with an internal 3×3 variant reducing parameters at comparable accuracy.

  • Reduction ratio: SE-ResNet-50 performance remains robust across reduction ratios, although increasing complexity does not improve performance monotonically.The reduction ratio controls SE-block capacity and computational cost.
  • Squeeze operator: Average pooling slightly outperforms max pooling as the squeeze operator, although SE performance is fairly robust to this choice.The study did not consider more sophisticated pooling alternatives.
  • Excitation non-linearity: Replacing sigmoid with tanh slightly worsens performance, while ReLU is dramatically worse and drops SE-ResNet-50 below the ResNet-50 baseline.These results indicate that careful construction of the excitation operator is important.
  • Integration stage: SE blocks improve performance when added at stages 2, 3, or 4, and their gains are complementary when combined across stages.The experiments integrate SE blocks into one intermediate stage at a time before evaluating their combination.
  • Block location: SE improvements are fairly robust to block location when applied before branch aggregation, whereas placing the unit after summation causes a performance drop.The comparison includes SE-PRE, SE-POST, and SE-Identity variants.
  • Block location: The SE 3×3 variant achieves comparable classification accuracy with fewer parameters by placing the block after the residual branch’s 3 × 3 convolution.The 3 × 3 layer has fewer channels, reducing the parameters introduced by its SE block.

7 ROLE OF SE BLOCKS

The ablation and activation analyses show that global information from the squeeze operation is important for performance, while excitation responses become increasingly class- and instance-specific with depth. In the final stages, excitations can saturate toward one, making the SE block approximate an identity operator.

  • Squeeze operation: The NoSqueeze ablation removes global average pooling and replaces the excitation fully connected layers with equal-channel 1 × 1 convolutions, preserving parameter count while eliminating direct global embeddings.This variant tests whether the global embedding produced by squeezing contributes to performance.
  • Squeeze operation: Global information has a significant influence on model performance, underscoring the importance of the squeeze operation.The comparison includes a standard ResNet-50 and evaluates both accuracy and computational complexity.
  • Excitation behavior: Excitation distributions are similar across classes in earlier layers, suggesting that early-stage channel importance is shared across different classes.The paper illustrates this behavior with an early module such as SE 2 3.
  • Excitation behavior: At greater depth, channel values become more class-specific, reflecting different class preferences for the discriminative value of features.This progression is illustrated by later modules such as SE 4 6 and SE 5 1.
  • Excitation behavior: SE responses vary across both classes and instances within a class, supporting increasingly class-specific model needs at different architectural depths.The final-stage SE 5 2 can approach saturation near one, reducing the block to the identity operator when all activations equal one.

8 CONCLUSION

The paper introduces SE blocks for dynamic channel-wise feature recalibration, improving network representational power. Experiments show that SENets achieve state-of-the-art performance across multiple datasets and tasks while offering insight into channel-wise feature dependencies.

  • Contribution: SE blocks improve network representational power through dynamic channel-wise feature recalibration.The proposed architectural unit enables networks to recalibrate channel-wise features dynamically.
  • Experimental findings: SENets achieve state-of-the-art performance across multiple datasets and tasks.The paper reports effectiveness across a wide range of experiments.
  • Interpretation: SE blocks provide insight into why previous architectures inadequately model channel-wise feature dependencies.The authors connect their findings to limitations in earlier architectures’ treatment of channel-wise dependencies.

APPENDIX: DETAILS OF SENET-154

SENet-154 incorporates SE blocks into a modified 64×4d ResNeXt-152 derived from ResNeXt-101 using ResNet-152’s block-stacking strategy. Its design reduces computation by halving bottleneck channels and replaces the initial 7 × 7 convolution with three consecutive 3 × 3 convolutions.

  • Architecture: SENet-154 integrates SE blocks into a modified 64×4d ResNeXt-152 architecture.The model extends ResNeXt-101 by adopting ResNet-152’s block-stacking strategy.
  • Design modifications: The first 1 × 1 convolutional channels in each bottleneck block were halved to reduce computational cost with minimal performance decrease.
  • Design modifications: The initial 7 × 7 convolutional layer was replaced with three consecutive 3 × 3 convolutional layers.
Loading 1709.01507v4…