PaperlayerView exact revision on arXiv

Source-linked AI summary

Residual Attention Network for Image Classification

Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, Xiaoou Tang

arXiv:1704.06904v1cs.CV

TL;DR

Image classification lacked attention mechanisms integrated with very deep feedforward convolutional networks. This paper introduces a Residual Attention Network built from stacked Attention Modules and reports state-of-the-art results across CIFAR-10, CIFAR-100, and ImageNet, including a 0.6% top-1 error reduction versus ResNet-200.

  • Problem

    Image classification lacked attention mechanisms integrated with very deep feedforward convolutional networks.

  • Method

    The Residual Attention Network stacks Attention Modules that combine trunk processing with bottom-up top-down masks and attention residual learning for end-to-end training.

  • Results

    Across CIFAR-10, CIFAR-100, and ImageNet, the network achieves state-of-the-art recognition performance, including a 0.6% top-1 error reduction versus ResNet-200.

  • Takeaways & Limitations

    The approach supports scalable, efficient deep attention networks while improving classification performance across benchmark datasets.

  • Takeaways & Limitations

    The noisy-label analysis assumes a clean label ratio r for the whole dataset.

Abstract

from arXiv · show

In this work, we propose "Residual Attention Network", a convolutional neural network using attention mechanism which can incorporate with state-of-art feed forward network architecture in an end-to-end training fashion. Our Residual Attention Network is built by stacking Attention Modules which generate attention-aware features. The attention-aware features from different modules change adaptively as layers going deeper. Inside each Attention Module, bottom-up top-down feedforward structure is used to unfold the feedforward and feedback attention process into a single feedforward process. Importantly, we propose attention residual learning to train very deep Residual Attention Networks which can be easily scaled up to hundreds of layers. Extensive analyses are conducted on CIFAR-10 and CIFAR-100 datasets to verify the effectiveness of every module mentioned above. Our Residual Attention Network achieves state-of-the-art object recognition performance on three benchmark datasets including CIFAR-10 (3.90% error), CIFAR-100 (20.45% error) and ImageNet (4.8% single model and single crop, top-5 error). Note that, our method achieves 0.6% top-1 accuracy improvement with 46% trunk depth and 69% forward FLOPs comparing to ResNet-200. The experiment also demonstrates that our network is robust against noisy labels.

1. Introduction

The paper introduces Residual Attention Network, a very deep feedforward convolutional network that uses mixed attention through stacked Attention Modules to generate adaptively changing attention-aware features. It further proposes attention residual learning and bottom-up top-down feedforward attention to optimize and enhance this architecture.

  • Introduction: Residual Attention Network applies a mixed attention mechanism within a very deep convolutional network for image classification.The authors identify the lack of attention mechanisms applied to feedforward networks for state-of-the-art image classification as motivation.
  • Introduction: Stacked Attention Modules generate attention-aware features that adapt as network depth increases and capture different attention types.The stacked structure enables different Attention Modules to capture different types of attention, such as suppressing background responses or highlighting object regions.
  • Introduction: Attention residual learning prevents performance degradation from directly stacking Attention Modules and enables optimization of networks with hundreds of layers.The mechanism is introduced specifically to address the performance drop caused by direct stacking.
  • Introduction: Bottom-up top-down feedforward attention is incorporated into Attention Modules to add soft weights.The structure unfolds attention processing within the feedforward architecture and has also been used in human pose estimation and image segmentation.

2. Related Work

Prior work introduced attention into deep networks through top-down, recurrent, sequential, region-proposal, control-gate, and soft-attention mechanisms. Residual Attention Network builds on residual learning and bottom-up/top-down structures while integrating soft attention into feedforward convolutional networks.

  • Attention mechanisms: Human perception motivates attention mechanisms that use top information to guide bottom-up feedforward processing, with applications in DBMs, RNNs, and LSTMs.DBMs provide top-down attention through reconstruction during training, while RNNs and LSTMs have also widely used attention mechanisms.
  • Residual learning: Residual learning enables very deep feedforward networks, and related work combines residual learning with attention mechanisms.The cited approaches use residual learning with attention, while this work splits a single image source into two information sources and combines them.
  • Attention in image classification: Image-classification attention methods include sequential decision processes, region proposals, and control gates.Sequential approaches use RNNs or LSTMs for end-to-end, goal-driven attention; region proposals precede feedforward classification; control gates influence feedforward processing during training.
  • Deep feedforward networks: Recent image-classification research emphasizes very deep feedforward convolutional networks, including VGG, Inception, residual learning, and stochastic depth.These networks mimic the bottom-up paths of the human cortex and aim to improve the discriminative ability of deep convolutional networks.
  • Soft attention: Residual Attention Network incorporates end-to-end trainable soft attention into a rapidly developing feedforward network structure.Its soft-attention design is inspired by segmentation and human pose estimation, which use cascaded bottom-up and top-down structures with fine-grained feature maps.

3. Residual Attention Network

Residual Attention Network stacks Attention Modules that pair trunk feature processing with specialized mask branches, using bottom-up top-down attention in an end-to-end trainable architecture. Attention residual learning preserves useful features while enabling deep networks, and adaptive mixed attention performs best in the reported experiment.

  • Attention Modules: Each Attention Module combines a trunk branch with a same-size mask branch that softly weights trunk features.The design can use pre-activation Residual Units, ResNeXt, or Inception as its basic unit.
  • Robustness: Mask branches can filter incorrect noisy-label gradients from updating trunk parameters, making Attention Modules robust to noisy labels.The mask functions as both a forward feature selector and a backward gradient update filter.
  • Attention Residual Learning: Attention residual learning prevents repeated soft masking from degrading deep features and preserves the trunk branch’s useful identity-mapping properties.Mask branches act as feature selectors that enhance good features and suppress noise, while allowing features to bypass the mask branch.
  • Attention Residual Learning: 452 layers were reached by the Residual Attention Network, whose performance surpassed ResNet-1001.Stacked modules progressively refine feature maps, and increasing network depth improved performance consistently.
  • Mask Branch: The mask branch unfolds feedforward and feedback attention through bottom-up pooling, symmetric top-down upsampling, sigmoid outputs, and skip connections.Pooling rapidly increases receptive field, while top-down processing combines global information with features at each position.
  • Attention Types: Mixed attention achieved the best performance because attention changed adaptively with features without additional constraints.The compared attention types were mixed, channel, and spatial attention.

4. Experiments

Experiments on CIFAR-10, CIFAR-100, and ImageNet evaluate the Residual Attention Network’s components, robustness, and performance across network backbones. The results show benefits from attention residual learning and encoder-decoder masks, strong noise resistance, and improved accuracy-efficiency tradeoffs over established baselines.

  • Attention Residual Learning: Attention residual learning consistently outperforms naive attention learning, with performance improving as the number of Attention Modules increases.Naive attention learning instead suffers obvious degradation as more Attention Modules are added.
  • Comparison of different mask structures: The Attention-Encoder-Decoder-56 achieves 5.52% test error versus 6.48% for Attention-Local-Conv-56, a 0.94% margin.Both mask structures use the same number of FLOPs, supporting the benefit of multi-scale information.
  • Noisy Label Robustness: Attention-92 has significantly lower test error than ResNet-164 at the same noise levels and degrades more slowly as label noise increases.The experiment uses CIFAR-10 with the clean label ratio represented by r.
  • Comparisons with state-of-the-art methods: Attention-92 achieves 4.99% CIFAR-10 test error and 21.71% CIFAR-100 test error, compared with 5.46% and 24.33% for ResNet-164 under similar parameter sizes.Attention-452 outperforms all baseline methods on both datasets, while Attention-236 outperforms ResNet-1001 using half the parameters.
  • ImageNet experiments: Attention-56 reduces ResNet-152’s top-1 error by 0.4% and top-5 error by 0.26%, using 52% of its parameters and 56% of its FLOPs.The comparison uses single-crop evaluation on the ImageNet validation set.
  • Comparisons with state-of-the-art methods: Attention-92 reduces ResNet-200’s top-1 error by 0.6%, while ResNet-200 contains 32% more parameters; the method also transfers to ResNeXt and Inception backbones.AttentionNeXt-56 matches ResNeXt-101 with fewer parameters and FLOPs, while AttentionInception-56 reduces Inception-ResNet-v1’s top-1 and top-5 errors by 0.94% and 0.21%.

5. Discussion

The discussion highlights a Residual Attention Network that stacks Attention Modules to capture mixed attention while providing an extensible convolutional neural network architecture.

  • The proposed Residual Attention Network stacks multiple Attention Modules.
  • Different Attention Modules capture different types of attention to guide feature learning.
  • Free form mixed attention performs better than constrained attention, including single attention, in activation-function experiments.
  • The network is presented as an extensible convolutional neural network.
Loading 1704.06904v1…