Source-linked AI summary
Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
Sergey Zagoruyko, Nikos Komodakis
TL;DR
The paper asks whether attention information can improve convolutional neural networks and proposes transferring teacher-network attention maps to smaller student networks. Across varied datasets and architectures, the approach produces significant improvements, with activation-based transfer outperforming full-activation transfer and combining with knowledge distillation.
Problem
The paper investigates whether attention information from a teacher network can improve the performance of convolutional neural networks.
Method
The paper defines activation-based and gradient-based spatial attention maps and transfers them from powerful teacher networks to smaller student networks.
Results
The approach yields significant improvements across datasets and residual and non-residual architectures, while activation-based transfer outperforms full-activation transfer and combines with knowledge distillation.
Takeaways & Limitations
Attention provides a transferable knowledge signal that can improve student convolutional networks across varied recognition settings.
Takeaways & Limitations
The experiments use deterministic single-run settings, and the behavior of batch normalization during gradient-based attention transfer remains unclear.
Abstract
from arXiv · showhide
Attention plays a critical role in human visual experience. Furthermore, it has recently been demonstrated that attention can also play an important role in the context of applying artificial neural networks to a variety of tasks from fields such as computer vision and NLP. In this work we show that, by properly defining attention for convolutional neural networks, we can actually use this type of information in order to significantly improve the performance of a student CNN network by forcing it to mimic the attention maps of a powerful teacher network. To that end, we propose several novel methods of transferring attention, showing consistent improvement across a variety of datasets and convolutional neural network architectures. Code and models for our experiments are available at https://github.com/szagoruyko/attention-transfer
1 INTRODUCTION
The paper defines CNN attention as spatial maps indicating which input regions influence decisions, then transfers these maps from powerful teacher networks to smaller students to improve performance across architectures and tasks.
- 1 INTRODUCTION: CNN attention is defined as spatial maps encoding the input areas a network focuses on when making output decisions.These maps can be defined at different layers to capture low-, mid-, and high-level information.
- 1 INTRODUCTION: The proposed approach transfers attention from a powerful teacher network to a smaller student network to improve the student’s performance.Attention transfer is presented as a knowledge-transfer mechanism between networks.
- 1 INTRODUCTION: The work proposes both activation-based and gradient-based spatial attention maps for attention transfer.These constitute two novel ways of representing and transferring attention.
- 1 INTRODUCTION: The approach yields significant improvements across varied datasets and deep architectures, including residual and non-residual networks.The experiments evaluate attention transfer across multiple datasets and convolutional neural network architectures.
- 1 INTRODUCTION: Activation-based attention transfer improves more than full-activation transfer and can be combined with knowledge distillation.This comparison and combination are stated among the paper’s experimental contributions.
2 RELATED WORK
Prior work developed attention mechanisms for neural networks, methods for visualizing CNN attention, and knowledge distillation from teacher to student networks. This work positions its attention transfer approach relative to these lines of research, including FitNets and attention-map visualization methods.
- Attention mechanisms: Attention-based methods originated from human attention theories and expanded from restricted Boltzmann machines to neural machine translation and computer-vision tasks.Examples include recurrent neural machine translation, image captioning, and visual question answering.
- Attention visualization: CNN attention visualization includes gradient-based Jacobians, weight-sharing deconvnets, class activation maps, and Grad-CAM.These methods differ in whether attention is class-discriminative and whether maps incorporate image-level details.
- Knowledge distillation: Knowledge distillation transfers knowledge from a powerful teacher network to improve training of a student network.The cited literature includes Hinton et al. (2015) and Bucila et al. (2006), while some work showed shallow networks can approximate deeper ones without accuracy loss.
- Relation to FitNets: Unlike FitNets, this work targets less deep student networks and transfers attention maps resembling gradient- and activation-based maps without introducing new weights.The authors motivate revisiting knowledge transfer because thin deep networks are less parallelizable than wider ones.
3 ATTENTION TRANSFER
The section defines attention maps from CNN activations and transfers normalized teacher attention to student networks through layer-wise losses. It also describes gradient-based input attention and its computational requirements, while noting that the experiments restrict gradients to the input layer.
- Activation-based attention: Activation-based attention maps aggregate absolute hidden activations across channels from a layer tensor A ∈ R^C×H×W.The mapping produces a flattened spatial map, using channel-wise statistics such as sums or maxima of |A_i|^p for p > 1.
- Teacher-student transfer: Attention transfer trains students to match teacher attention maps while also making correct predictions, with losses applied across selected corresponding layers.For residual networks, transfer can follow every residual block at the same depth or group outputs when depths differ; spatial maps may be interpolated to match resolution.
- Teacher-student transfer: L2-normalized attention maps are used in the transfer loss, and the paper emphasizes that normalization is important for successful student training.The experiments use p = 2, although l1 normalization is also described as possible.
- Teacher-student transfer: Attention transfer can be combined with knowledge distillation by adding cross-entropy between softened teacher and student label distributions with little additional computational cost.Teacher attention maps can be computed during the forward propagation already needed for distillation.
- Gradient-based attention: Gradient-based attention treats input gradients as sensitivity maps and matches student and teacher gradients with a distance such as l2.The resulting update requires a second propagation involving second-order mixed partial derivatives, and the paper considers gradients only with respect to the input layer.
4 EXPERIMENTAL SECTION
Experiments across CIFAR, fine-grained and scene datasets, and ImageNet show that attention transfer consistently improves student CNN performance across architectures and settings. The gains extend to activation- and gradient-based methods, with attention maps generally more useful for transfer than full activations and strong ImageNet improvements despite untuned hyperparameters.
- CIFAR activation-based transfer: Attention transfer significantly improves CIFAR-10 results across same- and different-depth WRN teacher/student pairs, with larger gains when combined with knowledge distillation.The experiments used WRN-16-2/WRN-16-1, WRN-40-1/WRN-16-1, and WRN-40-2/WRN-16-2 combinations.
- CIFAR activation-based transfer: 7.93 error with three combined transfer losses outperforms 8.11, 7.96, and 7.97 with one loss in groups 1, 2, and 3 respectively.Each individual loss provides additional attention-transfer information in the WRN-16-2/WRN-16-1 setting.
- CIFAR activation-based transfer: Sum-based attention mappings perform similarly to one another and better than max-based mappings, motivating use of the sum of squared attention mapping function.The comparison used WRN-16-2 as teacher and WRN-16-1 as student.
- CIFAR activation-based transfer: Attention maps produce better final accuracy improvements than full-activation transfer, although both methods greatly speed convergence.Full-activation transfer used FitNets-style l2 losses with 1 × 1 convolutions to match tensor shapes, while attention transfer used attention maps.
- CIFAR gradient-based transfer: Gradient-based attention transfer improves performance under constrained thin-NIN experiments, while gradient l2 minimization already works well and symmetry norm is among the best-performing attention norms.These single-run experiments used horizontal-flip augmentation, no random crops or batch normalization, deterministic algorithms, fixed seeds, and no weight decay.
- Larger-dataset transfer: 1.1% top-1 and 0.8% top-5 better validation accuracy are achieved by ResNet-18 with attention transfer from ResNet-34 on ImageNet, despite untuned hyperparameters.The method used squared-sum attention and only two losses in the two last residual-block groups; on CUB and Scenes, attention transfer also improved finetuning, closing the ResNet-18/ResNet-34 accuracy gap.
5 CONCLUSIONS
The paper presents several methods for transferring attention between networks, supported by experiments across multiple image-recognition datasets, and identifies spatially sensitive tasks for future study.
- 5 CONCLUSIONS: The authors present several methods for transferring attention from one network to another, with experiments across several image-recognition datasets.They propose exploring attention transfer in spatially important settings such as object detection and weakly supervised localization.
- 5 CONCLUSIONS: The findings are intended to advance knowledge distillation and understanding of convolutional neural networks.
A APPENDIX · A.1 FIGURES AND TABLES
The appendix presents visual and tabular evidence for attention transfer on convolutional networks. Figure 6 compares attention maps across pretrained, scene-trained, and attention-transfer models, while Table 5 reports ImageNet validation error under a specified transfer-loss schedule.
- A.1 FIGURES AND TABLES: Figure 6 compares top activation attention maps for four Scenes-network configurations.The configurations are pretrained ResNet-18, Scenes-trained ResNet-18, attention-transfer ResNet-18 with ResNet-34 as teacher, and Scenes-trained ResNet-34.
- A.1 FIGURES AND TABLES: Attention maps look more similar after transfer in the Scenes-network examples.The images were taken from the test set.
- A.1 FIGURES AND TABLES: The comparison includes original pretrained ResNet-18 using ImageNet training.This model is labeled ResNet-18-ImageNet.
- A.1 FIGURES AND TABLES: The comparison includes ResNet-18 trained on Scenes without attention transfer.This model is labeled ResNet-18-scenes.
- A.1 FIGURES AND TABLES: The attention-transfer model is a ResNet-18 trained on Scenes with ResNet-34 as teacher.This model is labeled ResNet-18-scenes-AT.
- A.1 FIGURES AND TABLES: Table 5 reports attention transfer validation error using a single crop on ImageNet.Transfer losses are added on epoch 60/100.
A.2 IMPLEMENTATION DETAILS
Experiments used the Torch framework, with ImageNet runs based on fb.resnet.torch and accelerated through data parallelism on two Titan X cards for both teacher and student networks.
- Implementation Details: Experiments were conducted in the Torch machine learning framework.The passage also notes that double propagation is implementable in modern automatically differentiable frameworks such as Torch, Theano, and TensorFlow.
- Implementation Details: ImageNet experiments used fb.resnet.torch and two Titan X cards with data parallelism for both teacher and student training.The setup was used to speed up training.
- Implementation Details: Code and experimental models are available at the project’s GitHub repository.Repository: https://github.com/szagoruyko/attention-transfer