Learn To Pay Attention

Saumya Jetley; Nicholas A. Lord; Namhoon Lee; Philip H. S. Torr

Source-linked AI summary

Learn To Pay Attention

Saumya Jetley, Nicholas A. Lord, Namhoon Lee, Philip H. S. Torr

arXiv:1804.02391v2Apr 6, 2018cs.CVcs.AI

TL;DR

CNNs’ visual reasoning is difficult to interpret, motivating a trainable attention module for image classification. The method uses a global representation to weight local features and restricts classification to their convex combination, yielding improved classification generalisation and useful attention maps for segmentation and robustness evaluation.

Problem
CNN visual reasoning remains largely inscrutable, limiting understanding of how classification decisions succeed or fail.
Method
The method learns multi-scale attention end to end by comparing local features with a global image representation and restricting classification to their convex weighted combination.
Results
The attention models improve classification and generalise across 6 unseen benchmark datasets, while their maps support weakly supervised segmentation and show promise against certain adversarial attacks.
Takeaways & Limitations
Learned attention maps highlight regions of interest while suppressing background clutter, supporting interpretation and weakly supervised segmentation from image-level classification models.

Abstract

from arXiv · show

We propose an end-to-end-trainable attention module for convolutional neural network (CNN) architectures built for image classification. The module takes as input the 2D feature vector maps which form the intermediate representations of the input image at different stages in the CNN pipeline, and outputs a 2D matrix of scores for each map. Standard CNN architectures are modified through the incorporation of this module, and trained under the constraint that a convex combination of the intermediate 2D feature vectors, as parameterised by the score matrices, must \textit{alone} be used for classification. Incentivised to amplify the relevant and suppress the irrelevant or misleading, the scores thus assume the role of attention values. Our experimental observations provide clear evidence to this effect: the learned attention maps neatly highlight the regions of interest while suppressing background clutter. Consequently, the proposed function is able to bootstrap standard CNN architectures for the task of image classification, demonstrating superior generalisation over 6 unseen benchmark datasets. When binarised, our attention maps outperform other CNN-based attention maps, traditional saliency maps, and top object proposals for weakly supervised segmentation as demonstrated on the Object Discovery dataset. We also demonstrate improved robustness against the fast gradient sign method of adversarial attack.

1 INTRODUCTION

The paper addresses CNNs’ inscrutable visual reasoning by introducing trainable, multi-scale attention for classification. The method constrains classification to attention-weighted local features and improves recognition, generalisation, and interpretability.

CNN visual reasoning remains largely inscrutable, limiting understanding of successes and failures.
Prior attention maps helped locate objects, identify segmentation masks, and expose discriminative visual properties, but were added post hoc to trained networks.
The proposed method estimates multi-scale attention end to end by repurposing the global image representation as a query for classification.
Classification uses only a weighted combination of local features, forcing the network to learn task-relevant attention patterns.
7% accuracy gain on CIFAR-100 is reported after adding attention to VGG, alongside superior generalisation on 6 domain-shifted benchmark datasets.

2 RELATED WORK

Related work divides CNN attention into post-hoc analysis and trainable mechanisms. Trainable approaches include hard stochastic attention and soft deterministic attention, while domain-shift methods use attention to focus on objects amid changing visual conditions.

CNN attention has been implemented through post-hoc network analysis or trainable attention mechanisms.
Post-hoc methods estimate spatial support from trained networks, whereas trainable mechanisms optimise attention-unit parameters with network weights.
Hard attention makes stochastic region-selection decisions and uses REINFORCE, making optimisation more difficult than soft attention.
Attention-based domain-shift methods target changes involving background content, occlusion, and object pose by selectively focusing on objects of interest.

3 APPROACH

The approach learns attention by comparing local intermediate features with a global image descriptor, then uses the resulting normalized weights to construct the classifier’s global representation. Multi-layer attention and compatibility functions are trained end to end.

3 APPROACH: The method seeks parsimonious spatial support by amplifying salient regions and suppressing irrelevant or confusing information.
3. APPROACH: Attention introduced at different VGG layers focuses progressively on surroundings, object parts, and the central object.
3.1 DESIGN AND TRAINING OF ATTENTION SUBMODULE: At each selected convolutional layer, local feature vectors are compared with a global feature vector to produce compatibility scores.
3.1 DESIGN AND TRAINING OF ATTENTION SUBMODULE: Softmax normalization turns compatibility scores into attention weights that form a convex, element-wise weighted average of local features.
3.1 DESIGN AND TRAINING OF ATTENTION SUBMODULE: The attention-weighted vectors replace the ordinary global descriptor, and classification is restricted to use those vectors alone.
3.1 DESIGN AND TRAINING OF ATTENTION SUBMODULE: Multiple layers can be combined by concatenating their global vectors or by averaging predictions from separate classifiers, with parameters learned end to end under cross-entropy.
3.2 CHOICE OF COMPATIBILITY FUNCTION C: The compatibility function can use an alignment-style design or a simplified additive mapping whose learned weights represent dataset-level objectness.

4 EXPERIMENTAL SETUP

The experiments apply attention to VGGNet and ResNet using multi-level local descriptors and compare alternative matching and classification configurations across several evaluation tasks.

VGGNet uses local feature maps from layers 7, 10, and 13, while ResNet uses outputs from computational blocks 2, 3, and 4.
The global vector g is mapped to a given dimensionality and shared by local features from different layers at that dimensionality.
The experiments compare dot-product and parametrised compatibility for matching global and local descriptors.
The classification stage compares concatenated descriptors with independently predicted probabilities averaged across levels.
Attention is evaluated on image classification, fine-grained object recognition, adversarial robustness, and attention-map segmentation quality.

5 RESULTS AND DISCUSSION

The attention mechanism improves classification, fine-grained recognition, cross-domain generalisation, segmentation, and low-noise adversarial robustness while producing interpretable, object-focused maps.

Image classification and fine-grained recognition: 2.5% and 7.4% improvements over baseline VGG are reported for CIFAR-10 and CIFAR-100 classification, respectively.For fine-grained recognition, improvements are 7.8% for CUB and 0.5% for SVHN.
Image classification and fine-grained recognition: Attention maps focus on relevant objects and parts, with different layers specialising in regions such as eyes, beaks, plumage, and feet.The maps also focus sharply on CIFAR-10 category objects in high-resolution ImageNet images.
Image classification and fine-grained recognition: Changing the global feature query can affect the predicted attention pattern, providing a form of post-hoc attention control.The global feature vector is used as the query in the attention calculations.
Robustness to adversarial attack: 5% lower fooling rate is observed for attention-aware VGG than baseline VGG at an L∞ noise norm of 1.As perturbation increases, the gap decreases and the two networks’ fooling rates saturate.
Weakly supervised semantic segmentation: 11% and 3% are the minimum margins by which binarised attention maps outperform existing attention methods and top object proposals across all three segmentation categories.They outperform saliency methods for cars but perform less well for airplanes and horses.

6 CONCLUSION

The paper proposes trainable attention landscapes for classification that focus on relevant image regions and support gains across classification, segmentation, and adversarial-robustness evaluations.

The trainable module generates probabilistic attention landscapes indicating where and in what proportion the network attends during image classification.
Applying attention at multiple network levels improves classification of seen and unseen categories by focusing on the object of interest.
The attention landscapes facilitate weakly supervised segmentation of the predominant object and can be refined using conditional random fields.
The proposed attention scheme shows promise for learning robustness to certain kinds of adversarial attacks.

A.1 DATASETS

The experiments cover classification, cross-domain classification, segmentation, and adversarial robustness across datasets ranging from simple digits to cluttered natural scenes.

The study evaluates attention models on CIFAR-10, CIFAR-100, SVHN, and CUB-200-2011 for image classification.
The attention-incorporating VGG model trained on CUB-200-2011 is used to investigate robustness to adversarial attacks.
Cross-domain classification is tested on 6 standard benchmarks, including STL, Caltech-256, and Action-40.
CUB images are cropped using ground-truth bounding boxes and resized, while cross-domain inputs are downsampled to reduce memory overhead.
The dataset collection spans objects in plain backgrounds and cluttered indoor or outdoor scenes, including digits, objects, humans, and complex activities.

A.2 NETWORK ARCHITECTURES

The architectures incorporate attention through progressive and multi-level mechanisms in VGG and ResNet, using compatibility scores to weight local features for classification.

Progressive attention networks: Progressive attention uses sigmoid-normalized lower-level compatibility scores to filter local features before they enter the next network layer.
Progressive attention networks: At the final progressive-attention level, softmax-normalized scores weight local features that are trained for image classification.
Progressive attention networks: The progressive attention approach and the proposed mechanism are implemented and evaluated with the VGG architecture.
Attention in ResNet: For ResNet, spatial average pooling after the final level is replaced by convolutional and max-pooling operations to produce the global feature vector g.
Attention in ResNet: RN-att applies attention at the last level, whereas RN-att2 applies it at the last two levels using the concat-pc framework.

A.3 TRAINING ROUTINES

VGG and ResNet models are trained with stochastic gradient descent using scheduled learning-rate decay, dataset-specific initialization choices, and long training schedules until convergence.

VGG training: VGG models use SGD with batch size 128, weight decay 5 × 10^-4, momentum 0.9, and 300 training epochs.The learning rate is halved every 25 epochs; initial rates are 1 for CIFAR and 0.1 for SVHN.
VGG training: For CUB, the VGG model is initialized with weights learned for CIFAR before training on the limited CUB data.
ResNet training: ResNet models use SGD with batch size 64, initial learning rate 0.1, weight decay 5 × 10^-4, and momentum 0.9.
ResNet training: ResNet learning rates are multiplied by 0.2 after 60, 120, and 160 epochs, with training continuing for 200 epochs until convergence.
Implementation: All models are implemented in Torch and trained on an NVIDIA Titan-X GPU, requiring around one to two days depending on the model and datasets.

A.4 TASK-SPECIFIC PROCESSING

The analysis tests how global feature vectors influence attention patterns across architectures and evaluates the resulting representations without target-domain fine-tuning. Dot-product attention responds to query-image features, whereas parameterised attention remains stable while highlighting object-relevant regions.

Adversarial robustness: Adversarial robustness is examined by generating images with the fast gradient sign method and observing fooling behaviour as perturbation L∞ norms increase.
Cross-domain classification: Cross-domain evaluation trains a linear SVM on final-layer features with C = 1 and 5-fold cross-validation, without fine-tuning on target datasets.The evaluation datasets are ZCA-whitened before classification.
Weakly supervised segmentation: For weakly supervised segmentation, attention maps from the last two levels are multiplied element-wise and square-rooted to reinterpret the result as a probability distribution.
Query-driven attention patterns: The analysis compares dot-product and parameterised compatibility functions using query images and cluttered target images from the Object Discovery cosegmentation set.The models are VGG-att2-concat-dp and VGG-att3-concat-pc, respectively.
Query-driven attention patterns: Attention changes are measured by comparing standard target-image processing with attention patterns produced after substituting global features from a distinct query image.The relative changes in spatial attention values are visualised across the figure columns.
Query-driven attention patterns: Dot-product attention increases attention magnitudes near or related to the object shown in the query image.This indicates that the global feature vector prominently guides the dot-product attention distribution.
Query-driven attention patterns: Parameterised attention is largely unaffected by changes in the global feature vector, because global features are often orders of magnitude smaller than corresponding local features.Its attention maps nevertheless consistently highlight object-relevant regions, apparently using object-centric features learned in the parameter vector.