Learn To Pay Attention

Motivation and background

• CNN reasoning remains difficult to interpret, while attention maps represent spatial importance and have been used to localize objects, identify segmentation masks, and reveal discriminative class properties.Prior examples include Simonyan et al. (2013), Cao et al. (2015), and Zhou et al. (2016); training smaller networks to mimic larger networks’ attention maps has also improved their classification accuracy (Zagoruyko & Komodakis, 2016).Source: §1 · p. 1

Contribution

• The contribution repurposes the global image representation as a query for estimating multi-scale attention in classification, which does not naturally provide a query.The attention is represented as a probabilistic map over input image locations and estimated end-to-end.Source: §1 · p. 2

Method overview

• The method defines compatibility between global and local features and requires classification to use only an attention-weighted combination of local features, encouraging task-relevant attention.Local features have receptive fields covering proper subsets of the image; global features have receptive fields covering the entire image.Source: §1 · p. 2

Experiments

• Adding attention to VGG improves CIFAR-100 accuracy by 7%, while attention-weighted representations improve fine-grained recognition and generalisation across 6 domain-shifted benchmark datasets.Experiments apply the mechanism to VGGNet and ResNet and capture coarse-to-fine attention maps at multiple levels.Source: §1 · p. 2
• On Object Discovery, trained attention maps outperform CNN-derived attention maps, traditional saliency maps, and top object proposals for weakly supervised segmentation.The cited CNN attention maps are from Zhou et al. (2016), saliency maps from Jiang et al. (2013) and Zhang & Sclaroff (2013), and the dataset from Rubinstein et al. (2013).Source: §1 · p. 2
• Attention-aware bird-recognition models provide limited resistance to adversarial fooling at low and moderate L∞-noise norms.Source: §1 · p. 2

Post hoc and trainable attention

• Prior CNN attention either post-processes fully trained classifiers or trains attention jointly with network weights; Zhou et al. (2016) improve localisation at the cost of classification accuracy.Post hoc examples include gradient-based maps and attention extracted through test-time tuning; trainable attention optimizes the attention unit alongside the original network.Source: §2 · pp. 2–3

Hard and soft attention

• Hard attention makes nondifferentiable region choices trained with REINFORCE, whereas soft attention is probabilistic and backpropagation-friendly; Seo et al. (2016) use nonuniform, non-rigid maps suited to natural shapes.Jaderberg et al. (2015) combine deterministic hard attention with differentiable transformation-parameter estimation; the paper follows Seo et al.'s soft-attention direction.Source: §2 · p. 3

Query-based attention

• Unlike Seo et al. (2016), this work queries images with a learned global representation rather than one-hot category labels and uses multiple-scale complementary attention instead of refining one map sequentially.The related work frames image-to-representation mapping as guiding inference, as in query-based tasks using captions, questions, or source sentences.Source: §2 · p. 3

Domain shift

• For domain shift, the work joins attention-based approaches that selectively focus on objects amid background changes, occlusion, and pose variation, rather than relying only on target-dataset fine-tuning.Traditional fine-tuning may require thousands of images per target category for successful adaptation.Source: §2 · p. 3

Weakly supervised semantic segmentation

• For weakly supervised segmentation, the work tests whether classification-trained, binarized spatial attention maps can segment objects using image-level category labels without pixel-level segmentation training.The related approach includes soft attention by Hong et al. (2016).Source: §2 · p. 3

Adversarial robustness

• For adversarial robustness, the work explores suppressing features in the spatial domain, building on accounts that link vulnerability to reliance on spurious features and robustness gains from masking feature dimensions.Goodfellow et al. (2014) describe imperceptible perturbations along the direction opposite the predicted class-score gradient; Wang et al. (2016) and Gao et al. (2017) discuss spurious features.Source: §2 · p. 3

3.1 DESIGN AND TRAINING OF ATTENTION SUBMODULE

• At selected convolutional layers, projected local feature vectors receive softmax-normalized compatibility scores against the global descriptor, and their weighted average replaces that descriptor for classification.The classifier is restricted to using the attention-weighted descriptors alone.Source: §3.1 · p. 4
• For multiple attention layers, the model either concatenates their weighted descriptors or averages class probabilities from separate linear classifiers; all free parameters are learned end to end with cross-entropy.A single attention layer instead maps its weighted descriptor to class probabilities through a linear layer and softmax.Source: §3.1 · p. 4

3.2 CHOICE OF COMPATIBILITY FUNCTION C

• The compatibility function can add local and global descriptors before a single fully connected mapping produces scores, limiting attention-unit parameters; its weights may learn features related to objectness.Score magnitudes also depend on descriptor alignment and local activation strength.Source: §3.2 · p. 5

3.3 INTUITION

• Attention is placed late in the CNN, before pooling, and across spatial resolutions so mature features and complementary details at different scales can contribute.Softmax normalization makes the weights a convex combination, forcing spatial locations to compete for attention.Source: §3.3 · p. 5

4 EXPERIMENTAL SETUP

• VGG and 164-layer ResNet architectures use added convolutional and pooling layers to form global features, with local descriptors from selected layers or blocks for attention.For VGG, the local maps come from layers 7, 10, and 13; for ResNet, they come from blocks 2, 3, and 4. When dimensions differ, the global vector is projected to the lower-dimensional local-feature space to limit classification parameters.Source: §4 · pp. 5–6
• The study compares against activation-based attention by Zhou et al. (2016) and progressive attention by Seo et al. (2016), while varying attention levels and descriptor matching and classification strategies.Net-att, Net-att2, and Net-att3 use attention at the last one, two, and three levels; matching uses dot product (dp) or parametrised compatibility (pc), while final classification uses concatenation (concat) or independent level predictions (indep).Source: §4 · pp. 5–6
• The evaluation covers image classification, fine-grained object category recognition, robustness to adversarial attack, and attention-map quality as segmentation of objects in predicted categories.The adversarial-attack setup follows the kind discussed by Goodfellow et al. (2014).Source: §4 · p. 6

5.1 IMAGE CLASSIFICATION AND FINE-GRAINED RECOGNITION

• VGG-att2concat-pc improves over baseline VGG by 2.5% and 7.4% on CIFAR-10 and CIFAR-100, and 7.8% and 0.5% on CUB and SVHN.Attention maps focus on objects and suppress background; different layers specialize on parts such as eyes, beaks, plumage, and feet. On CUB, the networks are on par with ImageNet-pretrained RN-34 despite using CIFAR-100 pretraining; introducing the mechanism into residual networks instead lowers performance by 0.9% on CIFAR-10 and 1.5% on CIFAR-100.Source: §5.1 · pp. 6–7, Table 1 · p. 6, Table 2 · p. 6

5.2 ROBUSTNESS TO ADVERSARIAL ATTACK

• At an L∞ noise norm of 1, attention-aware VGG has a 5% lower fooling rate than baseline VGG, but at perceptible noise its fooling rate is around a percentage higher.As noise increases, fooling rates saturate for both networks and their performance gap gradually decreases.Source: §5.2 · p. 7, Figure 6 · p. 7, Figure 5 · p. 7

5.3 CROSS-DOMAIN IMAGE CLASSIFICATION

• Attention-aware models trained on CIFAR-10/100 improve unseen-dataset classification over baseline models by an average margin of 6%.Low-resolution CIFAR images transfer to high-resolution images, including Event-8; CIFAR-100-trained models produce more diverse attention maps and capture nuanced regions that help classify new datasets. STL shares CIFAR-10’s categories and is evaluated directly without an SVM.Source: §5.3 · pp. 7–8, Table 3 · p. 7, Figure 7 · p. 8

5.4 WEAKLY SUPERVISED SEMANTIC SEGMENTATION

• Binarized attention maps outperform existing attention methods by at least 11% and top object proposals by 3% across all three categories, but beat saliency methods only for cars.The method performs single-image inference while comparing favorably with joint-inference methods; it does not compare with CNN-based object proposals trained using additional bounding-box annotations.Source: §5.4 · p. 8, Figure 9 · p. 8, Figure 8 · p. 8

• The trainable attention module generates probabilistic landscapes showing where and in what proportion a network attends to image regions for classification.Source: §6 · p. 8
• When deployed at multiple network levels, the method improves classification of seen and unseen categories by focusing on the object of interest.Source: §6 · p. 8
• The attention landscapes facilitate weakly supervised segmentation of the predominant object and can be refined with conditional random fields.Source: §6 · p. 8
• The attention scheme has shown promise for learning robustness to certain kinds of adversarial attacks.Source: §6 · p. 8

A.1 DATASETS

• Classification uses CIFAR-10, CIFAR100, SVHN, and CUB-200-2011.Source: §A.1 · p. 11
• The attention-incorporating VGG model trained on CUB-200-2011 is used to investigate robustness to adversarial attacks.Source: §A.1 · p. 11
• Cross-domain classification is tested on six standard benchmarks, including STL, Caltech-256, and Action-40.Source: §A.1 · p. 11
• Object Discovery (Rubinstein et al., 2013) is used to evaluate weakly supervised segmentation.Source: §A.1 · p. 11
• Except for SVHN and CUB, datasets receive mean-and-standard-deviation and colour normalisation.Source: §A.1 · p. 11
• CUB-200-2011 images are cropped using ground-truth bounding boxes and resized; cross-domain inputs are downsampled to reduce memory overhead.Source: §A.1 · p. 11
• The dataset collection spans plain-background objects to cluttered indoor and outdoor scenes, and simple digits to humans performing complex activities.Source: Table 4 · p. 11

Progressive attention networks

• Progressive VGG attention filters lower-level features with independently sigmoid-normalised compatibility scores, then uses softmax-normalised scores to combine final-level features for classification.The lower-level attention is at layer 10; final-level attention follows the VGG-att2-concat-pc setup.Source: §A.2 · p. 11

Attention in ResNet

• The ResNet baseline has four levels and 9n + 2 parameterised layers, with n residual blocks in each later level; n=18 yields a 164-layer network.The levels project RGB input through 16-, 64-, and 128-dimensional embedding spaces to 256 dimensions.Source: §A.2 · pp. 11–12
• Replacing final-level spatial average pooling with convolution and max pooling forms the global feature vector; RN-att and RN-att2 attend at the last level and last two levels, respectively, using concat-pc training.Source: §A.2 · p. 12

VGG training

• VGG models for CIFAR-10, CIFAR-100, and SVHN train from scratch for 300 epochs, with learning rates halved every 25 epochs.The initial learning rate is 1 for CIFAR experiments and 0.1 for SVHN; batch size is 128, weight decay is 5 × 10^-4, and momentum is 0.9.Source: §A.3 · p. 12

CUB training

• For CUB, VGG models start from CIFAR-100 weights and follow a transfer-learning schedule inspired by Redmon et al. (2015).Training starts at learning rate 0.1 for 30 epochs, doubles the rate twice over the next 60 epochs, then halves it every 30 epochs for 200 epochs.Source: §A.3 · p. 12

ResNet training

• ResNet models for CIFAR-10 and CIFAR-100 train from scratch for 200 epochs, with learning rate multiplied by 0.2 after epochs 60, 120, and 160.The schedule uses SGD with batch size 64, initial learning rate 0.1, weight decay 5 × 10^-4, and momentum 0.9.Source: §A.3 · p. 12

A.4 TASK-SPECIFIC PROCESSING

• Adversarial images are generated with the fast gradient sign method of Goodfellow et al. (2014), and network fooling is examined as perturbation L∞ norms increase.Source: §A.4 · p. 12
• Cross-domain classification uses final-layer features to train a linear SVM with C = 1 and reports 5-fold cross-validation without target-dataset finetuning.Evaluation datasets are ZCA-whitened; the setup is similar to Zhou et al. (2016).Source: §A.4 · p. 12
• For weakly supervised segmentation, proposed attention maps from the last two levels are multiplied, square-rooted, rescaled to [0, 1], and binarised with Otsu thresholding.Seo et al. (2016)'s sequential maps are multiplied without the square root; magnitude rescaling and binarisation are also applied to GAP maps (Zhou et al., 2016), and evaluation images receive colour normalisation.Source: §A.4 · p. 12

A.5 QUERY-DRIVEN ATTENTION PATTERNS

• The global feature vector serves as a query for local-region attention, so changing it can alter the estimated attention distribution depending on the compatibility function.Source: §A.5 · p. 12
• The query-driven analysis compares dot-product and parameterised compatibility in two VGG models using a central query image and cluttered same-category target images.The models are (VGG-att2)-concat-dp (DP) and (VGG-att3)-concat-pc (PC), and the images come from the Object Discovery package's extra cosegmentation set.Source: §A.5 · p. 13
• Two experimental rounds compare standard target-derived global and local features with target-local features paired with a category-query global feature, visualising attention and relative score changes at layer 10.Source: §A.5 · p. 13, Figure 10 · p. 14
• Dot-product attention responds to query-image features, while parameterised attention is largely unchanged by the global vector yet consistently highlights object-relevant regions.For parameterised attention, global features are often orders of magnitude smaller than local features; the authors suggest object-centric high-order features are learned in the weight vector u and generalise when test categories resemble training categories.Source: §A.5 · p. 13, Figure 10 · p. 14