Source-linked AI summary
Top-down Neural Attention by Excitation Backprop
Jianming Zhang, Zhe Lin, Jonathan Brandt, Xiaohui Shen, Stan Sclaroff
TL;DR
CNN classifiers need task-specific top-down attention maps that can support localization without extensive localization supervision. The paper models this attention with probabilistic Winner-Take-All and Excitation Backprop, adds contrastive attention, and reports strong localization results across benchmarks plus competitive phrase-to-region performance. Its evaluations are constrained by substantial computational costs for some comparisons.
Problem
The paper addresses the need to model task-specific top-down attention for CNN classifiers and the challenge of weakly supervised text-to-region association without fully annotated datasets.
Method
The method uses probabilistic Winner-Take-All, Excitation Backprop, and contrastive top-down signals to compute discriminative CNN attention maps.
Results
The method achieves superior weakly supervised localization on PASCAL VOC and MS COCO and competitive phrase-to-region performance against a fully supervised baseline on Flickr30k Entities.
Takeaways & Limitations
Top-down attention enables a CNN tag classifier trained on weakly labeled web images to localize visual concepts and support weakly supervised word-to-region association.
Takeaways & Limitations
Some comparisons are limited by computational resources, with LRP on COCO requiring over 950 hours on a single machine and evaluation restricted to CNN-S and GoogleNet.
Abstract
from arXiv · showhide
We aim to model the top-down attention of a Convolutional Neural Network (CNN) classifier for generating task-specific attention maps. Inspired by a top-down human visual attention model, we propose a new backpropagation scheme, called Excitation Backprop, to pass along top-down signals downwards in the network hierarchy via a probabilistic Winner-Take-All process. Furthermore, we introduce the concept of contrastive attention to make the top-down attention maps more discriminative. In experiments, we demonstrate the accuracy and generalizability of our method in weakly supervised localization tasks on the MS COCO, PASCAL VOC07 and ImageNet datasets. The usefulness of our method is further validated in the text-to-region association task. On the Flickr30k Entities dataset, we achieve promising performance in phrase localization by leveraging the top-down attention of a CNN model that has been trained on weakly labeled web images.
1 Introduction
The paper proposes probabilistic top-down attention for CNN classifiers, implemented through Excitation Backprop and enhanced with contrastive attention. It reports improved weakly supervised localization and competitive phrase-to-region performance using weakly labeled web images.
- Top-down task-driven attention supports efficient visual search by selecting neurons relevant to a given signal.
- The proposed CNN attention model replaces deterministic Winner-Take-All with a probabilistic Winner-Take-All formulation.This formulation models uncertainty in top-down selection.
- Excitation Backprop efficiently combines top-down and bottom-up information to compute winning probabilities and generate intermediate-layer attention maps.Intermediate maps avoid a complete backward sweep.
- Contrastive top-down attention captures the differential effect between paired top-down signals and improves attention-map discriminativeness.
- The method achieves superior weakly supervised localization performance on PASCAL VOC and MS COCO.The approach is also explored for localizing approximately 18K visual concepts with a classifier trained on 6M weakly labeled web images.
- Without localization supervision or a language model, the approach achieves competitive phrase-to-region performance against a fully supervised baseline on Flickr30k Entities.
2 Related Work
The related work situates the paper within biologically inspired top-down attention, CNN visualization, and weakly supervised localization. Existing localization approaches use feedback, relevance propagation, or multiple-instance learning, but coarse output maps limit spatial precision.
- Human visual attention research models top-down influences from knowledge, expectations, and behavioral goals on feature and location processing.
- The Selective Tuning model provides a biologically inspired top-down Winner-Take-All inference process related to the paper’s attention formulation.
- CNN grounding methods use error backpropagation, feedback architectures, or layer-wise relevance propagation to visualize regions relevant to predictions.
- Weakly supervised localization methods transform CNNs into fully convolutional networks and integrate Multiple Instance Learning through pooling over confidence maps.
- Large receptive fields and output strides make resulting score maps spatially coarse, motivating strategies such as image rescaling and shifting.
3 Method
The method models CNN top-down attention as probabilistic winner selection and propagates winning probabilities through the network using Excitation Backprop. It produces intermediate-layer attention maps and contrastive maps that emphasize task-relevant, discriminative regions.
- 3.1 Top-down Neural Attention based on Probabilistic WTA: Probabilistic WTA represents top-down attention as a distribution over output units and recursively samples winner neurons using conditional winning probabilities.Unlike deterministic WTA, this formulation produces soft attention maps and can represent uncertainty in the top-down signal.
- 3.1 Top-down Neural Attention based on Probabilistic WTA: The probabilistic WTA process is equivalent to an absorbing Markov chain whose walks start at output neurons and terminate at absorbing bottom-layer nodes.The fundamental matrix gives expected node visits, making marginal winning probability a linear function of the top-down output distribution.
- 3.2 Excitation Backprop: Excitation Backprop computes conditional winning probabilities from excitatory connection weights and nonnegative bottom-up responses, excluding inhibitory connections from competition.The method is designed for activation neurons whose responses are feature-detection signals and whose activations are nonnegative under the stated assumptions.
- 3.2 Excitation Backprop: Layer-wise propagation enables attention maps at any intermediate convolutional layer, formed by summing marginal winning probabilities across channels.Higher layers cover larger regions with lower spatial accuracy, whereas lower layers more precisely localize smaller-scale features.
- 3.3 Contrastive Top-down Attention: Contrastive MWP subtracts the non-target map from the target map, cancelling common winner neurons and amplifying discriminative neurons.A dual output unit with negated input weights supplies the contrastive non-target signal, and the resulting map requires a single backward pass.
4 Experiments
The experiments evaluate top-down attention for discriminative object localization, dominant-object localization, and text-to-region association. Across these settings, c-MWP generally performs strongly, while results also expose layer-selection and co-occurrence failure modes.
- The Pointing Game: The Pointing Game evaluates whether the maximum attention point falls within an annotated instance of the cued category, measuring spatial selectiveness across object categories.The difficult subset contains images where the target occupies less than one quarter of the image and at least one distracter category is present.
- The Pointing Game: c-MWP outperforms competing methods across VOC07 and COCO, including different CNN architectures, and exceeds the second-best method by about 10 percentage points on difficult sets using GoogleNet.It performs best in 69 of 80 COCO object categories, especially for small objects such as remote, tie, and baseball bat.
- The Pointing Game: At the I5b/out layer, c-MWP retains reasonable accuracy while Grad and Deconv fail to generate meaningful attention maps.The layer has low spatial resolution, but c-MWP remains conditioned on the layer’s activation values and can leverage its spatial information.
- The Pointing Game: Removing probabilistic normalization substantially degrades contrastive attention performance, while post-normalizing separately computed maps improves it but remains below the full method.The probabilistic formulation produces well-normalized maps that support direct subtraction of contrastive signals.
- Localizing Dominant Objects: For dominant-object localization, MWP is valuable because c-MWP often highlights only the most discriminative object part rather than fully covering the dominant object.This task favors complete object coverage, unlike the Pointing Game, which favors discriminativeness.
- Text-to-Region Association: Using a CNN tag classifier trained on approximately 6M weakly labeled thumbnail images, the method supports scalable word-to-region association without localization supervision.On Flickr30k Entities, the approach is evaluated on 1,000 test images with noun phrases manually associated with bounding-box regions.
- Text-to-Region Association: c-MWP consistently outperforms attention-map baselines across recall and mAP metrics, with a large margin over the second-best method in group-level mAP.The evaluation additionally reports performance on small instances whose bounding-box area is below 0.25 of the image size.
- Text-to-Region Association: With EdgeBoxes proposals, performance is lower than with MCG mainly because the EdgeBoxes bounding-box proposals are less accurate.The study also reports promising performance across several noun-phrase groups and comparisons with a fully supervised baseline.
5 Conclusion
The paper presents probabilistic top-down attention for CNNs through Excitation Backprop and contrastive signals, demonstrating localization performance across multiple settings.
- Excitation Backprop computes each neuron's Marginal Winning Probability for probabilistic top-down attention in CNN classifiers.The method propagates top-down signals through the network and can generate attention maps at intermediate convolutional layers.
- Contrastive top-down attention produces highly discriminative attention maps by propagating a pair of contrastive signals through a single backward pass.
- The method demonstrates accuracy and generalizability in large-scale Pointing Game evaluation and dominant-object localization.
- Without localization supervision or a language model, the method achieves competitive localization performance against a state-of-the-art fully supervised method on Flickr30k Entities.
A Speed Performance
Excitation Backprop is computationally more expensive than error backpropagation, but the implementation can stop at an intermediate layer for practical speed.
- The implementation measures Excitation Backprop speed on a NVIDIA K40c GPU for a single 224X335 image without batch mode.Figure 8 compares Excitation Backprop with error backpropagation in GPU mode across tested termination layers.
- Excitation Backprop has computational complexity about twice that of error backpropagation.The most time-consuming operations are the layer's forward and backward operations in Caffe.
- In practice, Excitation Backprop is applied only to an intermediate layer rather than through the complete network.
B Details about the Stock6M Dataset
Stock6M is constructed from weakly labeled stock-image thumbnails and tags, then filtered to about 6M images; its tags include both visual and non-visual concepts.
- Data collection and cleaning: The Stock6M dataset starts from about 17M thumbnail images and their stock-website tags.The source website provides professional photos and illustrations, with tags used for text-based image search.
- Data collection and cleaning: The tag dictionary contains the most frequent 18157 tags, selected using a frequency threshold of 1000.Most tags are unigrams, and images with fewer than five tags are removed.
- Data collection and cleaning: After removing images with fewer than five tags and more than three overlaps, the dataset contains about 6M images.
- Data collection and cleaning: Images whose tags are in alphabetical order are removed because that ordering does not reflect image relevance.The remaining tag ordering can roughly reflect relevance to the image.
- Frequent Tags and Example Images: The most frequent tags are often human-related, but many are non-visual tags such as healthy, business, holiday, and lifestyle.Figure 9 visualizes the most frequent tags in a word cloud, while Figure 10 shows example images with corresponding user tags.