Source-linked AI summary
Tell Me Where to Look: Guided Attention Inference Network
Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, Yun Fu
TL;DR
Weakly supervised attention maps can be incomplete and can reflect dataset bias, limiting their use as localization and segmentation priors. The paper introduces GAIN to train attention explicitly, add self-guidance, and incorporate extra supervision end to end. It reports state-of-the-art PASCAL VOC 2012 semantic-segmentation results under weak and limited pixel-level supervision.
Problem
Classification-only attention maps often cover small discriminative regions and may focus on correlated background cues, limiting their suitability as complete task-specific localization priors.
Method
GAIN makes attention maps trainable during end-to-end learning, applies self-guidance to expand them, and adds GAINext for extra supervision.
Results
The method achieves mIoU 55.3% on PASCAL VOC 2012 val and 56.8% on test under weak supervision, with 60.5% and 62.1% respectively when 1464 pixel-labeled images are used.
Takeaways & Limitations
The framework provides direct task-oriented guidance for attention maps and can improve attention completeness, localization or segmentation priors, and generalization under biased data.
Abstract
from arXiv · showhide
Weakly supervised learning with only coarse labels can obtain visual explanations of deep neural network such as attention maps by back-propagating gradients. These attention maps are then available as priors for tasks such as object localization and semantic segmentation. In one common framework we address three shortcomings of previous approaches in modeling such attention maps: We (1) first time make attention maps an explicit and natural component of the end-to-end training, (2) provide self-guidance directly on these maps by exploring supervision form the network itself to improve them, and (3) seamlessly bridge the gap between using weak and extra supervision if available. Despite its simplicity, experiments on the semantic segmentation task demonstrate the effectiveness of our methods. We clearly surpass the state-of-the-art on Pascal VOC 2012 val. and test set. Besides, the proposed framework provides a way not only explaining the focus of the learner but also feeding back with direct guidance towards specific tasks. Under mild assumptions our method can also be understood as a plug-in to existing weakly supervised learners to improve their generalization performance.
1. Introduction
The paper makes attention maps trainable during end-to-end weakly supervised learning, using direct and self-guided supervision to improve their task relevance and completeness. On PASCAL VOC 2012 semantic segmentation, the approach reports state-of-the-art performance under weak and limited pixel-level supervision.
- Motivation: Classification-only attention maps often cover only small, discriminative object regions, motivating direct task-specific supervision during training.More complete foreground attention can improve segmentation priors and related localization tasks.
- Motivation: Attention maps can expose dataset bias, such as a boat classifier focusing on correlated water backgrounds rather than the foreground object.The paper proposes controlling attention directly instead of relying only on dataset re-balancing.
- Contributions: GAIN applies supervision directly to attention maps while learning a weakly labeled task, making attention an explicit component of end-to-end training.The framework is designed to support task-specific guidance during training.
- Contributions: Self-guidance encourages attention to cover objects holistically rather than only their most discriminative parts.This is presented as a training scheme within the common GAIN framework.
- Results: 55.3% mIoU on PASCAL VOC 2012 val and 56.8% on test are reported for semantic segmentation under weak supervision.The paper describes these as new state-of-the-art results under weak supervision.
2. Related work
Prior attention-based methods mainly use network explanations as localization cues, but classification-trained attention can remain incomplete. GAIN instead supervises attention end to end and guides it toward task-relevant regions, including under dataset bias.
- Existing attention methods: Back-propagation-based visual explanations identify regions relevant to predicted classes or hidden-neuron activations.Existing methods include feedback CNNs and CAM for capturing or visualizing task-relevant regions.
- GAIN: GAIN differs by supervising network attention directly in an end-to-end model rather than only explaining the trained network.The paper validates that this supervision guides attention toward expected regions and benefits visual tasks.
- Existing attention methods: Classification-trained attention maps support weakly supervised localization and segmentation but often cover only small discriminative object regions.These tasks require denser and more complete regions than classification attention typically provides.
- GAIN: GAIN uses classification and attention-mining streams with shared parameters to include regions that contribute to classification decisions.The classification stream finds useful regions, while attention mining seeks to include all contributing regions.
- Dataset bias: GAIN addresses dataset bias by guiding attention toward task-critical areas instead of requiring complete dataset debiasing or a newly built dataset.The paper describes this as making the learned network robust to dataset bias.
3. Proposed method — GAIN
GAIN makes attention maps trainable within an end-to-end weakly supervised framework, using self-guidance to expand them toward complete task-relevant regions. GAINext additionally integrates sparse external supervision, allowing the same framework to span weak and partially supervised semantic segmentation.
- GAIN self-guidance: GAIN generates attention maps online from convolutional activations and class-specific importance weights, then makes constraints on those maps influence network learning.The maps integrate activation maps with a weight matrix followed by ReLU and are computed from the last convolutional representation.
- GAIN self-guidance: A soft threshold mask extracts regions beyond the current attention, producing an input I*_c for class-specific attention mining.The sigmoid approximates thresholding so the mask remains differentiable; its threshold and scale determine near-binary behavior.
- GAIN self-guidance: Attention Mining Loss minimizes the prediction score of the masked input for each ground-truth class, encouraging the attention map to cover additional object regions.The loss averages class scores over the image’s ground-truth labels.
- GAIN self-guidance: The self-guidance objective combines classification loss with Attention Mining Loss, and its guidance extends the network’s focus toward more complete segmentation-oriented regions.The final self-guidance loss uses weighting parameter α, set to 1 in the reported experiments.
- GAINext: GAINext adds an externally supervised loss to the shared framework, using pixel-level masks or other annotations to directly guide attention toward the task of interest.The extension retains the trainable attention-map generation of GAIN and adds an external supervision stream; ω controls emphasis on that supervision and was set to 10.
- GAINext: GAINext can be adapted beyond segmentation, including bounding-box supervision for learning features intended to be more robust to dataset bias.The paper presents this as an application to improving generalizability when training and testing distributions differ.
4. Semantic segmentation experiments
The experiments evaluate GAIN-based attention priors for weakly supervised semantic segmentation on PASCAL VOC 2012, including settings with limited pixel-level supervision. Results show improved segmentation performance and more complete class-region coverage than baseline attention priors.
- Experimental setting: Weakly supervised semantic segmentation uses localization cues from image-level models, making cue quality central to segmentation performance.The task classifies each pixel, while classification-trained attention often covers only the most discriminative object regions.
- Comparison with state-of-the-art: GAIN-based SEC achieves 55.3% and 56.8% mIoU on the VOC 2012 val. and test sets, outperforming SEC by 4.6% and 5.1%.These results are reported for methods using purely image-level labels.
- Comparison with state-of-the-art: GAIN improves over TPL by 2.2% and 3.0% on the VOC 2012 val. and test sets.Unlike approaches relying on iterative erasing or multiple networks, GAIN makes the attention map trainable and uses Lself guidance.
- Comparison with state-of-the-art: With pixel-level supervision on 1464 images, GAINext-based SEC reaches 60.5% and 62.1% mIoU on VOC 2012 val. and test sets.The authors report these results as a new state-of-the-art for the task.
- Effect of extra supervision: Increasing the amount of pixel-level supervision improves GAINext attention-map learning, while qualitative results show more complete and accurate object regions.GAIN-based methods can discover additional object parts or instances that the baseline misses, with less surrounding background in some GAINext maps.
5. Guided learning with biased data
The experiments test whether attention guidance improves classification generalization under biased data. GAIN and GAINext improve performance, with extra pixel-level or bounding-box supervision providing stronger robustness to shifts in foreground-background correlations, viewpoints, and backgrounds.
- Experimental setup: The experiments are designed to assess whether attention guidance makes classifiers robust to dataset bias and improves generalization.The boat dataset contrasts boats without water with water without boats; the camera tests include an altered viewpoint-and-background set.
- Boat experiment: GAIN raises biased-boat overall accuracy from 36% with Grad-CAM to 55% using only image-level supervision.On water-without-boat images, accuracy improves by 32%, with the relative error rate reduced by almost 50%.
- Boat experiment: GAINext reaches 66% overall accuracy with 9 pixel-level boat labels and 84% with 78 labels.The 9-label result is an 11% improvement over GAIN with self-guidance, while the 78-label setting converges across both scenarios.
- Boat experiment: Direct attention guidance helps decouple target objects from correlated background features such as water.The paper attributes this effect to pixel-level labels identifying relevant object parts and features.
- Industrial camera experiment: Grad-CAM achieves close to 100% accuracy on the in-distribution camera test set but close to 50% on the shifted test set.Testing Set 2 uses different camera viewpoints and backgrounds, exposing severe training-data bias.
- Industrial camera experiment: GAINext uses only 5% bounding-box supervision in the camera experiment to focus attention on critical areas and improve generalization on shifted images.The external supervision consists of 20 manually drawn boxes per class.
6. Conclusions
The framework makes attention maps explicit during training and guides them directly to improve their accuracy and completeness. The resulting system outperforms the state of the art without recursive runtime processing and is presented as useful for biased-data robustness and localization or segmentation priors.
- 6. Conclusions: The framework treats attention maps as first-class training components and provides direct guidance to make them more accurate and complete.It combines direct guidance with the attention map generated by a weakly supervised deep neural network.
- 6. Conclusions: The resulting system confidently outperforms the state of the art without recursive processing at runtime.The conclusion presents this as an outcome of extensive experiments.
- 6. Conclusions: The framework can improve robustness and generalization with biased data and provide more complete attention maps for localization and segmentation priors.The paper identifies other high-level tasks, including regression, as future application areas.