Source-linked AI summary
Self-Erasing Network for Integral Object Attention
Qibin Hou, Peng-Tao Jiang, Yunchao Wei, Ming-Ming Cheng
TL;DR
Adversarial erasing can expand attention into non-object regions as training proceeds, reducing attention-map quality. SeeNet addresses this with two self-erasing strategies based on object and background cues, then evaluates the resulting maps through weakly supervised semantic segmentation. On PASCAL VOC, the resulting segmentation substantially outperforms existing state-of-the-art results, while difficult multi-object and low-contrast scenes remain failure cases.
Problem
Adversarial erasing may spread attention into background regions as training iterations increase, degrading localization and harming segmentation supervision.
Method
SeeNet thresholds initial attention maps into zones and uses two self-erasing strategies, including C-ReLU-based background suppression, to confine attention to likely semantic regions.
Results
SeeNet’s proxy-ground-truth segmentation results greatly outperform existing state-of-the-art results on PASCAL VOC 2012.
Takeaways & Limitations
The approach produces attention maps that better focus on semantic objects, preserve relatively integral objects, and avoid unexpected non-object regions.
Takeaways & Limitations
Complex backgrounds, low object-background contrast, and multiple semantic objects can make whole-object localization and precise separation difficult.
Abstract
from arXiv · showhide
Recently, adversarial erasing for weakly-supervised object attention has been deeply studied due to its capability in localizing integral object regions. However, such a strategy raises one key problem that attention regions will gradually expand to non-object regions as training iterations continue, which significantly decreases the quality of the produced attention maps. To tackle such an issue as well as promote the quality of object attention, we introduce a simple yet effective Self-Erasing Network (SeeNet) to prohibit attentions from spreading to unexpected background regions. In particular, SeeNet leverages two self-erasing strategies to encourage networks to use reliable object and background cues for learning to attention. In this way, integral object regions can be effectively highlighted without including much more background regions. To test the quality of the generated attention maps, we employ the mined object regions as heuristic cues for learning semantic segmentation models. Experiments on Pascal VOC well demonstrate the superiority of our SeeNet over other state-of-the-art methods.
1 Introduction
Adversarial erasing can discover additional object regions, but continued training may spread attention into background and damage localization. SeeNet introduces self-erasing to confine attention and improves weakly supervised segmentation results on PASCAL VOC 2012.
- Adversarial erasing uses initial and complementary attention generators to discover additional object-related regions.The strategy erases detected regions before generating complementary attention.
- As training continues, attention may spread into background regions, degrading the initial generator’s localization ability.Rails may be classified as trains after train regions are erased, negatively affecting segmentation learning.
- SeeNet introduces self-erasing strategies that use background priors to suppress attention spread and discover high-quality attention in a potential zone.The method explicitly feeds attention networks roughly accurate background priors and restricts discovery to likely semantic regions.
- The generated attention maps are evaluated as heuristic cues for weakly supervised semantic segmentation on PASCAL VOC 2012.The paper reports substantial improvement over existing methods.
2 Related Work
Prior work visualized CNN attention and developed adversarial erasing to mine unseen semantic regions. Weakly supervised segmentation methods reduce annotation costs by using image-level labels, attention maps, saliency cues, or alternative learning strategies.
- Earlier Work: Earlier attention methods include back-propagation visualization, CAM, Grad-CAM, and architectures motivated by human visual systems.CAM uses global average pooling followed by a fully connected classifier, while Grad-CAM extends visualization across networks and tasks.
- Adversarial Erasing Strategy: Adversarial erasing removes initially detected regions and sends the erased images to another CNN to mine more discriminative semantic regions.Later methods extended this strategy to end-to-end training.
- Weakly-Supervised Semantic Segmentation: Weakly supervised semantic segmentation replaces expensive pixel-level annotations with weaker supervision such as scribbles, points, bounding boxes, or image-level labels.Image-level supervision is attractive because pixel annotations require substantial human labor.
- Ternary masks separate initial attention maps into high-attention, potential-object, and background-prior zones.Two thresholds define the three zones, with low-attention areas explicitly treated as background priors.
- Weakly-Supervised Semantic Segmentation: Many image-level methods use attention models or saliency cues to generate initial seeds, while others use multiple instance learning or EM algorithms.These approaches address segmentation under keyword-level or image-level supervision.
3 Self-Erasing Network
SeeNet uses ternary attention masks and two self-erasing strategies to restrict attention to likely semantic regions while suppressing background responses. Its three-branch network combines conditional feature reversal, background suppression, and attention-map fusion.
- 3.2 The Idea of Self-Erasing: SeeNet divides each image into attention, background, and potential zones using thresholds applied to an initial attention map.The potential zone is intended to contain semantic objects with high probability, while the background zone supplies a prior.
- 3.2 The Idea of Self-Erasing: C-ReLU conditionally reverses feature-map signs according to a binary mask, making potential-zone activations more prominent relative to background activations.The function is C-ReLU(x) = max(x, 0) × B(x), where B takes values from {−1, 1}.
- Network architecture and training: The architecture has three branches after a shared backbone: SA generates initial attention, while SB and SC incorporate C-ReLU layers.The overall loss is L = LSA + LSB + LSC, with SC receiving a zero label vector.
- Self-erasing strategy I: Self-erasing strategy I feeds the ternary mask into branch SB, erasing detected attention and reversing positive background responses to emphasize the potential zone.This branch is trained to deter discovery of background content while preserving distinctive potential regions.
- Self-erasing strategy II: Self-erasing strategy II uses branch SC to activate only the background zone and trains its probability of belonging to any semantic class toward zero.This branch helps correct wrongly predicted background attention and indirectly limits attention spread.
- Network architecture and training: At test time, SeeNet discards SC and fuses normalized SA and SB maps, additionally combining the original and horizontally flipped inputs.The final map takes the pixelwise maximum across the fused maps.
4 Weakly-Supervised Semantic Segmentation
The method builds proxy ground truths for weakly-supervised semantic segmentation by combining normalized saliency and attention cues, with a harmonic mean controlling their contribution.
- Generated attention maps are applied to weakly-supervised semantic segmentation as cues for producing proxy ground truths.The resulting proxy ground truths supervise a segmentation network.
- The harmonic mean combines saliency-based background information with class-specific attention maps when estimating pixel-class probabilities.The attention-map weight parameter w controls the importance of attention maps and is set to 1 in the experiments.
5 Experiments
Experiments on PASCAL VOC evaluate SeeNet through visual attention comparisons, ablations, and weakly-supervised segmentation benchmarks. SeeNet suppresses background expansion while retaining more integral object regions and achieves stronger segmentation results than the compared settings and methods.
- Experimental Setup: SeeNet is evaluated on PASCAL VOC 2012 using 10,582 training images, validation and test sets, and mean intersection-over-union as the metric.The validation and test sets contain 1,449 and 1,456 images, respectively.
- Ablation Studies: The experiments compare standard SeeNet with ACoL and a variant that zeros feature units associated with background regions.These settings are used to examine the role of the proposed self-erasing strategies.
- The Role of Self-Erasing: SeeNet focuses more accurately on small objects and mines relatively integral large objects while limiting attention spread into background regions.The SC branch supports background recognition, while conditional reversion protects attention areas from spreading.
- Comparison with the State-of-the-Arts: SeeNet greatly outperforms existing methods using the same VGGNet base model and exceeds DCSP by more than 2% on the validation set.Its segmentation results are also reported as better than AE-PSL and GAIN.
- Discussions: Failure cases arise when differently labeled objects are tied together, backgrounds are complex, object-background contrast is low, or multiple objects must be localized.Image-level labels make precise separation and complete localization difficult in these settings.
6 Conclusion
SeeNet extracts background priors from initial attention maps and uses two self-erasing strategies to prevent attention from spreading to unexpected regions. The resulting attention maps serve as cues for semantic segmentation, whose results reportedly outperform existing state-of-the-art results.
- SeeNet thresholds initial attention maps into three zones to extract roughly accurate background priors.These priors support the subsequent self-erasing strategies.
- Two self-erasing strategies use background priors to prohibit attention regions from spreading to unexpected regions.
- The self-erasing attention network confines observable regions to a potential zone where semantic objects are likely to exist.
- The mined attention regions are combined with saliency maps as heuristic cues for weakly-supervised semantic segmentation.
- Segmentation results based on the resulting proxy ground truths greatly outperform existing state-of-the-art results.