Source-linked AI summary
Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach
Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, Shuicheng Yan
TL;DR
Weakly supervised segmentation must obtain dense pixel-level object regions from image-level labels, whereas classification networks tend to focus on small, sparse discriminative areas. The paper progressively mines complementary regions through adversarial erasing and improves their supervision with online prohibitive segmentation learning, reporting 55.0% mIoU on PASCAL VOC 2012 val and 55.7% on test as new state-of-the-art results.
Problem
Classification networks focus on small, sparse discriminative regions, while semantic segmentation requires dense, interior, and integral object regions for pixel-wise inference.
Method
Adversarial erasing repeatedly removes mined regions so classification discovers complementary object areas, while online prohibitive segmentation learning modulates auxiliary supervision with classification confidences.
Results
55.0% mIoU on PASCAL VOC 2012 val and 55.7% on test are reported as new state-of-the-art results.
Takeaways & Limitations
The approach provides a simple weakly supervised segmentation solution in which progressively discovered regions supply pixel-level supervision and PSL provides auxiliary information to AE.
Takeaways & Limitations
A failure case on complex images may require a better erasing strategy using low-level visual features such as color and texture to refine and extend erased regions.
Abstract
from arXiv · showhide
We investigate a principle way to progressively mine discriminative object regions using classification networks to address the weakly-supervised semantic segmentation problems. Classification networks are only responsive to small and sparse discriminative regions from the object of interest, which deviates from the requirement of the segmentation task that needs to localize dense, interior and integral regions for pixel-wise inference. To mitigate this gap, we propose a new adversarial erasing approach for localizing and expanding object regions progressively. Starting with a single small object region, our proposed approach drives the classification network to sequentially discover new and complement object regions by erasing the current mined regions in an adversarial manner. These localized regions eventually constitute a dense and complete object region for learning semantic segmentation. To further enhance the quality of the discovered regions by adversarial erasing, an online prohibitive segmentation learning approach is developed to collaborate with adversarial erasing by providing auxiliary segmentation supervision modulated by the more reliable classification scores. Despite its apparent simplicity, the proposed approach achieves 55.0% and 55.7% mean Intersection-over-Union (mIoU) scores on PASCAL VOC 2012 val and test sets, which are the new state-of-the-arts.
1. Introduction
The paper addresses the mismatch between efficient image-level classification supervision and segmentation’s need for dense object regions. It progressively mines regions with adversarial erasing and complements them with prohibitive segmentation learning, achieving state-of-the-art PASCAL VOC results.
- Motivation: Image-level labels are efficient to collect, but classification networks typically identify small, sparse regions rather than the dense, integral regions required for segmentation.This mismatch complicates assigning image-level labels to pixels for end-to-end segmentation learning.
- Adversarial Erasing: Adversarial erasing repeatedly removes the currently discovered region and retrains classification to uncover additional discriminative regions.The erased regions are progressively merged into a pixel-level foreground mask for segmentation supervision.
- Online PSL: Online prohibitive segmentation learning uses classification confidences to modulate category-specific response maps, reducing noisy contributions to the auxiliary supervision mask.It is designed to recover ignored object-related regions and improve segmentation training alongside AE.
- Contributions: The proposed AE approach progressively expands target regions into contiguous object masks usable for training semantic segmentation models.This provides a classification-based route to richer pixel-level supervision.
- Contributions: Online PSL is proposed as a collaborative method that uses image-level classification confidences to reduce supervision-mask noise and improve segmentation-network training.The contribution is explicitly framed as complementary to AE.
- Results: 55.0% mIoU on PASCAL VOC 2012 val and 55.7% on test are reported as new state-of-the-art results.These results are presented as the method’s benchmark achievement.
2. Related Work
Prior weakly supervised segmentation methods use increasingly coarse annotations or image-level labels, while classification-based approaches offer efficiency but often produce sparse object cues. The paper’s AE approach targets this sparsity by progressively mining denser regions for segmentation supervision.
- Weak Supervision: Weakly supervised segmentation reduces pixel-level annotation demands through bounding boxes, scribbles, instance points, and image-level labels.Image-level labels are described as the simplest supervision for training segmentation models.
- Image-Level Methods: Image-level methods include multiple-instance learning, constrained CNNs, expectation-maximization, and progressively enhanced training frameworks.These methods address segmentation with only image-level labels but earlier approaches are described as unsatisfactory.
- Adversarial Erasing: AE progressively erases discovered regions and mines additional discriminative regions whose union provides richer supervision for semantic segmentation.The procedure repeatedly updates processed images and combines mined regions across steps.
- Classification-Based Methods: Classification-based approaches efficiently localize object cues, but prior methods can obtain only small and sparse object-related seeds for supervision.The paper contrasts this limitation with AE’s ability to mine dense object-related regions.
3. Classification to Semantic Segmentation
The approach combines adversarial erasing to progressively mine object regions with online prohibitive segmentation learning to refine supervision and train semantic segmentation networks.
- Approach: The proposed approach includes object region mining with adversarial erasing and online prohibitive segmentation learning.These are the two stated components of the classification-to-segmentation approach.
- Object Region Mining with AE: Adversarial erasing alternates classification learning with erasing discovered discriminative regions to progressively expand object localization.The classification network is repeatedly trained and then used to identify regions for erasure and further mining.
- Segmentation Mask Generation: The mined foreground regions are combined with saliency-derived background regions to form segmentation masks while ignoring conflicting, low-saliency, and unlabeled pixels.The resulting masks provide initial supervision for segmentation training.
- Online PSL for Semantic Segmentation: Online PSL weights segmentation score maps by image-level classification confidences, suppressing low-confidence categories and enhancing dominant categories.The weighted maps produce an additional segmentation mask for online training alongside the mined mask.
- Online PSL for Semantic Segmentation: Online training progressively improves the segmentation network, enabling increasingly accurate auxiliary masks for later supervision.At testing time, classification confidences below a threshold are set to zero before weighting the predicted segmentation maps.
4. Experiments
Experiments on PASCAL VOC 2012 evaluate adversarial erasing, online prohibitive segmentation learning, and their combination for weakly supervised semantic segmentation. The approach improves segmentation supervision by progressively mining object regions, while excessive erasing can reduce performance.
- Experimental Setup: The evaluation uses PASCAL VOC 2012 train, validation, and test subsets with image-level labels and pixel IoU averaged over 21 categories.Training is augmented to 10,582 images; experiments use 1,464 train, 1,449 validation, and 1,456 test images before augmentation.
- Object Region Mining with AE: Adding mined regions from successive AE steps generally increases mIoU because the segmentation network receives denser supervision.Regions from the first three AE steps are used for final masks; after four steps, performance drops by 2.1% because over erasing introduces detrimental contextual regions.
- Online PSL for Semantic Segmentation: Online PSL improves performance by 3.2% over training without PSL by providing additional segmentation supervision.PSL uses classification confidences to modulate category-specific response maps and suppress low-confidence maps, reducing noise in the supervision mask.
- Online PSL for Semantic Segmentation: An additional PSL++ training round raises performance from 54.1% to 55.0%, while using ground-truth labels for testing reaches 56.1%.The extra gain is attributed to training on CRF-refined predicted masks; further rounds provide no additional improvement.
5. Conclusion
The paper presents adversarial erasing to progressively discover and expand discriminative object regions for weakly supervised semantic segmentation, alongside online prohibitive segmentation learning for auxiliary information.
- Adversarial erasing progressively discovers and expands object discriminative regions for weakly supervised semantic segmentation.
- The discovered regions provide pixel-level supervision for training the segmentation network.
- Online prohibitive segmentation learning mines auxiliary information to improve adversarial erasing.
- The authors identify adversarial erasing as a direction for weakly supervised semantic segmentation and propose future improvements to its erasing strategy and integration with PSL.