Source-linked AI summary
Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation
Yazhou Yao, Tao Chen, Guosen Xie, Chuanyi Zhang, Fumin Shen, Qi Wu, Zhenmin Tang, Jian Zhang
TL;DR
Weakly supervised semantic segmentation seeks to reduce the intensive cost of pixel-wise annotation, but existing pseudo-label expansion mainly covers salient regions. The paper mines non-salient objects using global reasoning, potential object mining, and masking, achieving state-of-the-art results on PASCAL VOC.
Problem
Image-level weak supervision reduces annotation burden, but existing CAM-based methods produce incomplete pseudo-labels concentrated in salient regions.
Method
The approach combines graph-based global reasoning, potential object mining, and non-salient region masking to discover objects outside salient regions.
Results
The method achieves state-of-the-art results on the PASCAL VOC dataset compared with current methods.
Takeaways & Limitations
Reducing pseudo-label false negatives and mining non-salient objects improves the weakly supervised segmentation framework’s ability to discover missed objects.
Abstract
from arXiv · showhide
Semantic segmentation aims to classify every pixel of an input image. Considering the difficulty of acquiring dense labels, researchers have recently been resorting to weak labels to alleviate the annotation burden of segmentation. However, existing works mainly concentrate on expanding the seed of pseudo labels within the image's salient region. In this work, we propose a non-salient region object mining approach for weakly supervised semantic segmentation. We introduce a graph-based global reasoning unit to strengthen the classification network's ability to capture global relations among disjoint and distant regions. This helps the network activate the object features outside the salient area. To further mine the non-salient region objects, we propose to exert the segmentation network's self-correction ability. Specifically, a potential object mining module is proposed to reduce the false-negative rate in pseudo labels. Moreover, we propose a non-salient region masking module for complex images to generate masked pseudo labels. Our non-salient region masking module helps further discover the objects in the non-salient region. Extensive experiments on the PASCAL VOC dataset demonstrate state-of-the-art results compared to current methods.
1. Introduction
Weakly supervised semantic segmentation reduces the burden of pixel-wise annotation but remains limited by pseudo-labels concentrated in salient regions. This paper mines objects in non-salient regions through global reasoning, potential object mining, and masking.
- Motivation: Image-level labels reduce annotation effort, but weakly supervised semantic segmentation remains challenging.Pixel-wise annotation is labor-intensive and time-consuming.
- Limitations of existing methods: CAM-based methods generate sparse, incomplete pseudo-labels that mainly identify discriminative object parts.Existing approaches expand activated regions, but mainly within salient areas.
- Proposed approach: A graph-based global reasoning unit captures relations among disjoint and distant regions to activate object features outside salient areas.This targets objects scattered near image corners or edges, where local CNN relations are insufficient.
- Proposed approach: Potential object mining discovers objects activated by naive CAMs outside conspicuous regions and reduces pseudo-label false negatives.The module uses the segmentation network’s self-correction ability to improve pseudo-label quality.
- Proposed approach: Non-salient region masking generates masked pseudo-labels for complex images, helping discover additional objects outside salient regions.The module is designed for images containing two or more object categories.
2. Related Work
Semantic segmentation assigns labels to every pixel, while weakly supervised segmentation seeks to reduce annotation requirements using weaker labels. Related approaches use architectural context modeling and CAM-derived seeds, but the proposed framework targets non-salient object discovery.
- 2.1. Semantic Segmentation: Semantic segmentation assigns a semantic label to every pixel and uses architectures that recover resolution and model broader context.Related methods include encoder-decoder designs, dilated convolution, pyramid pooling, context encoding, and architecture search.
- 2.2. Weakly Supervised Semantic Segmentation: Weakly supervised semantic segmentation uses weaker annotations to alleviate pixel-wise labeling burdens, with image-level labels widely used because they are easy to obtain.Image-level labels are also available in existing large-scale datasets or can be automatically generated.
- 2.2. Weakly Supervised Semantic Segmentation: CAM-based weakly supervised methods generate pixel-level seeds and expand them toward complete object regions.The literature includes seed, expand, and constrain strategies and online attention accumulation.
3. The Proposed Approach
The framework mines object regions beyond salient areas by combining global reasoning, potential-object mining, and masking-based pseudo-label refinement.
- 3.1. CAM Generation: A graph-based global reasoning unit captures relations among disjoint, distant regions before classification, activating object features outside salient areas.Encoder features are projected into a latent interaction space, processed by graph convolution, and reverse-projected to the original space.
- 3.1. CAM Generation: CAMs and OA-CAMs provide complementary pseudo-label cues: CAMs have high precision but low recall, whereas OA-CAMs cover more object pixels with lower precision.OA-CAMs strengthen lower attention values using an integral attention model, while CAMs supply precise potential-object evidence.
- 3.2. Potential Object Mining: Potential object mining marks high-attention background pixels as ignored rather than assigning potential classes, reducing false negatives without requiring boundaries outside salient regions.Thresholds are selected using the median when class c appears in the initial label, otherwise the top quartile of CAM attention values above Tbg.
- 3.2. Potential Object Mining: The segmentation network’s predictions are used to generate higher-quality pseudo labels for retraining and self-correction.The enriched pseudo labels contain more ignored pixels, allowing the segmentation network to predict labels for potential object regions during training.
- 3.3. Non-Salient Region Masking: Non-salient region masking handles complex images by expanding predicted object regions, extracting and dilating masks, then masking the expanded prediction map.The module assumes object labels within salient regions are correct with high probability; dilation introduces a small portion of surrounding background.
4. Experiments
Experiments on PASCAL VOC 2012 evaluate the method across backbones, components, qualitative examples, and parameter settings. The results support global reasoning, potential object mining, and selective non-salient masking for discovering objects beyond salient regions.
- Experimental Results: The method achieves 65.5% validation and 65.3% test mIoU with VGG, and 68.3% validation and 68.5% test mIoU with ResNet.With COCO-pretrained ResNet weights, performance reaches 70.4% validation and 70.2% test mIoU.
- Experimental Results: The approach outperforms methods using additional training data and pseudo-label refinement, including STC, WebS-i2, Hong et al., DSRG, and CONTA.Reported validation improvements are 15.7%, 12.1%, 7.4%, 6.9%, and 2.2%, respectively.
- Element-Wise Component Analysis: Global reasoning improves the segmentation result from 67.7% to 68.8%, while potential object mining adds 0.2% and retraining reaches 69.7%.The component analysis attributes these gains to capturing disjoint and distant relations and reducing false negatives in pseudo labels.
- Ablation Studies for NSRM: Non-salient region masking further improves the result to 70.4% by exploiting objects outside conspicuous regions.Applying NSRM to all images instead of dividing simple and complex images reduces performance from 70.4% to 68.8%.
- Qualitative Results: Qualitative examples show global reasoning discovering distant objects, POM finding objects outside salient regions, and the full method predicting additional buses, plants, and cars.Figure 4 compares baseline, GR, GR + POM, and the full method across single-category and multi-category images.
- Parameter Analysis: The best NSRM performance occurs with dilation kernel sizes between 5 and 30, and the experiments set r = 30; the interaction-space node count is set to N = 64.Too-large or too-small dilation kernels and excessively large node counts provide limited improvement.
5. Conclusions
The paper proposes mining objects in non-salient regions for weakly supervised semantic segmentation through global reasoning, pseudo-label correction, and masking. Experiments on PASCAL VOC 2012 demonstrate the approach’s superiority.
- 5. Conclusions: The method combines graph-based global reasoning, potential object mining, and non-salient region masking to discover objects beyond salient areas.Global reasoning captures disjoint and distant relations, POM reduces pseudo-label false negatives, and NSRM generates masked pseudo labels for complex images.