Source-linked AI summary
Self-supervised Image-specific Prototype Exploration for Weakly Supervised Semantic Segmentation
Qi Chen, Lingxiao Yang, Jianhuang Lai, Xiaohua Xie
TL;DR
Image-level WSSS reduces annotation costs, but CAM often produces incomplete localization by focusing on discriminative regions. SIPE explores image-specific prototypes with IPE and regularizes general CAM against IS-CAM using GSC, achieving state-of-the-art performance on two benchmarks with image-level labels.
Problem
CAM-based WSSS localization focuses on discriminative regions and ignores other useful image information, resulting in incomplete localization maps.
Method
SIPE combines two-step Image-specific Prototype Exploration for IS-CAM with General-Specific Consistency between general CAM and IS-CAM.
Results
SIPE achieves new state-of-the-art performance on PASCAL VOC 2012 and MS COCO 2014 using only image-level labels.
Takeaways & Limitations
Image-specific prototypes provide more complete localization maps, while GSC further optimizes feature representation and supports self-correction in prototype exploration.
Takeaways & Limitations
The prototype representation uses hierarchical features because shallow-layer information is considered more suitable for background-related modeling than semantic features alone.
Abstract
from arXiv · showhide
Weakly Supervised Semantic Segmentation (WSSS) based on image-level labels has attracted much attention due to low annotation costs. Existing methods often rely on Class Activation Mapping (CAM) that measures the correlation between image pixels and classifier weight. However, the classifier focuses only on the discriminative regions while ignoring other useful information in each image, resulting in incomplete localization maps. To address this issue, we propose a Self-supervised Image-specific Prototype Exploration (SIPE) that consists of an Image-specific Prototype Exploration (IPE) and a General-Specific Consistency (GSC) loss. Specifically, IPE tailors prototypes for every image to capture complete regions, formed our Image-Specific CAM (IS-CAM), which is realized by two sequential steps. In addition, GSC is proposed to construct the consistency of general CAM and our specific IS-CAM, which further optimizes the feature representation and empowers a self-correction ability of prototype exploration. Extensive experiments are conducted on PASCAL VOC 2012 and MS COCO 2014 segmentation benchmark and results show our SIPE achieves new state-of-the-art performance using only image-level labels. The code is available at https://github.com/chenqi1126/SIPE.
1. Introduction
Image-level labels reduce annotation costs for WSSS, but CAM-based localization often misses useful object regions. SIPE addresses this by exploring image-specific prototypes and enforcing consistency between general CAM and IS-CAM.
- Motivation: Image-level labels provide a less costly alternative to pixel-level annotations for weakly supervised semantic segmentation.Other weak labels include bounding boxes, scribbles, and points.
- Motivation: CAM uses learned class centers to correlate with image pixels and generate localization maps.This class-center representation is treated as a general representation of each class.
- Motivation: CAM concentrates activation on primary regions while ignoring distant cues, producing incomplete localization maps.The paper illustrates this issue with cat images, where the head is emphasized and the body is overlooked.
- Proposed Framework: SIPE tailors image-specific prototypes through Image-specific Prototype Exploration to capture more complete localization regions.IPE uses structure-aware seed locating and background-aware prototype modeling.
- Proposed Framework: General-Specific Consistency regularizes general CAM and IS-CAM, optimizing feature representation and supporting self-correction during prototype exploration.The consistency loss is part of the proposed SIPE framework.
2. Related Work
Related WSSS methods commonly refine CAM-derived localization maps, while SIPE instead uses image-specific prototypes to account for image distinctiveness and discover more complete regions.
- Image-level WSSS: Image-level WSSS has attracted attention because it requires lower annotation costs than fully supervised segmentation.Existing methods generate localization maps with CAM and refine them as pseudo labels.
- Prior Approaches: Erasure methods seek additional object regions by removing discriminative regions from images or feature maps.The passage notes that excessive erasure may confuse the classifier.
- SIPE: SIPE considers image distinctiveness by introducing image-specific prototypes to discover complete regions and construct self-supervision for feature representation.This contrasts with approaches centered on generic class representations.
3. Approach
SIPE combines image-specific prototype exploration with general-specific consistency to improve weakly supervised localization. It locates structure-aware seeds, models hierarchical foreground and background prototypes, forms IS-CAM, and regularizes it against CAM.
- SIPE combines Image-specific Prototype Exploration (IPE) with General-Specific Consistency (GSC) to improve weakly supervised semantic segmentation.IPE explores image-specific prototypes, while GSC regularizes original CAM and IS-CAM.
- Structure-aware Seed Locating: Inter-pixel semantic correlations define spatial structure, and each pixel is assigned the class with maximal class-wise structure similarity.The structure similarity is evaluated by class-wise IoU between a pixel’s structure map and CAM.
- Structure-aware Seed Locating: The resulting process locates seed regions for both foreground and background classes in parallel.Background pixels unassociated with foreground pixels are more likely assigned to the background class.
- Background-aware Prototype Modeling: Hierarchical features fuse four backbone stages, providing a feature space for centroid-based foreground and background prototypes.Shallow-layer features contribute low-level visual information suited to modeling background-related information.
- Image-Specific CAM: IS-CAM correlates pixels with prototypes tailored to each image, producing more complete object regions than CAM’s classifier-weight centers.Background prototype modeling supplies background localization cues that help determine foreground regions.
- General-Specific Consistency: GSC minimizes the difference between CAM activated by classifier weights and IS-CAM activated by image-specific prototypes using L1-normalized consistency.The loss is averaged over K foreground classes and one background class, alongside multi-label classification loss.
4. Experiments
Experiments evaluate SIPE on PASCAL VOC 2012 and MS COCO 2014 using image-level labels, measuring localization and segmentation quality with mIoU. SIPE improves localization maps, segmentation results, and key components through image-specific prototypes, GSC, and background-aware modeling.
- Experimental Settings: Experiments use PASCAL VOC 2012 and MS COCO 2014, with only image-level classification labels available during training and mIoU as the evaluation metric.PASCAL VOC includes 20 foreground classes and one background class; MS COCO contains 81 classes.
- Localization Maps: 58.6% mIoU is achieved by SIPE on PASCAL VOC localization maps, increasing to 64.7% with denseCRF post-processing.The method is reported to outperform the compared localization approaches and capture complete foreground and background regions.
- Segmentation Results: 68.8% mIoU on the PASCAL VOC validation set and 69.7% on the test set are achieved using only image-level labels.With a ResNet38 backbone, SIPE reaches 69.5% mIoU on the test set and is reported to exceed existing methods using that backbone.
- Segmentation Results: 43.6% mIoU is achieved on the MS COCO 2014 validation set with ResNet38, 7.2% higher than the previous SOTA CSE.Using ResNet101, SIPE exceeds EPS by 4.9%.
- Ablation Studies: GSC improves IS-CAM quality by 5.4%, while adding image-specific prototypes improves original CAM by 2.1% mIoU.The ablation study evaluates IPE and GSC as the two main contributions.
- Ablation Studies: Structure-aware seed locating outperforms fixed- and adaptive-threshold CAM, whose threshold variants reach at most 53.3% mIoU.Argmax on CAM produces only a slight gain of +0.4%.
- Ablation Studies: Hierarchical features with Background Prototype Modeling achieve the highest performance among the tested prototype options, including estimated-based pseudo labels.Semantic features with BPM leave mIoU basically unchanged, while hierarchical features without BPM reduce mIoU by over 2%.
5. Conclusion
The paper proposes SIPE for weakly supervised semantic segmentation with image-level labels. IPE generates image-specific localization maps through structure-aware seed locating and background-aware prototype modeling, while GSC regularizes general CAM and IS-CAM; experiments report state-of-the-art performance on two benchmarks.
- 5. Conclusion: SIPE combines Image-specific Prototype Exploration and General-Specific Consistency for weakly supervised semantic segmentation.The framework is designed for training with image-level labels.
- 5. Conclusion: IPE uses structure-aware seed locating and background-aware prototype modeling to produce more favorable localization maps.These components support image-specific prototype exploration.
- 5. Conclusion: GSC regularizes general CAM and Image-Specific CAM to empower feature representation.The consistency loss is part of SIPE’s self-supervised training framework.
- 5. Conclusion: Extensive experiments report new state-of-the-art performance on PASCAL VOC 2012 and MS COCO 2014 using image-level labels.The conclusion attributes the reported results to the proposed SIPE framework.