Source-linked AI summary
Weakly-Supervised Semantic Segmentation via Sub-category Exploration
Yu-Ting Chang, Qiaosong Wang, Wei-Chih Hung, Robinson Piramuthu, Yi-Hsuan Tsai, Ming-Hsuan Yang
TL;DR
Weakly-supervised segmentation often starts from CAM response maps that cover only discriminative object parts. The paper addresses this with iterative self-supervised sub-category exploration, reporting improved response maps and favorable segmentation performance against existing methods.
Problem
CAM-based classification can produce incomplete response maps because identifying discriminative object parts is sufficient for optimizing classification.
Method
The method clusters image features within each parent class, assigns sub-category pseudo-labels, and jointly trains parent and sub-category classifiers iteratively.
Results
The approach improves initial response-map quality and performs favorably against existing weakly-supervised semantic segmentation methods.
Takeaways & Limitations
Using more than one sub-category produces better response maps, with iterative training progressively improving mIoU and F-Score over original CAM.
Takeaways & Limitations
The method assumes annotated parent classes for defining and clustering sub-categories.
Abstract
from arXiv · showhide
Existing weakly-supervised semantic segmentation methods using image-level annotations typically rely on initial responses to locate object regions. However, such response maps generated by the classification network usually focus on discriminative object parts, due to the fact that the network does not need the entire object for optimizing the objective function. To enforce the network to pay attention to other parts of an object, we propose a simple yet effective approach that introduces a self-supervised task by exploiting the sub-category information. Specifically, we perform clustering on image features to generate pseudo sub-categories labels within each annotated parent class, and construct a sub-category objective to assign the network to a more challenging task. By iteratively clustering image features, the training process does not limit itself to the most discriminative object parts, hence improving the quality of the response maps. We conduct extensive analysis to validate the proposed method and show that our approach performs favorably against the state-of-the-art approaches.
1. Introduction
Weakly-supervised segmentation methods often begin with CAM-based response maps, but these maps may highlight only discriminative object parts. The paper introduces iterative self-supervised sub-category exploration to improve response maps and final segmentation.
- Motivation: Image-level weak supervision avoids pixel-wise annotation but requires an initial response map before refinement and segmentation training.Existing pipelines predict an initial category-wise response, refine it into pseudo ground truth, and train a segmentation network.
- Motivation: CAM response maps may localize only discriminative object parts because classification does not require observing the entire object.This produces incomplete object localization and impairs the classifier’s ability to locate objects.
- Approach: The method discovers sub-categories by clustering image features within each annotated parent class and uses the assignments as pseudo-labels for a sub-category objective.The parent classification objective and the self-supervised sub-category objective are jointly optimized.
- Results: Extensive experiments and analyses on PASCAL VOC 2012 show improved response-map quality and favorable final semantic segmentation results.The study also analyzes sub-category behavior with respect to object size/type, context, and coexistence.
- Approach: Iterative clustering and pseudo-training make the classification task more challenging, enhancing feature representations and improving initial class activation maps.The approach alternates feature clustering and pseudo-training rather than limiting learning to the parent-class objective.
2. Related Work
Prior work addresses incomplete CAM cues through erasing, attention aggregation, and response refinement, while this paper uses self-supervised sub-category exploration to improve the initial prediction directly.
- Initial Prediction for WSSS: CAM is widely used to generate initial localization cues, but classification-driven activation often produces incomplete masks focused on small discriminative object regions.Initial seeds are important because they provide priors for generating segmentation maps.
- Initial Prediction for WSSS: Erasing-based methods seek diverse object parts, but some use fixed-size random patches or require repetitive training and response aggregation.These methods mitigate incomplete localization by deliberately hiding or erasing object regions.
- Initial Prediction for WSSS: The proposed method differs by imposing a harder self-supervised sub-category task to enhance feature representations and response maps.It targets the classification network’s learning objective rather than discovering complementary regions through iterative erasing or consolidating attention maps.
- Response Refinement for WSSS: Response-refinement methods expand initial cues using losses, alternating region expansion, multi-dilation branches, or seeded region growing.These approaches operate after initial localization to generate pseudo ground truths or train the segmentation network.
- Response Refinement for WSSS: Because refinement begins from CAM seeds, incomplete or incorrect initial regions can remain difficult to correct or may spread inaccurate attention.This motivates focusing on improving initial prediction before refinement.
- Unsupervised Representation Learning: The framework relates to unsupervised representation learning by deriving pseudo-labels from raw-input features without requiring human annotations for the sub-category task.K-means is described as a scalable unsupervised feature-learning module.
3. Weakly-supervised Semantic Segmentation
The framework improves weakly supervised segmentation by adding self-supervised sub-category discovery to the classification network, targeting incomplete CAM response maps. It clusters features within parent classes, jointly trains parent and sub-category objectives, and iteratively updates features and pseudo-labels.
- Algorithm Overview: The method introduces self-supervised sub-category discovery as a harder task to make the classification network learn from more object parts.The sub-category task is integrated without replacing the original parent-classification objective.
- Algorithm Overview: CAM-based initial response maps often highlight only discriminative object parts because classification does not require seeing the entire object.This incomplete localization motivates improving the initial response rather than only refining pseudo-labels or training the segmentation network.
- Sub-category Exploration: For each parent class, K-means clusters extracted image features into K sub-categories, whose cluster indices become pseudo-labels for the sub-category objective.Images lacking a parent-class label receive zero labels for all corresponding sub-categories.
- Sub-category Exploration: The parent and sub-category classifiers share the feature extractor and are jointly optimized with a weighted combination of their classification losses.Parent supervision guides the feature space, while the sub-category objective explores a feature sub-space and supplies additional gradients used for CAM computation.
- Sub-category Exploration: Iterative optimization alternates feature clustering with network training so enhanced features can improve subsequent sub-category assignments and response maps.Algorithm 1 cycles through feature extraction, pseudo-label generation, joint optimization, and CAM computation.
4. Experimental Results
Experiments on PASCAL VOC 2012 show that sub-category exploration improves initial response maps and yields favorable final segmentation performance. Ablations indicate robustness across sub-category counts, iterative gains, and interpretable clustering behavior.
- Main Results: On PASCAL VOC 2012, the method improves initial response quality and produces favorable final semantic segmentation results against state-of-the-art approaches.The evaluation uses image-level labels and compares both initial responses and downstream segmentation performance.
- Improvement on Initial Response: The method generates more complete object regions, whereas conventional CAM tends to focus on small discriminative parts.The broader localization is important because refinement uses the response map as input.
- Ablation Study and Analysis: The proposed method consistently outperforms original CAM across K = {5, 10, 20, 50} and remains robust over this range.K = 10 is selected for the experiments as a balance between efficiency and accuracy.
- Ablation Study and Analysis: Both mIoU and F-Score gradually improve over successive training rounds compared with round #0, which corresponds to original CAM.F-Score accounts for recall and precision and helps assess whether activation maps cover object parts.
- Ablation Study and Analysis: Clustering organizes objects by size, context, type, pose, and interactions with other categories, indicating enhanced feature representations.Examples include persons grouped by co-occurring horse, motorcycle, or boat categories.
- Semantic Segmentation Performance: Without CRF, the method reaches mIoU 64.8%, similar to FickleNet at mIoU 64.9%; with CRF, 11/20 classes improve.Qualitative results are reported as close to ground-truth segmentation.
5. Conclusions
The paper concludes that self-supervised sub-category discovery improves class activation maps and final weakly-supervised segmentation without extra inference complexity. Iterative clustering and sub-category objectives provide the basis for this improvement.
- 5. Conclusions: The method improves class activation maps by discovering sub-categories through a self-supervised task.The approach performs clustering on image features for each parent class and trains with sub-category objectives.
- 5. Conclusions: The resulting better activation maps improve final semantic segmentation performance without introducing extra complexity or inference time.The conclusion contrasts this approach with schemes that aggregate multiple response maps.