Source-linked AI summary
Cross Language Image Matching for Weakly Supervised Semantic Segmentation
Jinheng Xie, Xianxu Hou, Kai Ye, Linlin Shen
TL;DR
Weakly supervised semantic segmentation relies on fixed image-level labels, leaving conventional CAMs prone to incomplete object activation and closely related background activation. CLIMS adds CLIP-based natural-language supervision and specialized losses to improve CAMs, and it significantly outperforms prior state-of-the-art methods on PASCAL VOC2012.
Problem
Fixed image-level category supervision leaves conventional CAMs activating discriminative parts and closely related backgrounds, limiting initial CAM quality.
Method
CLIMS uses a CLIP-based text-driven evaluator with object, background, matching, co-occurring-background suppression, and area-regularization components for CAM learning.
Results
CLIMS significantly outperforms previous state-of-the-art methods on PASCAL VOC2012.
Takeaways & Limitations
CLIMS generates more complete and compact initial CAMs and refined pseudo-ground-truth masks than baseline and state-of-the-art methods.
Takeaways & Limitations
The object-region and text-label matching loss alone cannot encourage nondiscriminative object regions or suppress background regions activated in CAMs.
Abstract
from arXiv · showhide
It has been widely known that CAM (Class Activation Map) usually only activates discriminative object regions and falsely includes lots of object-related backgrounds. As only a fixed set of image-level object labels are available to the WSSS (weakly supervised semantic segmentation) model, it could be very difficult to suppress those diverse background regions consisting of open set objects. In this paper, we propose a novel Cross Language Image Matching (CLIMS) framework, based on the recently introduced Contrastive Language-Image Pre-training (CLIP) model, for WSSS. The core idea of our framework is to introduce natural language supervision to activate more complete object regions and suppress closely-related open background regions. In particular, we design object, background region and text label matching losses to guide the model to excite more reasonable object regions for CAM of each category. In addition, we design a co-occurring background suppression loss to prevent the model from activating closely-related background regions, with a predefined set of class-related background text descriptions. These designs enable the proposed CLIMS to generate a more complete and compact activation map for the target objects. Extensive experiments on PASCAL VOC2012 dataset show that our CLIMS significantly outperforms the previous state-of-the-art methods.
1. Introduction
CLIMS addresses incomplete object activation and irrelevant-background activation in weakly supervised semantic segmentation by adding natural-language supervision to CAM learning. Its losses and regularization produce more complete, compact initial CAMs, and experiments report gains over prior methods.
- Motivation: WSSS uses image-level labels to avoid the costly pixel-level annotations required by fully supervised semantic segmentation.The paper focuses on learning semantic segmentation using only image-level labels.
- Motivation: Conventional CAMs activate closely related backgrounds and underestimate object contents, limiting the quality of pseudo-labels and later segmentation stages.These issues arise because supervision covers only a fixed set of object categories.
- Approach: CLIMS introduces CLIP-based natural-language supervision so activation maps can use an open-world set of visual concepts rather than only predetermined categories.The framework replaces GAP and fully connected layers with convolutional layers that directly generate class activation maps under CLIP supervision.
- Approach: Three CLIP-based losses match object and background regions with text labels and suppress co-occurring backgrounds, while area regularization constrains activated-region size.Together, these components target CAM correctness, completeness, background suppression, and compactness.
- Results: CLIMS significantly outperforms previous state-of-the-art methods on the PASCAL VOC2012 dataset.The introduction reports this result without specifying a numerical metric in the supplied passage.
2. Related Work
Prior WSSS research mainly follows the CAM pipeline, improving initial activation maps through attention strategies, object subcategories, or co-attention. CLIP provides a complementary open-world image-text matching basis because it recognizes more diverse categories than small fixed-label datasets.
- Weakly supervised semantic segmentation: Most WSSS methods follow a three-stage CAM pipeline involving initial activation-map generation, CAM refinement, and segmentation-network training.The related-work discussion organizes prior methods according to these stages.
- Weakly supervised semantic segmentation: Self-erasing strategies, object-subcategory mining, and neural co-attention have been used to discover more complete object regions in initial CAMs.These approaches target missing object parts or shared and unshared semantics between training images.
- Contrastive Language-Image Pre-training: CLIP uses image and text encoders to learn image-text similarity in a zero-shot setting.Its pretraining uses 400 million image-text pairs.
- Contrastive Language-Image Pre-training: Because CLIP recognizes a larger and more diverse category set than small fixed-label datasets such as PASCAL VOC20, it motivates text-driven supervision for WSSS.This broader category coverage supports the paper’s open-world framing.
3. Methodology
CLIMS replaces conventional CAM supervision with a CLIP-based text-driven evaluator that guides initial CAMs toward complete objects while suppressing related backgrounds. Its framework combines object-region matching, background-region matching, co-occurring background suppression, and area regularization.
- Conventional CAM limitations: Conventional CAMs can activate only discriminative object parts and unnecessarily activate closely related backgrounds because supervision is limited.
- Backbone network: CLIMS removes GAP and applies sigmoid activation after convolutional weights to generate class-specific activation maps directly.
- Text-driven evaluator: The text-driven evaluator masks foreground and background regions, encodes them with CLIP, and compares their representations with object and background text labels.
- Object-region matching: Object-region and text-label matching guides initial CAMs toward target objects, but alone cannot explore nondiscriminative object regions or suppress activated backgrounds.
- Background-region matching: Background-region and text-label matching recovers more complete object contents by reducing target-object pixels left in the masked background region.
- Background suppression and regularization: Co-occurring background suppression minimizes similarity to class-related background texts, while area regularization excludes irrelevant background regions from activation maps.
4. Experiments
Experiments on PASCAL VOC2012 evaluate CLIMS under standard WSSS settings, showing improved CAM quality, pseudo-label quality, segmentation performance, and robustness across its loss components.
- Experimental Setup: PASCAL VOC2012 experiments use 20 categories, an augmented 10,582-image training set, and mIoU for evaluation.The dataset contains 1,464 training, 1,449 validation, and 1,456 test images.
- CAM and Pseudo-Label Quality: CLIMS initial CAMs reach 56.6% mIoU, while PSA-refined pseudo labels reach 70.0% mIoU on the PASCAL VOC2012 train set.The refined pseudo labels are 5.3 and 2.0 percentage points higher than PuzzleCAM and AdvCAM, respectively.
- Segmentation Performance: 69.3% and 68.7% mIoU are obtained on the validation and test sets when DeepLabV2 is trained with CLIMS pseudo labels.These results exceed Adv-CAM by 1.2 and 0.7 percentage points on validation and test, respectively.
- Ablation Studies: LCBS increases boat IoU from 7.1% to 58.2% and train IoU from 30.7% to 63.9% on the PASCAL VOC2012 train set.The experiments use predefined class-related backgrounds such as river, sea, lake, railroad, railway, and tree.
- Sensitivity Analysis: Performance remains stable as α varies from 7 to 13, β from 22 to 28, γ from 28 to 31, and δ from 1 to 1.3.The sensitivity analysis is conducted on the PASCAL VOC2012 train set.
5. Conclusion
CLIMS introduces natural-language supervision to improve WSSS by addressing incomplete object activation and closely related background activation. Experiments show more complete and compact CAMs and refined pseudo-ground-truth masks than baseline and state-of-the-art methods.
- CLIMS introduces natural-language supervision for weakly supervised semantic segmentation.
- Its four loss functions address incomplete object contents and unnecessary activation of closely related backgrounds.
- Experiments on PASCAL VOC2012 validate CLIMS’s effectiveness.
- CLIMS generates more complete and compact initial CAMs and refined pseudo-ground-truth masks than baseline and state-of-the-art methods.