Source-linked AI summary
CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation
Yuqi Lin, Minghao Chen, Wenxiao Wang, Boxi Wu, Ke Li, Binbin Lin, Haifeng Liu, Xiaofei He
TL;DR
Image-level WSSS is costly because conventional methods use multi-stage pipelines, motivating a simpler approach. CLIP-ES directly exploits frozen CLIP with redesigned CAM generation, refinement, and training, achieving state-of-the-art results on PASCAL VOC 2012 and COCO 2014 while generating pseudo masks 10x more efficiently.
Problem
Image-level WSSS must simplify its costly multi-stage training pipeline while maintaining high performance.
Method
CLIP-ES uses frozen ViT-based CLIP, softmax-GradCAM with text-driven strategies, CAA refinement, and confidence-guided loss for pseudo-mask training.
Results
CLIP-ES achieves state-of-the-art performance on PASCAL VOC 2012 and COCO 2014 and generates pseudo masks 10x more efficiently than other methods.
Takeaways & Limitations
CLIP can localize categories from image-level labels without further training and can generate segmentation masks for new classes.
Takeaways & Limitations
The method uses ViT-based CLIP because the CNN-based model fails to explore global context and suffers from the discriminative-part domain.
Abstract
from arXiv · showhide
Weakly supervised semantic segmentation (WSSS) with image-level labels is a challenging task. Mainstream approaches follow a multi-stage framework and suffer from high training costs. In this paper, we explore the potential of Contrastive Language-Image Pre-training models (CLIP) to localize different categories with only image-level labels and without further training. To efficiently generate high-quality segmentation masks from CLIP, we propose a novel WSSS framework called CLIP-ES. Our framework improves all three stages of WSSS with special designs for CLIP: 1) We introduce the softmax function into GradCAM and exploit the zero-shot ability of CLIP to suppress the confusion caused by non-target classes and backgrounds. Meanwhile, to take full advantage of CLIP, we re-explore text inputs under the WSSS setting and customize two text-driven strategies: sharpness-based prompt selection and synonym fusion. 2) To simplify the stage of CAM refinement, we propose a real-time class-aware attention-based affinity (CAA) module based on the inherent multi-head self-attention (MHSA) in CLIP-ViTs. 3) When training the final segmentation model with the masks generated by CLIP, we introduced a confidence-guided loss (CGL) focus on confident regions. Our CLIP-ES achieves SOTA performance on Pascal VOC 2012 and MS COCO 2014 while only taking 10% time of previous methods for the pseudo mask generation. Code is available at https://github.com/linyq2117/CLIP-ES.
1. Introduction
CLIP-ES addresses the high cost and complexity of image-level WSSS by using frozen CLIP to generate and refine pseudo masks without further training. It redesigns CAM generation, refinement, and segmentation training while achieving state-of-the-art performance and faster pseudo-mask generation.
- Problem: Image-level WSSS reduces annotation costs but commonly requires multiple trained models across CAM generation, refinement, and segmentation stages.Previous methods train a dataset-specific classification model and affinity network before training the final segmentation model.
- CAM generation: Softmax-GradCAM and a class-related background set make categories mutually exclusive and reduce confusion among target, non-target, and background categories.The method addresses redundant CAMs caused by multi-label classification and incomplete localization caused by the classification-localization gap.
- CAM refinement: CAA uses CLIP-ViT’s intrinsic multi-head self-attention to refine class-wise CAMs in real time instead of training a separate affinity network.CAA bridges class-agnostic attention maps and class-wise CAMs and can be integrated into the first stage.
- Segmentation training: CGL ignores noisy positions in pseudo masks when training the final segmentation model.The loss focuses training on confident regions rather than treating all noisy pseudo-mask locations as ground truth.
- Framework: CLIP-ES uses frozen CLIP to localize objects from image-level labels without further training.The framework exploits CLIP’s zero-shot and text-image capabilities while retaining flexibility across new classes and datasets.
- Results: 10x efficiency is reported for pseudo-mask generation while the framework achieves state-of-the-art performance.The reported efficiency comparison concerns pseudo-mask generation relative to other methods.
2. Related Work
Prior WSSS methods improve incomplete or redundant CAMs through auxiliary training, erasing, affinity learning, saliency maps, and other refinements. CLIP-ES instead directly uses CLIP to generate CAMs, making the process simpler and more efficient.
- WSSS methods: Traditional WSSS methods generate incomplete or redundant CAMs and refine them across multiple stages.The standard pipeline includes initial CAM generation, affinity or saliency-based refinement, and final segmentation training.
- Initial CAM generation: Initial CAM completeness has been improved using auxiliary tasks, erasing strategies, accumulated activations, cross-image mining, and self-supervised mechanisms.These approaches add training objectives or manipulate images and feature maps to discover more object regions.
- CAM refinement: CAM refinement commonly learns pairwise semantic affinity or uses saliency maps to propagate semantics and distinguish objects or background.Examples include pixel-affinity networks, boundary synthesis, and additional saliency estimation.
- CLIP-based WSSS: CLIP has been introduced to WSSS for activating object regions and suppressing background, but CLIMS still uses another CNN to generate CAMs.CLIP-ES directly uses CLIP for CAM generation and explores text-object relationships under the WSSS setting.
3. Method
CLIP-ES adapts GradCAM and CLIP text inputs for more discriminative CAMs, then uses class-aware attention to refine them efficiently. Its prompt strategies exploit score sharpness and synonyms for weakly supervised segmentation.
- Softmax-GradCAM: Softmax-GradCAM makes CLIP class scores mutually exclusive, reducing confusion between target, non-target foreground, and background categories.The resulting gradients include both target-class and competing-class terms, suppressing non-target feature contributions.
- Softmax-GradCAM: Class-related background suppression adds common background categories to CLIP text inputs, using zero-shot classification without retraining.This expands competition beyond dataset-defined categories and suppresses pixels associated with class-related backgrounds.
- Sharpness-based Prompt Selection: Sharpness-based prompt selection uses score dispersion across classes to choose prompts associated with stronger segmentation performance.On PASCAL VOC 2012, sharpness was roughly negatively correlated with mIoU across 20 randomly selected prompts; the adopted prompt was “a clean origami {}.”
- Text-driven Strategies: Synonym fusion enriches category semantics by placing multiple synonyms in one sentence, disambiguating polysemous words with one forward pass.The paper merges synonyms at the sentence level rather than using separate passes or feature-level and CAM-level alternatives.
- Class-aware Attention-based Affinity: CAA refines class-wise CAMs with class-aware masks derived from CLIP-ViT attention, avoiding a separately trained affinity network.Sinkhorn-normalized attention is symmetrized, and CAM-derived bounding boxes mask affinity propagation; attention and CAM are extracted in one forward pass.
4. Experiments
Experiments show that CLIP-ES generates strong pseudo masks efficiently and achieves state-of-the-art segmentation performance on PASCAL VOC 2012 and MS COCO 2014. Ablations attribute gains to softmax-based CAM generation, CAA refinement, confidence-guided loss, and synonym fusion.
- Quality of Generated CAMs: 75.0% mIoU after dense CRF post-processing makes the generated CAMs accurate enough to omit a separately trained affinity network.
- Time and Memory Efficiency: More than 10x efficiency in time and memory results from avoiding dataset-specific classification and affinity-network training.CAA is integrated into the initial CAM stage, while single-scale inference preserves competitive inference speed.
- Segmentation Performance: 73.8% and 73.9% mIoU on the PASCAL VOC 2012 validation and test sets establish new state-of-the-art results.On MS COCO 2014, CLIP-ES achieves 45.4% mIoU on the validation set.
- Segmentation Performance: 45.4% mIoU on the MS COCO 2014 validation set gives CLIP-ES the best reported performance in that evaluation.
- Ablation Study: Softmax-based GradCAM raises performance from 49.4% to 53.3% for the evaluated VOC categories.The class-related background set further improves mIoU by 22.8% for boat, 13.7% for train, and 9.2% overall.
- Ablation Study: CAA improves vanilla MHSA refinement by introducing class-aware masks, while CGL boosts segmentation by emphasizing confident regions in noisy pseudo masks.Synonym fusion also improves class-level results, especially for person.
5. Conclusion
CLIP-ES is a text-driven WSSS framework that improves pseudo-mask quality and reduces training cost without further training. It achieves state-of-the-art performance and may generate masks for new classes.
- CLIP-ES improves each WSSS stage through strategies designed for CLIP.The framework targets CAM generation, CAM refinement, and final segmentation-mask training.
- CLIP-ES efficiently generates pseudo masks without further training.
- The framework achieves state-of-the-art performance on PASCAL VOC 2012 and COCO 2014 and may generate masks for new classes.
Appendix
The appendix qualitatively compares CNN and ViT architectures and clstoken and avgtoken variants for WSSS.
- Figure S1 compares CNN and ViT architectures for WSSS.
- Figure S1 compares clstoken and avgtoken variants for WSSS.
- The comparison is qualitative rather than a reported numerical evaluation.
A. More Analysis about GradCAM-CLIP
The analysis favors ViT-based CLIP and average-token pooling for GradCAM localization, while synonym-fusion strategies produce similar results.
- ViT-based CLIP produces more complete object CAMs than the ResNet-based model.The ResNet-based model is reported to suffer heavily from the discriminative-part domain problem.
- Average pooling over remaining tokens localizes objects more completely and accurately than using the class token.
- Table S1 quantitatively compares CNN and ViT architectures together with clstoken and avgtoken for WSSS.
- Table S2 compares synonym-fusion strategies on the PASCAL VOC 2012 training set.
- Synonym fusion at sentence, feature, and CAM levels yields similar results with only slight variation across categories.
C. Hyper-parameter Selection for λ
The λ analysis evaluates CAM-threshold choices across PASCAL VOC 2012 and COCO 2014, finding dataset-dependent best thresholds and selecting separate values.
- The study varies λ from 0 to 0.8 in increments of 0.1 to evaluate CAM quality.The evaluation uses PASCAL VOC 2012 and the first 2,000 images from the COCO 2014 training set.
- The best λ threshold varies across datasets.
- Figure S2 reports the effect of λ on generated CAM quality for PASCAL VOC 2012 and part of the COCO 2014 training set.
- The selected λ values are 0.4 for VOC and 0.7 for COCO.
D. Hyper-parameter Selection for µ in CGL
The experiments set µ to 0.95 for CGL because dense CRF postprocessing makes most pixels sufficiently confident, while µ has little effect on segmentation performance.
- µ is set to 0.95 because only a small minority of pixels have confidence below 0.95 after dense CRF postprocessing.These lower-confidence pixels are mainly near object boundaries.
- Changing µ does not remarkably affect segmentation performance.
- VOC’s original ignored percentage is about 5.4%.
E. Training Details of DeepLabV2
DeepLabV2 training uses dataset-specific image augmentation, crop sizes, batch settings, iteration counts, learning rates, and testing with multi-scale inference plus dense CRF postprocessing.
- Training configuration: VOC images use random scales from 0.5 to 1.5, 321x321 crops, batch size 10, and 20k training iterations by default.
- Training configuration: COCO images use random scales from 0.5 to 2.0, 481x481 crops, batch size 5, and 100k training iterations.
- Training configuration: The initial learning rates are 2e-4 for ImageNet-pretrained models and 2.5e-5 for COCO-pretrained models.
- Training and testing: COCO training adopts balanced cross-entropy loss, while testing uses multi-scale inference and dense CRF postprocessing.
- Resource comparison: Comparisons with AdvCAM, CLIMS, and MCTFormer use their open-source code and default procedures on a TITAN RTX GPU with 24 GB memory.Dense CRF uses 20 workers, and affinity-network training reaches about 18GB for both PSA and IRN.
- Background categories: The method defines 25 class-related background categories for VOC and removes sign and keyboard for COCO because they are already COCO categories.
H. More Qualitative Results
Additional qualitative visualizations present generated pseudo labels and confidence maps for PASCAL VOC 2012 and MS COCO 2014. The reported results are satisfactory in both simple and complex scenes.
- The qualitative results include generated pseudo labels and corresponding confidence maps.
- The visualizations cover PASCAL VOC 2012 and MS COCO 2014 datasets.
- The proposed framework produces satisfactory segmentation results in both simple and complex scenes.