Source-linked AI summary
SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation
Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, Tianrui Li
TL;DR
Open-vocabulary semantic segmentation remains under-explored despite CLIP’s ability to learn rich visual concepts from large-scale image-text data. SegCLIP adapts CLIP with learnable patch centers and auxiliary representation losses, achieving comparable or superior accuracy across several benchmarks. The method transfers segmentation knowledge without annotation-based training, while boundary quality and pseudo-label generation remain practical limitations.
Problem
Pixel-level annotation is expensive, and fixed labeled categories limit generalization, motivating annotation-free open-vocabulary segmentation from image-text pretraining.
Method
SegCLIP modifies a ViT-based CLIP image encoder to aggregate patches with learnable centers into semantic regions, using reconstruction and superpixel-based KL losses.
Results
SegCLIP achieves superior mIoU to GroupViT1-s on ADE20K and Cityscapes, scoring 8.7% versus 4.9% and 11.0% versus 4.2%, respectively.
Takeaways & Limitations
SegCLIP generates plausible open-vocabulary segmentation from annotation-free text-image datasets without training on segmentation labels or seen segmentation classes before inference.
Takeaways & Limitations
Regular image patches can produce rough boundaries, so the study recommends smaller patch sizes; offline superpixel generation also remains a target for end-to-end improvement.
Abstract
from arXiv · showhide
Recently, the contrastive language-image pre-training, e.g., CLIP, has demonstrated promising results on various downstream tasks. The pre-trained model can capture enriched visual concepts for images by learning from a large scale of text-image data. However, transferring the learned visual knowledge to open-vocabulary semantic segmentation is still under-explored. In this paper, we propose a CLIP-based model named SegCLIP for the topic of open-vocabulary segmentation in an annotation-free manner. The SegCLIP achieves segmentation based on ViT and the main idea is to gather patches with learnable centers to semantic regions through training on text-image pairs. The gathering operation can dynamically capture the semantic groups, which can be used to generate the final segmentation results. We further propose a reconstruction loss on masked patches and a superpixel-based KL loss with pseudo-labels to enhance the visual representation. Experimental results show that our model achieves comparable or superior segmentation accuracy on the PASCAL VOC 2012 (+0.3% mIoU), PASCAL Context (+2.3% mIoU), and COCO (+2.2% mIoU) compared with baselines. We release the code at https://github.com/ArrowLuo/SegCLIP.
1. Introduction
Open-vocabulary segmentation seeks to avoid costly pixel annotations and fixed category vocabularies by transferring image-text knowledge to pixel-level semantic regions. SegCLIP uses learnable patch grouping within a ViT-based CLIP architecture, augmented by reconstruction and superpixel-based KL losses.
- Pixel-level labeling is expensive, while restricted labeled categories weaken semantic segmentation generalization.
- Open-vocabulary segmentation leverages large-scale class-agnostic image-text pretraining to segment images containing arbitrary categories.
- The group-based design learns semantic regions directly with image-text contrastive training and can benefit from pretrained CLIP weights.
- SegCLIP modifies CLIP’s ViT image encoder with a semantic group module that aggregates patches around learnable centers into irregular-shaped segments.
- A reconstruction loss recovers masked patches from visual context, while a superpixel-based KL loss learns a better mapping matrix and preserves pixel-level feature consistency.
2. Model
SegCLIP adapts CLIP’s dual-encoder ViT architecture for open-vocabulary segmentation by grouping image patches into semantic regions. It combines learnable patch aggregation with reconstruction, superpixel-based KL, and contrastive losses, then compares region and label features at inference.
- Main Architecture: SegCLIP uses CLIP-like text and image encoders, with a semantic group module inserted into the ViT-based image encoder.The module aggregates low-layer patch features before later Transformer layers produce region features.
- Semantic Group Module: Learnable centers dynamically group regular image patches into arbitrary-shaped semantic regions through cross-attention and a Gumbel-Softmax mapping matrix.Each patch is assigned to only one semantic center because each mapping-matrix row is one-hot.
- Reconstruction Loss: The reconstruction loss recovers masked patches from visual context using irregular-shaped segments and a mapping matrix rather than regular patches.The reconstruction objective is the mean squared error between the reconstructed and original images.
- Superpixel based KL Loss: The superpixel-based KL loss guides the mapping matrix toward pixel-level consistency by aligning each patch’s assignment probability with the average probability of its super-patch.Reducing this loss encourages patches within the same superpixel to gather into one region.
- Training and Inference: SegCLIP trains end to end with the sum of contrastive, reconstruction, and superpixel-based KL losses, then uses final region features with text label features for segmentation.Inference requires no further dataset-specific finetuning, and label text is formed with the template “a photo of a {label name}.”
3. Experiments
Experiments evaluate SegCLIP through loss and architectural ablations, then compare it with baselines on three annotation-free segmentation benchmarks. The model improves over GroupViT and benefits substantially from CLIP initialization.
- Experimental Setup: SegCLIP is evaluated on PASCAL VOC 2012, PASCAL Context, and COCO using mIoU on validation splits.The datasets contain 20, 59, and 80 foreground classes, respectively.
- Ablation Studies: The reconstruction loss improves mIoU across all three datasets, with gains of 1.19%, 0.92%, and 0.66% without superpixel-based KL loss.With superpixel-based KL loss, the corresponding gains are 4.11%, 0.56%, and 0.52%.
- Ablation Studies: The superpixel-based KL loss consistently improves mIoU by 0.54%, 0.72%, and 1.07% without reconstruction loss.With reconstruction loss, gains are 3.46%, 0.36%, and 0.93% on PASCAL VOC, PASCAL Context, and COCO, respectively.
- Ablation Studies: Plugging the semantic group module at layer 10 performs better than other tested layers, while overly small or large plugged points reduce mIoU.The authors attribute this to potential harm to pre-trained CLIP weights and segment-irrelevant features.
- Ablation Studies: Eight learnable centers provide better or comparable performance, with mIoU relatively insensitive to using 6, 8, or 10 centers.The model therefore uses 8 centers by default.
- Ablation Studies: Adding a cross-attention layer improves over no cross-attention, while four layers may harm performance because the training datasets are insufficient for deep layers.Two layers perform best in the reported comparison, with one and three layers comparable.
- Comparisons with State-of-the-Art Methods: 0.3%, 2.3%, and 2.2% gains over GroupViT are reported on VOC, Context, and COCO, respectively, for initialized SegCLIP.SegCLIP trained from scratch improves over GroupViT1-s by 5.2%, 4.3%, and 2.3%; CLIP initialization further improves mIoU by 19.3%, 5.6%, and 11.3% over scratch training.
- Qualitative Results: Qualitative results show plausible segments and reasonable tags, including single objects, multiple same-class objects, and objects from different classes.Initialized SegCLIP produces better semantics and tags than training from scratch in the cited examples.
4. Related Work
Related work connects open-vocabulary segmentation to vision-language pre-training and distinguishes classification-based methods from group-based methods. SegCLIP extends the group-based direction with CLIP weight reuse and auxiliary representation-learning objectives.
- Vision-Language Pre-Training: Vision-language pre-training commonly uses large-scale visual-linguistic pairs and designs architectures or pre-training objectives for downstream tasks.CLIP and ALIGN are cited as typical image-classification models trained with visual-text contrastive learning.
- Open-Vocabulary Semantic Segmentation: SegCLIP is presented as a vision-language pre-training model for segmentation that combines CLIP transfer with reconstruction and superpixel-based KL losses.The stated goal includes reusing CLIP for segmentation and reducing training-resource costs.
- Open-Vocabulary Semantic Segmentation: Open-vocabulary semantic segmentation targets images containing arbitrary text-described categories rather than fixed labeling vocabularies.Prior approaches include zero-shot, weakly supervised, and CLIP-based methods.
- Open-Vocabulary Semantic Segmentation: Prior segmentation methods use pixel-text matching, pseudo per-pixel labels, mask proposals, segment classification, or prompt-based segmentation.These approaches differ in whether they use decoders, ground-truth labels, pseudo-labels, or proposal-based pipelines.
- Open-Vocabulary Semantic Segmentation: Unlike previous methods requiring mask proposals or segmentation decoders, SegCLIP aggregates patches into segments with a plugged semantic group module.It differs from GroupViT through its architecture, CLIP pre-trained weight reuse, scratch training with noisy image-text pairs, and two additional objectives.
5. Conclusion and Future Work
SegCLIP demonstrates annotation-free transfer from image-text pretraining to open-vocabulary semantic segmentation, while highlighting boundary, scene-complexity, and superpixel-generation limitations for future work.
- Conclusion: SegCLIP generates plausible segmentation results from annotation-free text-image training without training on segmentation labels or seen segmentation classes before inference.
- Conclusion: The reconstruction and superpixel-based KL losses improve performance, underscoring the importance of image-encoder capacity for semantic representation.
- Future Work: Regular input patches can produce rough boundaries, so reducing patch size is recommended for smoother and more precise predictions.
- Future Work: 44.2%, 22.0%, and 21.4% mIoU with patch size 32 compared with 52.5%, 24.7%, and 26.5% with patch size 16 on VOC, Context, and COCO, respectively.Patch size 32 yields 49 patches per image, whereas patch size 16 yields 196 patches per image.
- Future Work: 8.7% and 11.0% mIoU on ADE20K and Cityscapes exceed GroupViT1-s results of 4.9% and 4.2%, while scene complexity remains important.The experiments use validation sets to assess more complex scenes.
- Future Work: The offline, non-end-to-end superpixel process, potentially biased finely divided superpixels, and limited pretraining scale remain areas for improvement.Suggested directions include end-to-end training, class-agnostic pseudo-label generation, and post-pretraining on CC12M and YFCC.