PaperlayerView exact revision on arXiv

Source-linked AI summary

ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation

Ziqin Zhou, Bowen Zhang, Yinjie Lei, Lingqiao Liu, Yifan Liu

arXiv:2212.03588v3cs.CV

TL;DR

Existing CLIP-based zero-shot segmentation pipelines require region proposals and separate image encoding, while direct one-stage text-patch matching can overfit seen classes. ZegCLIP addresses this with three designs added to a one-stage framework, achieving strong performance across inductive and transductive benchmarks and about a fivefold inference speedup over a two-stage method.

  • Problem

    Two-stage CLIP-based zero-shot segmentation requires proposal generation and separate image encoding, while direct pixel-level adaptation can overfit seen classes and fail to generalize to unseen classes.

  • Method

    ZegCLIP uses one-stage text-patch matching with a lightweight decoder and incorporates Deep Prompt Tuning, Non-mutually Exclusive Loss, and Relationship Descriptor.

  • Results

    ZegCLIP outperforms state-of-the-art methods by a large margin in both inductive and transductive settings across three public benchmarks.

  • Takeaways & Limitations

    The resulting one-stage system retains CLIP’s zero-shot capability while improving pixel-level generalization to unseen classes.

  • Takeaways & Limitations

    MaskCLIP+ is invalid in the inductive setting when unseen class names are unknown during training, while the transductive setting assumes unseen class names and images are available before testing.

Abstract

from arXiv · show

Recently, CLIP has been applied to pixel-level zero-shot learning tasks via a two-stage scheme. The general idea is to first generate class-agnostic region proposals and then feed the cropped proposal regions to CLIP to utilize its image-level zero-shot classification capability. While effective, such a scheme requires two image encoders, one for proposal generation and one for CLIP, leading to a complicated pipeline and high computational cost. In this work, we pursue a simpler-and-efficient one-stage solution that directly extends CLIP's zero-shot prediction capability from image to pixel level. Our investigation starts with a straightforward extension as our baseline that generates semantic masks by comparing the similarity between text and patch embeddings extracted from CLIP. However, such a paradigm could heavily overfit the seen classes and fail to generalize to unseen classes. To handle this issue, we propose three simple-but-effective designs and figure out that they can significantly retain the inherent zero-shot capacity of CLIP and improve pixel-level generalization ability. Incorporating those modifications leads to an efficient zero-shot semantic segmentation system called ZegCLIP. Through extensive experiments on three public benchmarks, ZegCLIP demonstrates superior performance, outperforming the state-of-the-art methods by a large margin under both "inductive" and "transductive" zero-shot settings. In addition, compared with the two-stage method, our one-stage ZegCLIP achieves a speedup of about 5 times faster during inference. We release the code at https://github.com/ZiqinZhou66/ZegCLIP.git.

1. Introduction

Zero-shot semantic segmentation seeks pixel-level predictions for unseen classes without extensive pixel annotations, but CLIP-based two-stage methods require costly proposal generation and repeated image encoding. ZegCLIP simplifies this pipeline with a one-stage text-patch matching framework and three designs that improve unseen-class generalization.

  • Motivation: Zero-shot semantic segmentation aims to produce pixel-level class predictions from semantic descriptions while training on limited annotated classes.Its challenge comes from generalizing beyond the seen classes.
  • Existing limitation: Two-stage CLIP-based methods first generate region proposals and then classify cropped regions, requiring separate proposal and CLIP image encoding processes.This adds computational overhead and prevents CLIP knowledge from being used during proposal generation.
  • Approach: ZegCLIP directly extends CLIP from image-level to pixel-level prediction through a one-stage framework that matches text prompts with local patch embeddings.A lightweight transformer-based decoder performs the text-patch matching.
  • Baseline: The vanilla one-stage baseline improves seen-class segmentation but tends to overfit seen classes and fails to produce reasonable unseen-class segments.The paper identifies this as a loss of CLIP’s zero-shot capability during training.
  • Designs: Three designs—Deep Prompt Tuning, Non-mutually Exclusive Loss, and Relationship Descriptor—are incorporated to alleviate overfitting and improve pixel-level generalization.Figure 1 reports their step-by-step quantitative improvements on VOC.

2. Related Works

Prior work applies vision-language models and CLIP to dense prediction, but zero-shot segmentation methods either generalize semantic mappings from seen to unseen classes or classify generated regions at additional computational cost. Related approaches also include prompt tuning and self-training for adapting pretrained models and exploiting unlabeled unseen-class content.

  • Vision-language models: Pretrained vision-language models connect image representations with text embeddings and support tasks including retrieval, dense prediction, referring expressions, and visual question answering.CLIP is pretrained contrastively on 400 million text-image pairs and provides zero-shot classification ability.
  • Semantic segmentation: Semantic segmentation commonly treats prediction as per-pixel classification or separates mask generation from semantic classification, typically with a closed set of classes.These principles motivate adapting image-level vision-language knowledge to pixel-level prediction.
  • Zero-shot segmentation: Earlier zero-shot semantic segmentation methods focus on improving the generalization of semantic mappings from seen classes to unseen classes.The task remains challenging because of the imbalance between seen and unseen classes.
  • CLIP-based methods: Zegformer and zsseg use proposal generators followed by CLIP classification of each region, whereas MaskCLIP+ uses CLIP-generated novel-class pseudo annotations for self-training.Proposal classification increases computational cost, while MaskCLIP+ targets the transductive setting.
  • Prompt tuning: Prompt tuning adapts pretrained models by freezing original parameters and introducing learnable prompts, with visual prompt tuning inserting trainable parameters into transformer layers.This provides an alternative to directly fine-tuning the pretrained model.
  • Self-training: Transductive zero-shot segmentation uses ground-truth seen-class masks together with pseudo labels for unseen classes during self-training, unlike the inductive setting.The transductive setting assumes unseen class names and images are available before testing, while unseen names and images are unavailable during inductive training.

3. Method

ZegCLIP adapts CLIP for one-stage zero-shot semantic segmentation by matching class text embeddings with image patch embeddings. It addresses seen-class overfitting through deep prompt tuning, non-mutually exclusive loss, and relationship descriptors.

  • 3.1. Problem Definition: The generalized setting trains on pixel annotations for seen classes and evaluates segmentation on both seen and unseen classes.In the inductive setting, unseen class names and images are unavailable during training; the transductive setting assumes unseen class names are known before testing.
  • 3.2. Baseline: One-stage Text-Patch Matching: The one-stage baseline uses a lightweight transformer decoder to compute similarities between CLIP text embeddings and patch embeddings, avoiding proposal generation.Semantic masks are formed from scaled dot-product attention between projected class queries and image patch keys, followed by argmax over classes.
  • 3.2. Baseline: One-stage Text-Patch Matching: The baseline performs poorly on unseen classes because training the text-patch matching capability only on seen-class data can cause severe overfitting.This failure motivates the three design modifications introduced in the method.
  • 3.3. Design 1: Deep Prompt Tuning (DPT): Deep prompt tuning freezes CLIP’s original parameters and adds learnable prompt tokens at each transformer layer instead of fine-tuning the backbone.Compared with fine-tuning, it achieves similar seen-class performance while significantly improving unseen-class segmentation.
  • 3.4. Design 2: Non-mutually Exclusive Loss (NEL): Non-mutually exclusive loss replaces softmax-based exclusive classification with sigmoid and BCE-based independent class prediction, augmented by focal and dice losses.Compared with cross-entropy, BCE performs better on unseen classes, while focal and dice losses further improve performance.
  • 3.5. Design 3: Relationship Descriptor (RD): The relationship descriptor combines text-image matching information with original text embeddings to form image-specific text queries for the decoder.This design dramatically improves segmentation results on both seen and unseen categories.

4. Experiments

ZegCLIP is evaluated on three public zero-shot semantic segmentation benchmarks using seen/unseen class metrics and comparisons with prior methods. It achieves strong performance in both inductive and transductive settings while substantially reducing computational cost relative to a two-stage approach.

  • 4.1. Datasets: Experiments cover PASCAL VOC 2012, COCO-Stuff 164K, and PASCAL Context, with datasets partitioned into seen and unseen classes.VOC uses 15 seen and 5 unseen classes; COCO-Stuff uses 156 seen and 15 unseen; PASCAL Context uses 50 known and 10 unseen classes.
  • 4.2. Evaluation Metrics: Evaluation reports pAcc, mIoU(S), mIoU(U), and hIoU across seen and unseen classes.mIoU(S) and mIoU(U) denote mean class-wise IoU on seen and unseen classes, respectively.
  • 4.4. Comparison with State-of-the-art methods: ZegCLIP outperforms previous methods in the inductive setting, especially on unseen classes, and also performs strongly in the transductive setting.The authors report that this demonstrates superior generalization over prior approaches; after self-training, unseen performance improves while seen performance remains strong.
  • 4.3. Qualitative Results: Qualitative results show ZegCLIP segmenting seen and unseen classes and distinguishing similar unseen categories such as tree, grass, and playingfield.Figure 4 compares original images, Baseline-FT, ZegCLIP, and ground-truth masks, with white and red tags marking seen and unseen classes.
  • 4.4. Comparison with State-of-the-art methods: Approximately 14M learnable parameters and 117 Flops(G) make ZegCLIP 23% and 6% of Zegformer’s corresponding amounts, respectively.Both models are compared in Table 3, evaluated on a single 1080Ti GPU.
  • 4.4. Comparison with State-of-the-art methods: About 5 times faster inference gives the one-stage ZegCLIP a substantial speed advantage over the two-stage method.The comparison is made against Zegformer [12].

A. Effect of different formats of text query ˆt

The study examines how element-wise text-image operations form image-specific text queries for the segment decoder. Dot product and absolute difference provide more general information than sum or concatenation, and the selected format is used in ZegCLIP.

  • Relationship Descriptor: Relationship Descriptor combines text embeddings with CLIP’s image [cls] token to create image-specific text-image descriptions for the segment decoder.The resulting query is intended to improve zero-shot segmentation on unseen classes.
  • Qualitative comparison: Figure 4 compares original images, Baseline-FT predictions, ZegCLIP visualizations, and ground truths, marking seen and unseen classes with white and red tags.The figure reports qualitative results on COCO-Stuff 164K.
  • Text-query formats: Dot product and absolute difference between text embedding t and image token g provide more general information, whereas sum and concatenation perform poorly on seen and unseen classes.The authors select the red-line query format for the final ZegCLIP model.

B. Detailed results of applying designs on baseline

Ablations show that the three designs address complementary weaknesses of the one-stage baseline. Deep prompt tuning, non-mutually exclusive loss, and Relationship Descriptor together preserve CLIP’s transfer ability while improving seen and unseen segmentation.

  • Baseline analysis: Baseline-Fix performs unsatisfactorily on seen and unseen classes because its fixed CLIP image representation is weak for dense prediction.The baseline uses a learnable segment decoder while keeping CLIP fixed.
  • Baseline analysis: Baseline-FT improves seen-class performance but dramatically deteriorates on unseen classes, indicating overfitting after updating on seen classes.The reported issue is that fine-tuning appears to destroy CLIP’s zero-shot transfer ability.
  • Design ablations: Non-mutually Exclusive Loss significantly improves performance in both seen and unseen scenarios.This is identified as Design 2 in the ablation discussion.
  • Design ablations: Relationship Descriptor produces the largest gains, almost doubling unseen performance in many cases.The relationship description also improves segmentation on both seen and unseen categories.
  • Design ablations: Deep Prompt Tuning works well with Non-mutually Exclusive Loss and Relationship Descriptor, whereas replacing it with CLIP fine-tuning leaves seen-class overfitting unresolved.The final combination is reported to maintain strong performance on seen and unseen classes simultaneously.

C. Effect of advanced loss function

The loss-function study compares plain binary cross entropy with a version augmented by focal and dice losses. The designs target non-mutually exclusive segmentation and category imbalance.

  • Advanced loss function: The advanced loss combines binary cross entropy with focal loss and dice loss to address imbalance among categories.The comparison reports separate “plain” and “plus” versions.
  • Advanced loss function: “Plain” denotes binary cross entropy alone, while “plus” adds focal loss and dice loss to BCE.These variants are compared in Table 6.

D. Generalization ability to other datasets

ZegCLIP shows stronger cross-domain generalization than Zegformer when trained on seen classes from a source dataset and evaluated on seen and unseen classes in target datasets.

  • ZegCLIP achieves better cross-domain generalization than Zegformer across evaluations on other datasets.The comparison uses a model trained on seen classes from a source dataset and tested on both seen and unseen classes in target datasets.

5. Conclusion

The paper presents ZegCLIP as an efficient one-stage method that transfers CLIP’s image-level zero-shot knowledge to pixel-level segmentation. It reports improvements on novel classes, broad benchmark gains, and inference roughly five times faster than two-stage methods.

  • ZegCLIP is an efficient one-stage CLIP-based method for transferring image-level zero-shot classification to pixel-level semantic segmentation.The method uses text embeddings as queries and supports both inductive and transductive zero-shot settings.
  • 5. Conclusion: Across three public benchmarks, ZegCLIP outperforms previous state-of-the-art methods and runs about 5 times faster than two-stage methods during inference.The paper attributes the speed advantage to encoding the original image only once in the one-stage framework.
  • A. Effect of the number of deep prompt tokens: 10 deep prompt tokens suffice on VOC, while 100 on COCO and 35 on Context improve segmentation performance.The best prompt-token count generally increases with dataset scale and per-pixel classification complexity, although too many visual prompts can be detrimental.
  • B. Effect of the depth of deep prompt tokens: Prompt tokens inserted across all 12 ViT layers achieve the best performance, while bottom-layer insertion generally outperforms top-layer insertion.The all-layer setting is the default in the reported experiments.
  • C. Effect of single and multiple text templates: Multiple text descriptions provide reasonable improvements on COCO-Stuff 164K and PASCAL Context compared with a single template.VOC uses one template, whereas the larger datasets use multiple templates for class embeddings.

D. Brief frameworks of related Two-stage and our One-stage method

Prior CLIP-based methods generate region proposals and classify each cropped region in a two-stage pipeline, whereas the proposed one-stage approach directly matches text and patch embeddings. ZegCLIP adds three designs to address the baseline’s seen-class overfitting while retaining performance on seen and unseen classes.

  • Related two-stage methods: Two-stage methods first generate class-agnostic region proposals and then encode each cropped proposal with CLIP for image-level classification.This requires separate proposal-generation and proposal-classification processing.
  • Our one-stage method: The one-stage baseline uses a lightweight transformer decoder to compute similarities between CLIP text-wise and patch-wise embeddings for semantic masks.The text encoder is frozen, while the image encoder may be fixed or fine-tuned.
  • Our one-stage method: The baseline overfits seen classes, motivating three designs intended to improve generalization to unseen classes.Combining the designs yields ZegCLIP, which reports significant performance on both seen and unseen classes.
  • Our one-stage method: ZegCLIP’s one-stage framework encodes the original image once and achieves about 5 times faster inference than two-stage methods.Two-stage methods repeatedly feed N cropped class-agnostic images into CLIP for classification.

F. More visualization details

The visualizations compare Baseline-FT and ZegCLIP segmentation outputs and semantic-mask heat maps for different text-query embeddings. ZegCLIP improves seen and unseen segmentation and more clearly separates similar novel classes.

  • Figure 8 compares predicted segmentation results and decoder-generated semantic masks for Baseline-FT and ZegCLIP.Red class names indicate novel categories.
  • ZegCLIP improves segmentation on both seen and unseen classes compared with Baseline-FT.Its heat maps more clearly classify similar unseen classes, including distinguishing grass from tree in a COCO test image.
Loading 2212.03588v3…