PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupling Zero-Shot Semantic Segmentation

Jian Ding, Nan Xue, Gui-Song Xia, Dengxin Dai

arXiv:2112.07910v2cs.CV

TL;DR

ZS3 must recognize unseen categories, but pixel-level formulations have limited ability to integrate image-text vision-language models. The paper decouples segmentation into class-agnostic grouping and segment-level zero-shot classification, then introduces ZegFormer; it reports large gains on PASCAL VOC and COCO-Stuff and comparable performance to a fully supervised model on ADE20k-Full.

  • Problem

    Existing ZS3 methods formulate the task as pixel-level zero-shot classification and have limited capability to integrate vision-language models pretrained on image-text pairs.

  • Method

    The paper decouples ZS3 into class-agnostic pixel grouping and segment-level zero-shot classification, using ZegFormer to generate segment-level embeddings and optionally image embeddings from a pretrained vision-language model.

  • Results

    ZegFormer outperforms previous methods by 22 points in mIoU for unseen classes on PASCAL VOC and 3 points on COCO-Stuff, while remaining comparable to MaskFormer on ADE20k-Full.

  • Takeaways & Limitations

    The decoupled formulation provides a way to study ZS3 and integrate large-scale pretrained vision-language models such as CLIP.

  • Takeaways & Limitations

    ZegFormer-seg does not perform well when the training-data scale is small, possibly because its transformer structure needs substantial training data.

Abstract

from arXiv · show

Zero-shot semantic segmentation (ZS3) aims to segment the novel categories that have not been seen in the training. Existing works formulate ZS3 as a pixel-level zeroshot classification problem, and transfer semantic knowledge from seen classes to unseen ones with the help of language models pre-trained only with texts. While simple, the pixel-level ZS3 formulation shows the limited capability to integrate vision-language models that are often pre-trained with image-text pairs and currently demonstrate great potential for vision tasks. Inspired by the observation that humans often perform segment-level semantic labeling, we propose to decouple the ZS3 into two sub-tasks: 1) a classagnostic grouping task to group the pixels into segments. 2) a zero-shot classification task on segments. The former task does not involve category information and can be directly transferred to group pixels for unseen classes. The latter task performs at segment-level and provides a natural way to leverage large-scale vision-language models pre-trained with image-text pairs (e.g. CLIP) for ZS3. Based on the decoupling formulation, we propose a simple and effective zero-shot semantic segmentation model, called ZegFormer, which outperforms the previous methods on ZS3 standard benchmarks by large margins, e.g., 22 points on the PASCAL VOC and 3 points on the COCO-Stuff in terms of mIoU for unseen classes. Code will be released at https://github.com/dingjiansw101/ZegFormer.

1. Introduction

The paper reframes zero-shot semantic segmentation as class-agnostic pixel grouping followed by segment-level zero-shot classification, enabling integration of pre-trained vision-language models. ZegFormer instantiates this formulation and achieves strong benchmark results.

  • The decoupling formulation provides a more natural and flexible way to integrate large-scale pre-trained vision-language models such as CLIP.
  • Existing ZS3 methods formulate segmentation as pixel-level zero-shot classification and typically transfer knowledge using text-only language models.
  • Humans can first group pixels into segments without knowing object names, motivating a class-agnostic grouping task followed by segment-level semantic labeling.
  • Because grouping does not involve semantic categories, a model learned from seen classes can transfer this task to unseen classes.
  • ZegFormer uses a transformer decoder with mask and semantic projections for class-agnostic grouping and segment-level zero-shot classification.
  • 22 points on PASCAL VOC and 3 points on COCO-Stuff separate ZegFormer from previous methods in mIoU for unseen classes.

2. Related Works

Related work covers zero-shot image classification and zero-shot segmentation, highlighting a gap between image-level vision-language features and the pixel-level features used by earlier ZS3 methods.

  • Zero-shot image classification transfers knowledge from seen to unseen categories through semantic representations, including attributes, ontologies, word vectors, and image-text pretraining.
  • CLIP was trained on 400 million image-text pairs and achieved strong performance across more than 30 vision datasets, including comparisons with supervised models.
  • The paper identifies a feature-level gap between pixel-level representations in prior ZS3 models and image-level representations in vision-language models.
  • Prior zero-shot segmentation methods commonly formulate ZS3 as pixel-level zero-shot classification, using semantic embeddings to classify pixels or generate unseen-class pixel features.

3. Methodology

ZegFormer decouples zero-shot semantic segmentation into class-agnostic grouping and segment-level classification, using transformer-generated segment embeddings and vision-language text and image embeddings. During inference, predicted masks and fused segment class scores produce the final pixel-level segmentation.

  • Decoupling Formulation of ZS3: Semantic segmentation is represented by mappings that partition the image domain into disjoint homogeneous segments and assign each segment a category label.The category set may be closed in fully supervised settings, but generalized ZS3 allows both seen and unseen test classes.
  • ZegFormer: ZegFormer uses a transformer decoder to convert N segment queries and feature maps into N segment-level embeddings.Each embedding is processed in parallel by mask and semantic projection layers.
  • Class-Agnostic Grouping: The mask projection performs class-agnostic grouping by combining mask embeddings with high-resolution pixel-decoder features to predict binary masks.Because grouping does not involve category information, the grouping task can transfer from seen to unseen classes.
  • Segment Classification: The semantic projection performs segment-level zero-shot classification against text embeddings generated from class-name prompts and a vision-language text encoder.Seen-class text embeddings are used during training, while seen and unseen class embeddings are used during inference.
  • Inference: A pre-trained image encoder classifies segment sub-images, and its scores are fused with semantic segment scores before combining class scores with predicted masks.The full ZegFormer inference process uses both segment classification scores to obtain final semantic segmentation results.

4. Experiments

Experiments evaluate ZegFormer against baselines, alternative preprocessing choices, and prior GZS3 methods across COCO-Stuff, PASCAL VOC, and ADE20k-Full. Results support stronger grouping and segmentation performance, while image- and segment-based classification scores are complementary.

  • Datasets and evaluation: The evaluation covers GZS3 on COCO-Stuff, PASCAL VOC, and the challenging ADE20k-Full benchmark.COCO-Stuff uses 15 unseen classes, PASCAL VOC uses 5, and ADE20k-Full evaluates 847 classes split by frequency.
  • Comparisons with the baseline: ZegFormer-seg benefits more from CLIP text embeddings than SPNet-FPN, with improvements of 10.6 points versus 4.1 points over fastText plus word2vec.The comparison also reports that CLIP text embeddings outperform fastText plus word2vec for both methods.
  • Comparisons with the baseline: The decoupled formulation produces stronger class-agnostic grouping for unseen classes than pixel-level zero-shot classification.The grouping metric is evaluated on COCO-Stuff against SPNet-FPN.
  • Ablation study on preprocessing: Crop-and-mask preprocessing achieves the best segment-classification performance, whereas crop-only preprocessing performs below ZegFormer-seg.Cropping alone can include multiple categories, while masking alone retains unnecessary pixels outside the segment.
  • Ablation study on ZegFormer: ZegFormer-img performs better on thing categories but worse on stuff categories, so its scores are complementary to ZegFormer-seg and are fused.The comparison is reported using IoU of unseen classes on COCO-Stuff.
  • Comparison with the state-of-the-art: ZegFormer outperforms prior methods by 22 mIoU points on PASCAL VOC and 3 points on COCO-Stuff for unseen classes, while remaining comparable to a fully supervised model on ADE20k-Full.On ADE20k-Full, it surpasses SPNet-FPN by 4 points in unseen-class mIoU; the supervised model reaches 5.6 points.

5. Visualization

Visualization results show that decoupling improves unseen-category segmentation and remains more stable as the number of class names grows. A richer class-name set also provides more informative text embeddings for inference.

  • Pixel-level classification is inconsistent because pixels within one segment have large visual variations.
  • ZegFormer-seg is better than SPNet-FPN at segmenting unseen COCO-Stuff categories.
  • With 847 inference class names, SPNet-FPN exhibits increased competition among pixel-level classifications, whereas the decoupling model is not influenced by the number of classes.

6. Conclusion

The paper reformulates ZS3 through decoupling and presents ZegFormer as a strong baseline. It also identifies a small-data limitation for the segmentation-only variant and leaves few-shot segmentation for future work.

  • The paper reformulates ZS3 by decoupling it and proposes ZegFormer as a simple, effective model with significant advantages over previous work.
  • ZegFormer has potential for few-shot semantic segmentation, which the paper leaves for further research.
  • ZegFormer-seg does not perform well when the training dataset is small, possibly because transformer training needs substantial data.

A.1. Results with the ZS3 setting

The supplementary evaluation reports results under the ZS3 setting, where models predict only unseen labels and seen-class pixels are ignored. SPNet-FPN is included as a re-implemented baseline with FPN and CLIP text embeddings.

  • Under the ZS3 setting, models predict only unseen labels, while pixels belonging to seen classes are ignored.
  • SPNet-FPN is a re-implementation of SPNet with FPN and CLIP text embeddings, serving as a baseline on COCO-Stuff.

A.2. Speed and Accuracy Analyses

The analyses compare computational complexity, speed, accuracy, and backbone effects across benchmarks. Decoupling reduces classification-head complexity when the number of segments is much smaller than the number of image locations, while ZegFormer retains stronger generalized zero-shot performance across backbone comparisons.

  • Computational Complexity: The pixel-level classification head has complexity O(H × W × C × K), versus O(N×C×K) for decoupling, with N usually much smaller than H × W.
  • Speed and Accuracy: The COCO-Stuff and ADE20k-Full speed tables report FPS under their respective class-count and input-size settings.
  • Speed and Accuracy: Pixel-level zero-shot classification becomes much slower as K grows, and SPNet-FPN is slow on ADE20k-Full.
  • Different Backbones: SPNet-FPN with R-101 is comparable to ZegFormer-seg R-50 in supervised evaluation but much lower in GZS3 evaluation.

B. More Visualization Results

ZegFormer-Seg with R-50 and SPNet-FPN with R-50 are visualized after training on 156 COCO-Stuff classes while segmenting 847 classes.

  • Both models are trained on COCO-Stuff with 156 classes and required to segment 847 classes.The visualization results are presented in Figures 7 and 8.

C. More Implementation Details

ZegFormer uses implementation choices for pixel decoding, transformer decoding, mask projection, image cropping, and prompt-based text embeddings. Prompt-template ensembling provides a slight performance improvement over a single template.

  • Pixel and transformer decoders: ZegFormer and SPNet-FPN use an FPN pixel decoder with output stride 4 and six Transformer decoder layers.The same loss is applied after each decoder layer.
  • Mask projection: The ZegFormer mask projection layer contains two hidden layers with 256 channels.
  • Training crops: Training crops are 640 × 640 for COCO-Stuff and 512 × 512 for ADE20k-Full and PASCAL VOC.
  • Prompt templates: For each category, multiple prompt templates generate text embeddings that are ensembled by averaging.The listed templates vary descriptions by size and scene context.
  • Prompt ablation: A prompt-ensemble setup slightly improves performance compared with using one prompt template.The single-template baseline uses “A photo of the {} in the scene.”
Loading 2112.07910v2…