PaperlayerView exact revision on arXiv

Source-linked AI summary

A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model

Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, Xiang Bai

arXiv:2112.14757v2cs.CV

TL;DR

Open-vocabulary semantic segmentation must transfer image-level vision-language recognition to pixel-level predictions despite their granularity mismatch. The paper addresses this with a two-stage framework that generates mask proposals and classifies them with pre-trained CLIP. It reports improvements over FCN in cross-dataset evaluation and over prior zero-shot methods on Pascal VOC 2012 and COCO Stuff.

  • Problem

    Existing semantic segmentation uses limited category sets, while transferring CLIP’s image-level recognition capability to pixel-level segmentation remains unclear.

  • Method

    The framework generates class-agnostic mask proposals and uses pre-trained CLIP with prompted class descriptions to classify each proposal.

  • Results

    +29.5 hIoU on Pascal VOC 2012 and +8.9 hIoU on COCO Stuff over previous best methods with additional self-training.

  • Takeaways & Limitations

    The framework provides a simple baseline for open-vocabulary and zero-shot semantic segmentation.

  • Takeaways & Limitations

    The evaluation defines seen and unseen classes as disjoint, with unseen classes represented by semantic words or class descriptions.

Abstract

from arXiv · show

Recently, open-vocabulary image classification by vision language pre-training has demonstrated incredible achievements, that the model can classify arbitrary categories without seeing additional annotated images of that category. However, it is still unclear how to make the open-vocabulary recognition work well on broader vision problems. This paper targets open-vocabulary semantic segmentation by building it on an off-the-shelf pre-trained vision-language model, i.e., CLIP. However, semantic segmentation and the CLIP model perform on different visual granularity, that semantic segmentation processes on pixels while CLIP performs on images. To remedy the discrepancy in processing granularity, we refuse the use of the prevalent one-stage FCN based framework, and advocate a two-stage semantic segmentation framework, with the first stage extracting generalizable mask proposals and the second stage leveraging an image based CLIP model to perform open-vocabulary classification on the masked image crops which are generated in the first stage. Our experimental results show that this two-stage framework can achieve superior performance than FCN when trained only on COCO Stuff dataset and evaluated on other datasets without fine-tuning. Moreover, this simple framework also surpasses previous state-of-the-arts of zero-shot semantic segmentation by a large margin: +29.5 hIoU on the Pascal VOC 2012 dataset, and +8.9 hIoU on the COCO Stuff dataset. With its simplicity and strong performance, we hope this framework to serve as a baseline to facilitate future research. The code are made publicly available at~\url{https://github.com/MendelXu/zsseg.baseline}.

1 Introduction

Open-vocabulary semantic segmentation addresses limited category coverage by using additional information beyond segmentation annotations, but transferring CLIP’s image-level recognition to pixel-level prediction remains unclear. The paper proposes a two-stage mask-generation and classification approach and reports strong cross-dataset and zero-shot results.

  • Motivation: Existing semantic segmentation datasets usually contain tens or hundreds of categories because pixel-level annotation is costly.
  • Motivation: Open-vocabulary semantic segmentation generalizes zero-shot segmentation by allowing additional data or information to segment arbitrary classes.
  • Problem: CLIP’s image-level vision-category alignment does not directly match FCN’s pixel-level classification granularity, making effective transfer unclear.
  • Approach: The proposed framework decouples segmentation into class-agnostic mask proposal generation followed by mask category classification.
  • Results: +13.1 mIoU on Cityscapes, +19.6 mIoU on Pascal Context, +5.6 mIoU on ADE20k with 150 classes, and +2.9 mIoU on ADE20k with 847 classes over FCN in cross-dataset evaluation.The model is trained on one dataset and evaluated on another without fine-tuning.
  • Results: +29.5 hIoU on Pascal VOC 2012 and +8.9 hIoU on COCO Stuff over previous best methods using additional self-training.The comparisons are reported in the zero-shot setting.

3 Preliminary

The paper defines open-vocabulary segmentation through disjoint seen and unseen classes and uses a two-stage framework to bridge image-level CLIP recognition with pixel-level segmentation. It also evaluates cross-dataset generalization without fine-tuning.

  • Zero-Shot Setting: Open-vocabulary segmentation predicts masks for unseen classes learned from labeled data of disjoint seen classes.
  • Zero-Shot Setting: Seen and unseen classes may be represented by semantic words such as dog, cat, and apple, or by class descriptions.
  • Framework: The framework first generates mask proposals and then uses pre-trained CLIP to classify each proposal.
  • Cross-Dataset Setting: Cross-dataset evaluation trains on one dataset and evaluates on another without fine-tuning, adding domain-gap challenges beyond unseen classes.
  • CLIP: CLIP estimates image-text semantic similarity from image and text encoder representations, enabling classification with prompted class names.
  • CLIP: The approach extends CLIP’s image-level zero-shot classification capability to pixel-level open-vocabulary semantic segmentation.

4 Two-Stage Open-Vocabulary Semantic Segmentation

The framework generates class-agnostic mask proposals, then applies CLIP-based classification to masked foreground crops and assembles the predictions into segmentation results. It also explores prompt designs and aggregation for overlapping proposals.

  • Framework overview: Mask proposals are first generated, classified with CLIP using masked image crops and text-derived class weights, then assembled into final segmentation results.The image and text encoders compare each foreground crop with prompt-generated class representations.
  • Mask Proposal Generation: The proposal generator can use GPB-UCM, Selective Search, or MaskFormer, with MaskFormer serving as the default because it produces high-quality unseen-class proposals.GPB-UCM emphasizes contour alignment, while Selective Search effectively localizes objects.
  • Region Classification: Direct CLIP classification binarizes each proposal, removes background, crops the foreground, resizes it to 2242, and feeds it to the CLIP image encoder.A second strategy retrains an image encoder while using fixed CLIP text features as classifier weights to preserve unseen-class generalization.
  • Region Classification: The two CLIP-based classification strategies complement each other, so their results are ensembled by default.The ensemble combines direct masked-crop classification with the retrained image-encoder strategy.
  • Prompt Design: Prompt design includes selecting among 80 hand-crafted CLIP prompts or learning prompt tokens on seen classes for generalization to unseen classes.Hand-crafted prompts are evaluated on training data, while learning-based prompts use a sequence of trainable tokens followed by the class token.
  • Mask Prediction Assembly: Overlapping proposals are handled by aggregating their mask and category probabilities for each pixel, which is assigned to the category with the highest resulting value.The aggregated category probabilities are not guaranteed to sum to 1.

5 Fully Convolution Network Approach

The paper also evaluates FCN-based alternatives that classify pixels directly, either using CLIP features or a retrained encoder with fixed text-derived classifier weights.

  • FCN formulation: FCN formulates segmentation as pixel-wise classification using a high-resolution feature map and learned classifiers applied independently to pixels.This is the conventional fully convolutional alternative to the proposal-based framework.
  • CLIP-based FCN: Directly using CLIP’s feature map may degrade performance because CLIP represents images with the [CLS] token and was pretrained at 224 × 224 resolution.Semantic segmentation commonly requires higher resolution, such as a shorter side of 640.
  • Retrained FCN: A retrained FCN vision encoder uses fixed classifier weights generated by CLIP’s text encoder, providing some generalization ability to unseen classes.The same two strategies are ensembled by default when not otherwise specified.

6 Experiments

Experiments evaluate cross-dataset generalization and zero-shot segmentation across several datasets, using mIoU and hIoU as the principal metrics for the two settings.

  • Datasets: The evaluation covers COCO Stuff, Pascal VOC 2012, Pascal Context, Cityscapes, and ADE20K.COCO Stuff contains 117k training images, 5k validation images, and 171 annotated classes.
  • Evaluation settings: Cross-dataset experiments train on COCO Stuff and test on other datasets without fine-tuning, measuring generalization with class-wise mIoU.The setting tests transfer across datasets rather than only transfer to unseen classes within one dataset.
  • Evaluation settings: Zero-shot experiments evaluate COCO Stuff and Pascal VOC 2012 using seen/unseen class splits and harmonic mean IoU as the major metric.COCO Stuff uses 156 seen and 15 unseen classes, while Pascal VOC 2012 uses 15 seen and 5 unseen classes.
  • Implementation details: Experiments use a MaskFormer model with a ResNet-101 backbone as the default proposal generator and CLIP with a ViT-B/16 backbone.Training uses AdamW, poly learning-rate scheduling, and separate iteration budgets for zero-shot and cross-dataset settings.
  • Cross-dataset results: Table 1 compares methods trained on COCO Stuff and evaluated on other datasets, with dataset names indicating their class counts and both methods using the same prompt engineering.The table is designed to isolate the cross-dataset comparison under matched prompt tuning.

2012 dataset,

The experiments show that the two-stage framework generalizes across datasets and substantially outperforms prior zero-shot segmentation methods, with proposal generation and CLIP classification both contributing to performance.

  • Zero-Shot Setting: The method achieves 41.5 hIoU and 43.6 mIoU-unseen on COCO Stuff with self-training, exceeding STRICT by +8.9 hIoU and +13.3 mIoU-unseen.Without self-training, it achieves 37.8 hIoU and 36.3 mIoU-unseen, surpassing CaGNet by +19.5 hIoU and +24.1 mIoU-unseen.
  • Image Encoder and Pre-Training: 49.5 hIoU with ResNet-101 using only ImageNet-1K data exceeds the 33.4 hIoU obtained by integrating CLIP with SPNet under the same backbone.The comparison indicates that performance is attributed not only to pre-training data but also to the two-stage framework.
  • Proposal Generation: MaskFormer proposals outperform Selective Search and GPB-UCM when masked regions are directly classified by CLIP, while the alternative proposal methods remain comparable to prior state-of-the-art mIoU-unseen results.The comparison is designed to isolate proposal-generation performance from a learnable classifier.
  • Proposal Generalization: MaskFormer proposal generators trained on COCO Stuff generalize well to ADE20K, and the reverse transfer also retains acceptable performance degradation.An oracle experiment evaluates proposal quality through final segmentation performance because a simple proposal-recall metric is unavailable.
  • Region Classification: The retrained vision encoder performs better on seen classes, CLIP performs better on unseen classes, and ensembling them significantly improves both.The results support complementary behavior between the two region-classification strategies.
  • CLIP Variants: Among evaluated CLIP variants, ViT-B/16 achieves the best performance, while all tested models perform well as region classifiers.The comparison uses manual prompts and omits learnable prompts because of experimental overhead.

7 Conclusion

The paper proposes a simple two-stage framework that generates class-agnostic masks and classifies them with pre-trained CLIP. It reports large-margin gains over prior zero-shot segmentation methods and presents the framework as a strong baseline for future research.

  • 7 Conclusion: The framework separates open-vocabulary segmentation into binary mask proposal generation and CLIP-based classification of each proposal.This design uses a pre-trained vision-language model to classify mask proposals.
  • 7 Conclusion: The method outperforms previous state-of-the-art zero-shot semantic segmentation methods by large margins on Pascal VOC 2012 and COCO Stuff.The paper reports extensive experiments supporting the framework.
  • 7 Conclusion: The framework demonstrates the potential of pre-trained vision-language models for open-vocabulary and zero-shot semantic segmentation and serves as a baseline for future research.The authors describe the approach as simple and effective.

A Definition of hIoU

hIoU is defined as the harmonic mean of mIoU on seen and unseen classes, combining performance across both groups.

  • hIoU is the harmonic mean of mIoU_seen and mIoU_unseen.It is computed as 2 × mIoU_seen × mIoU_unseen divided by their sum.

B Sliding Window Testing in Fully Convolutional Network

Sliding-window testing substantially improves FCN performance, but the resulting FCN remains below the two-stage framework.

  • 11.7 hIoU and 10.4 mIoU-unseen were achieved by FCN without sliding-window testing.
  • +9.2 hIoU and +5.6 mIoU-unseen resulted from applying sliding-window testing to FCN.Each image is split into several 224 × 224 patches for testing.
  • The sliding-window gain is attributed to inconsistent image sizes between CLIP pre-training and testing.
  • -16.8 hIoU and -20.3 mIoU-unseen separated sliding-window FCN from the two-stage approach.The comparison indicates the two-stage framework is more suitable for the CLIP model.

C.1 Prompt Engineering for Image

The study examines image and text prompt design, prompt-data choices, and MaskFormer–CLIP ablations, finding strong benefits from learnable text prompts and the two-stage design but instability from learned image-background fills.

  • C.1 Prompt Engineering for Image: Image prompt engineering crops each mask proposal, expands its bounding box by ratio r, and includes additional context.The prompted crop is passed to the CLIP image encoder.
  • C.1 Prompt Engineering for Image: Background filling strongly affects segmentation performance, while learnable pixel prompts and mask tokens are unstable and usually harmful.The explored alternatives include preserved context, zero fill, dataset-mean fill, learnable pixel prompts, and mask tokens.
  • C.1 Prompt Engineering for Image: Learnable image prompts did not improve performance like learnable text prompts, leaving image prompt design open for future research.
  • C.2 Prompt Engineering for Text: +9.9 hIoU was obtained by the learnable text prompt over the manually searched prompt.The learnable prompt was trained on seen classes but improved both seen and unseen classes by +9.6 mIoU-seen and +10.2 mIoU-unseen.
  • C.2 Prompt Engineering for Text: 32 samples per category gave the best unseen-class performance for prompt lengths of 16 or 32.More training samples degraded performance, which the authors speculate may reflect overfitting.
  • D Detailed Study on MaskFormer and CLIP: MaskFormer outperformed FCN by +26.8 hIoU in the reorganized ablation results.
  • D Detailed Study on MaskFormer and CLIP: CLIP pre-training outperformed ImageNet pre-training by +24.7 hIoU.
  • D Detailed Study on MaskFormer and CLIP: The proposed method outperformed SPNet by +24.4 hIoU with the same pre-training data.

E The Randomness of the Data split

The experiments compare official and random seen/unseen class splits and provide qualitative cross-dataset examples on Pascal Context and ADE20k.

  • The official unseen/seen split used for comparison has thing/stuff ratios of 0.88 for seen and 0.87 for unseen classes.
  • A more balanced unseen thing/stuff ratio yielded higher hIoU on randomly generated splits.
  • Figures 5 and 6 show original images, ground-truth maps, and predictions for cross-dataset evaluation on Pascal Context and ADE20k.White ground-truth regions are ignored during annotation.
Loading 2112.14757v2…