PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP

Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, Diana Marculescu

arXiv:2210.04150v3cs.CVcs.LG

TL;DR

Two-stage open-vocabulary segmentation is limited by CLIP’s poor classification of masked images. The paper adapts CLIP with caption-mined mask-category pairs and mask prompt tuning, achieving 29.6% mIoU on ADE20K-150 and matching 2017 supervised specialists without dataset-specific adaptation.

  • Problem

    Pre-trained CLIP performs poorly on masked images because they have a substantial domain gap from its natural-image training data.

  • Method

    The paper adapts CLIP using diverse mask-category pairs mined from image captions and replaces masked-image zero tokens with learnable prompt tokens.

  • Results

    29.6% mIoU on ADE20K-150 is 8.5% higher than OpenSeg under the same zero-shot setting.

  • Takeaways & Limitations

    Open-vocabulary generalist models can match supervised specialist models from 2017 without dataset-specific adaptations.

  • Takeaways & Limitations

    The mask proposal generator is not strictly class-agnostic, and its finer-grained segmentation depends on training-set class definitions.

Abstract

from arXiv · show

Open-vocabulary semantic segmentation aims to segment an image into semantic regions according to text descriptions, which may not have been seen during training. Recent two-stage methods first generate class-agnostic mask proposals and then leverage pre-trained vision-language models, e.g., CLIP, to classify masked regions. We identify the performance bottleneck of this paradigm to be the pre-trained CLIP model, since it does not perform well on masked images. To address this, we propose to finetune CLIP on a collection of masked image regions and their corresponding text descriptions. We collect training data by mining an existing image-caption dataset (e.g., COCO Captions), using CLIP to match masked image regions to nouns in the image captions. Compared with the more precise and manually annotated segmentation labels with fixed classes (e.g., COCO-Stuff), we find our noisy but diverse dataset can better retain CLIP's generalization ability. Along with finetuning the entire model, we utilize the "blank" areas in masked images using a method we dub mask prompt tuning. Experiments demonstrate mask prompt tuning brings significant improvement without modifying any weights of CLIP, and it can further improve a fully finetuned model. In particular, when trained on COCO and evaluated on ADE20K-150, our best model achieves 29.6% mIoU, which is +8.5% higher than the previous state-of-the-art. For the first time, open-vocabulary generalist models match the performance of supervised specialist models in 2017 without dataset-specific adaptations.

1. Introduction

The paper identifies pre-trained CLIP as the bottleneck in two-stage open-vocabulary segmentation because masked images differ substantially from its natural-image training data. It adapts CLIP using diverse caption-mined mask-category pairs and mask prompt tuning, achieving strong zero-shot results.

  • Open-vocabulary segmentation targets arbitrary text-described categories, unlike conventional models trained on predefined classes.
  • Two-stage methods generate class-agnostic mask proposals before using pre-trained CLIP to classify the masked regions.
  • 20.1% mIoU is achieved by pre-trained CLIP with oracle masks on ADE20K-150, revealing weak masked-image classification.
  • The authors mine diverse mask-category pairs from image captions instead of relying only on precise masks with limited fixed classes.
  • Mask prompt tuning replaces blank-region zero tokens with learnable prompts and can operate without changing CLIP weights.
  • 29.6% mIoU on ADE20K-150 is 8.5% higher than OpenSeg, while the model also reaches 9.0% on A-847 and 12.4% on PC-459.

2. Related Work

Related work applies vision-language models and prompt tuning to open-vocabulary vision tasks. This paper differs by adapting CLIP specifically to masked images and replacing masked tokens with learnable prompts.

  • Pre-trained CLIP provides open-vocabulary classification through vision-language features learned from image-text pairs.
  • Unlike RegionCLIP, this method adapts CLIP for masked images rather than complete region crops and leverages blank areas.
  • Prior open-vocabulary segmentation methods include generative feature synthesis, word-embedding alignment, text-supervised grouping, and direct use of CLIP.
  • Mask prompt tuning replaces masked image tokens with learnable prompts, unlike visual prompt tuning that inserts tokens before image tokens.
  • Mask prompt tuning further improves a fully finetuned model, an additional improvement not reported by prior work.

3. Method

The method combines mask proposal generation with CLIP-based open-vocabulary classification, adapting CLIP to masked images through caption-mined mask-category pairs and mask prompt tuning.

  • 3.1. Two-stage models for open-vocabulary semantic segmentation: The two-stage model generates mask proposals with MaskFormer and classifies each masked region using CLIP text embeddings.MaskFormer produces proposal embeddings, while CLIP compares region and text embeddings; predictions are fused for semantic segmentation.
  • 3.1. Two-stage models for open-vocabulary semantic segmentation: The proposal generator is not strictly class-agnostic because its notion of objects is determined by the training-set classes.The paper identifies general class-agnostic proposal generation as beyond its scope.
  • 3.2. Collecting diverse mask-category pairs from captions: Caption mining expands training supervision beyond fixed segmentation classes by extracting nouns and pairing them with matching masked proposals.COCO-Captions yields 1.3M mask-category pairs with 27K nouns using five captions per image, or 440K pairs with 12K nouns using one caption per image.
  • 3.3. Mask prompt tuning: The method adapts CLIP to masked images with mask prompt tuning, replacing zero tokens from fully masked patches with learnable mask prompts.Masked backgrounds become zero tokens that introduce domain shift; mask prompt tuning addresses these tokens without changing CLIP weights.
  • 3.3. Mask prompt tuning: Mask prompt tuning uses far fewer trainable parameters than full-model finetuning, requires no CLIP weight changes, and can further improve a fully finetuned model.The paper reports that prompt tuning alone significantly improves performance and that combining it with full finetuning yields additional gains.

4. Experiments

Experiments evaluate OVSeg in zero-shot settings across multiple segmentation benchmarks, showing strong performance against open-vocabulary and selected supervised models. Ablations and qualitative examples examine training data, mask prompt tuning, arbitrary queries, and evaluation ambiguity.

  • Experimental setup: OVSeg is trained on COCO and evaluated without dataset-specific adaptation on ADE20K, Pascal VOC, and Pascal Context.Evaluations include A-150/A-847, PAS-20, and PC-59/PC-459 category settings.
  • Main results: Under the R101c scale, OVSeg achieves 7.1% mIoU on A-847 and 11.0% mIoU on PC-459, outperforming other open-vocabulary models.Changing COCO-Stuff annotations from 171 to 156 categories produces only a 0.8% mIoU drop on A-150.
  • Main results: 29.6% mIoU on A-150 and 55.5% mIoU on PC-59 exceed prior zero-shot state-of-the-art results by 8.5% and 13.6%, respectively.On the challenging A-847 and PC-459 benchmarks, the model reaches 9.0% and 12.4% mIoU.
  • Main results: 94.5% mIoU on PAS-20 is 4.5% higher than the cited specialist state-of-the-art, while A-150 performance is similar to supervised FCN-8s.The paper qualifies these specialist comparisons because the models use different training data, backbones, and architectures.
  • Collecting mask-category pairs: Ground-truth mask-category pairs spanning 171 nouns reduce A-847 performance by 2.0%, consistent with overfitting to closed-set classes.The experiment collects 965K pairs from supervised COCO-Stuff labels.
  • Qualitative results: User-defined queries identify and segment unseen concepts including Saturn V, Oculus, and the Golden Gate Bridge.These examples illustrate querying arbitrary categories without training specifically on those concepts.
  • Ambiguity of open-vocabulary evaluation: A-150 evaluation can mark semantically reasonable predictions as wrong when language categories overlap, such as “skyscrapers” for “building” or “road” for “rail”.The paper identifies better open-vocabulary evaluation metrics as future work.

5. Conclusion

The paper adapts CLIP for masked images using diverse mask-category pairs and mask prompt tuning, enabling zero-shot open-vocabulary segmentation. Its generalist model matches supervised specialist performance reported in 2017 without dataset-specific adaptation.

  • Diverse mask-category pairs mined from image-caption data adapt CLIP for masked images while retaining open-vocabulary classification ability.
  • Mask prompt tuning adapts CLIP without changing its original weights.
  • The proposed model performs zero-shot segmentation on arbitrary datasets without dataset-specific adaptation.
  • Open-vocabulary generalist models can match supervised specialist models in 2017.

Ethics Statement

The experiments use public computer vision datasets and open-sourced CLIP models. The authors do not foresee inherent concerns across the listed ethical categories, while noting that the datasets and models may contain bias.

  • The experiments use public COCO, ADE20K, and Pascal datasets together with open-sourced CLIP models.
  • The authors do not foresee inherent concerns involving discrimination, inappropriate applications, privacy, security, or research practice.
  • Public datasets and pre-trained models may be subject to bias.

Reproducibility Statement

The authors state that their code is reproducible and can be implemented from the method description and training details.

  • The code is reproducible based on the method description in Section 3 and training details in Section 4.1.

A.1. Crop with or without mask

The crop-with-mask strategy outperforms direct cropping in the reported experiment, although the authors hypothesize that background pixels make masked crops more difficult.

  • A.1. Crop with or without mask: 13.8% mIoU is achieved with unmasked direct crops, which is -6.3% worse than using masked crops.
  • A.1. Crop with or without mask: The authors hypothesize that mask-based crops introduce many background pixels, making prediction more difficult.
  • A.1. Crop with or without mask: ZegFormer reports a similar conclusion in its ablation study of final-crop strategies.

A.2. Text templates

The method uses multiple natural-language templates for each category, then averages the resulting text embeddings to form the category representation.

  • Each category is paired with multiple text templates describing objects in photographic scenes.Templates vary phrasing and object scale, including small, medium, and large instances.
  • The resulting text embeddings are ensembled by a simple average for each category.
  • The templates include generic photographic descriptions and scene-based statements such as an object appearing in the scene.

A.3. Class prediction ensemble weight

This section evaluates class-prediction ensemble weighting, prompt-tuning configurations, training-pair sources, class-wise performance, and inference-time cost in the two-stage model.

  • Class prediction ensemble weight: CLIP adaptation improves CLIP-only performance over the baseline, while ensemble prediction remains essential for strong results.The reported settings use dataset-dependent ensemble weights: λ = 0.7 for ADE20K and λ = 0.6 for PAS-20 and Pascal Context.
  • Combining mask prompt tuning and fine-tuning: FT ->MPT is the default combination, applying mask prompt tuning after fine-tuning without changing the fine-tuned CLIP weights.Applying MPT before FT produces a 1.5% drop on A-150, while simultaneous optimization is also reported as unfavorable.
  • Prompt depth: 25.7% mIoU versus 21.8% is achieved with one prompt layer without fine-tuning on A-150, and deeper prompts improve performance in that setting.Prompt depth is evaluated at 1, 3, 6, and 12 layers, with and without a fully fine-tuned model.
  • Comparison with other prompt tuning methods: 25.5% mIoU on A-150 from VPT is 1.0% worse than MPT, while VPT requires 40% more computation.The passage attributes the difference to masked prompts preventing zero masked tokens and mitigating domain distribution shifts.
  • Training-pair sources: 26.7% mIoU versus 28.8% mIoU on A-150 results when ground-truth COCO-Stuff pairs are combined with pseudo-labeled pairs rather than using pseudo-labeled pairs alone.The authors attribute the lower result to class-distribution domination by ground-truth annotations and possible overfitting.
  • Class-wise IoU over seen and unseen categories: 37.6% versus 21.9% average IoU is reported for seen versus unseen ADE20K categories, respectively.The model is trained on COCO and evaluated across all 150 ADE20K categories; the figure reports class IoU for each category.
  • Inference speed discussions: 0.2s versus 0.6s per image is reported for MaskFormer and CLIP region classification on an NVIDIA A5000 GPU.Processing hundreds of regions with CLIP is identified as time-intensive and left for future efficiency work.
Loading 2210.04150v3…