PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Segmentation Using Text and Image Prompts

Timo Lüddecke, Alexander S. Ecker

arXiv:2112.10003v2cs.CV

TL;DR

Fixed-category segmentation requires retraining to accommodate new classes or complex queries. CLIPSeg adds a lightweight decoder to frozen CLIP and conditions segmentation on text or image prompts, achieving competitive performance across three low-shot tasks while generalizing to unseen classes and expressions.

  • Problem

    Image segmentation is commonly trained for fixed object classes, making later incorporation of additional classes or complex queries require retraining on encompassing data.

  • Method

    CLIPSeg extends a frozen CLIP backbone with a transformer-based decoder that generates binary segmentation maps from text prompts or support images.

  • Results

    CLIPSeg achieves competitive performance across referring expression, zero-shot, and one-shot segmentation and generalizes to classes and expressions without prior segmentation examples.

  • Takeaways & Limitations

    Text or image prompts let the trained-once system adapt segmentation to new tasks and generalized affordance or property queries at inference time.

  • Takeaways & Limitations

    The experiments cover only a small number of benchmarks, and the model depends on large-scale CLIP pretraining while focusing on images.

Abstract

from arXiv · show

Image segmentation is usually addressed by training a model for a fixed set of object classes. Incorporating additional classes or more complex queries later is expensive as it requires re-training the model on a dataset that encompasses these expressions. Here we propose a system that can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text or an image. This approach enables us to create a unified model (trained once) for three common segmentation tasks, which come with distinct challenges: referring expression segmentation, zero-shot segmentation and one-shot segmentation. We build upon the CLIP model as a backbone which we extend with a transformer-based decoder that enables dense prediction. After training on an extended version of the PhraseCut dataset, our system generates a binary segmentation map for an image based on a free-text prompt or on an additional image expressing the query. We analyze different variants of the latter image-based prompts in detail. This novel hybrid input allows for dynamic adaptation not only to the three segmentation tasks mentioned above, but to any binary segmentation task where a text or image query can be formulated. Finally, we find our system to adapt well to generalized queries involving affordances or properties. Code is available at https://eckerlab.org/code/clipseg.

1 Introduction

CLIPSeg addresses the limits of fixed-category segmentation by using text or image prompts to support multiple low-shot segmentation tasks with one model.

  • Motivation: Classical segmentation models are limited to categories seen during training, motivating methods that handle unseen categories, image supports, or complex text queries.The related settings include generalized zero-shot, one-shot, and referring expression segmentation.
  • Approach: CLIPSeg uses a pretrained CLIP backbone and a thin conditional decoder to segment an image from an arbitrary text query or example image.The shared text-visual embedding space conditions the decoder on either modality.
  • Unified setting: The model provides a unified framework for referring expression, zero-shot, and one-shot segmentation tasks, with dynamic target adaptation at inference time.The task comparison identifies dynamic adaptation as a property of the approaches beyond classic segmentation.
  • Results: CLIPSeg achieves competitive performance across three low-shot segmentation tasks and generalizes to classes and expressions without prior segmentation examples.The introduction reports this as a central result without specifying a numerical score.
  • Contribution: The model accepts multimodal prompts and extends beyond task-specific training toward flexible segmentation queries.Its contribution includes text and image target specification and a unified model trained for several benchmarks.

2 Related Work

Prior work covers foundation-model transfer and specialized referring-expression, zero-shot, and one-shot segmentation, while CLIP extensions motivate prompt-sensitive multimodal conditioning.

  • Foundation Models and Segmentation: Foundation models transfer representations from large-scale pretraining to multiple downstream tasks, and CLIP is a prominent image-text foundation model.CLIP has demonstrated strong performance on several image-classification tasks.
  • Foundation Models and Segmentation: CLIPSeg extends CLIP with a transformer decoder rather than convolutional layers for segmentation.This distinguishes its decoder design within the paper’s related-work discussion.
  • Referring Expression Segmentation: Referring expression segmentation maps a natural-language phrase to all matching pixels without requiring unseen-category or visual-support generalization.The paper uses PhraseCut because it is larger in images and classes than other referring-expression datasets.
  • Zero-Shot Segmentation: Zero-shot segmentation targets categories absent from training, including generalized settings containing both seen and unseen categories.Existing methods address bias toward seen classes using mechanisms such as synthesized pixel-wise features from label embeddings.
  • One-Shot Semantic Segmentation: One-shot semantic segmentation supplies a single class example, usually a masked support image, at test time.Subsequent methods add increasingly complex feature-processing and attention mechanisms.
  • CLIP Extensions: Prior CLIP extensions show that prompt design affects downstream behavior and apply CLIP across image editing, robotics, and model analysis.This motivates studying prompt construction beyond object names alone.

3 CLIPSeg Method

CLIPSeg freezes a visual CLIP encoder and adds a compact transformer decoder conditioned by text or engineered image prompts, trained using an expanded PhraseCut dataset.

  • Backbone and decoder: CLIPSeg uses frozen ViT-B/16 CLIP features with a small, parameter-efficient transformer decoder trained for segmentation.The larger ViT-L/14@336px variant was not used because its weights were unavailable.
  • Decoder Architecture: The decoder uses U-Net-inspired skip connections from selected CLIP layers to maintain a compact, purely transformer-based architecture.Projected CLIP activations are added before decoder transformer blocks.
  • Conditional prompts: FiLM conditioning supplies the segmentation target through either a CLIP text embedding or a visual embedding from a feature-engineered prompt image.The decoder outputs a binary segmentation map from the conditioned activations.
  • Image-Text Interpolation: Image-text interpolation combines support-image and text embeddings in CLIP’s shared embedding space to condition segmentation.The method treats text and image queries as alternative or interpolated target representations.
  • PhraseCut + Visual prompts (PC+): PhraseCut+ extends PhraseCut with visual support samples, negative samples, phrase augmentation, and crops that preserve partial object visibility.This enables training with image-text interpolation and a joint text-and-image model.
  • Visual prompt analysis: Visual prompt construction strongly changes CLIP’s target-object predictions, so the study compares masking and highlighting variants systematically.The analysis uses probability differences between the original image and masked prompt variants.

4 Visual Prompt Engineering

The paper evaluates ways to encode a masked support image as a visual prompt for CLIPSeg. Combining cropping, background blurring, and background darkening yields the strongest alignment improvement.

  • The study compares visual and text embeddings using cosine alignment, with modified images highlighting the target object.Alignments across highlighting techniques are softmax-normalized to compare target-object probabilities.
  • Visual prompt engineering combines a support image and mask into a new image before visual-transformer processing.This avoids applying the mask directly inside the model and treats prompt design as an input-engineering problem.
  • Cropping, background blurring, and background darkening each improve alignment between object text prompts and images.The evaluation covers 1,600 samples and measures average improvement in object probability.
  • The combination of cropping, background blurring, and background darkening performs best and is used thereafter.The paper therefore adopts this combined visual-prompt variant for subsequent one-shot segmentation experiments.

5 Experiments

Experiments assess CLIPSeg across referring-expression, generalized zero-shot, and one-shot segmentation, followed by qualitative tests of generalized prompts. The model is competitive across these settings and can handle unseen classes, expressions, affordance-like queries, and properties.

  • Experimental setup: The experiments evaluate flexible few-shot segmentation with either text or image prompts across three established segmentation benchmarks.Models generate binary query-matching maps, with a multi-label adaptation used when required for Pascal zero-shot segmentation.
  • Experimental setup: The universal PC+ model uses extended PhraseCut training with 20% negative samples, visual samples, and image-text interpolation.PC is trained on original PhraseCut, whereas PC+ is the extended universal variant.
  • 5.1 Referring Expression Segmentation: CLIPSeg outperforms HULANet on PhraseCut, although MDETR performs better under full-resolution processing and two rounds of PhraseCut fine-tuning.ViTSeg generally performs worse than CLIPSeg, indicating that CLIP pre-training is helpful in this comparison.
  • 5.2 Generalized Zero-Shot Segmentation: In generalized zero-shot segmentation, the model predicts one binary map per Pascal class and selects the highest-probability class for each pixel.The evaluation uses Pascal-VOC with unseen classes removed from PhraseCut+ training through WordNet-based filtering.
  • 5.2 Generalized Zero-Shot Segmentation: CLIPSeg’s zero-shot results are more balanced between seen and unseen classes than models trained specifically on Pascal-VOC seen classes.The paper presents this experiment as a reference rather than a direct competition because the training setup differs.
  • 5.3 One-Shot Semantic Segmentation: On Pascal-5i, CLIPSeg achieves competitive one-shot performance, with only HSNet performing better; it also works when trained on COCO-20i.For zero-shot evaluation using text input under the one-shot protocol, CLIPSeg outperforms the cited PFENet extension by a large margin.
  • 5.4 One Model For All: Generalized Prompts: Generalized affordance-like prompts and rare object-class queries work surprisingly well despite differing from the PhraseCut training prompts.Prompt details affect segmentation, while color information strongly influences predicted object probabilities; false positives occur rarely.

6 Conclusion

CLIPSeg adapts segmentation to new tasks through text or image prompts at inference time, achieving competitive performance across three low-shot tasks and generalizing to novel queries. Its scope remains constrained by limited benchmarks, reliance on CLIP pre-training, restricted image sizes, and potential dataset biases.

  • CLIPSeg adapts to new tasks using text or image prompts at inference time instead of expensive retraining on new data.
  • The model achieves competitive performance on referring expression, zero-shot, and one-shot image segmentation tasks.
  • CLIPSeg generalizes to novel prompts involving affordances and properties, demonstrated both qualitatively and quantitatively.
  • The experiments cover only a small number of benchmarks and depend on large-scale CLIP pre-training, while using a non-best-performing CLIP model because of weight availability.
  • Image-size variation is supported only within certain limits, and video applications may suffer from missing temporal consistency.
  • The approach may replicate biases from PhraseCut and especially from the unpublished CLIP training dataset.

Appendix

The appendix examines training details, image-size behavior, prompt and object effects, and qualitative comparisons with an ImageNet-trained baseline. It reports limited sensitivity to alternative text prompts, better performance on larger objects, balanced class performance, and CLIP advantages over ViTSeg.

  • Implementation: CLIPSeg experiments use PyTorch with CLIP ViT-B/16 and train on PhraseCut for 20,000 iterations using binary cross entropy.Training uses batches of 64, an initial learning rate of 0.001, cosine decay, and automatic mixed precision.
  • Image-size dependency: The publicly available CLIP ViT-B/16 and ViT-B/32 models were trained on 224 × 224 pixel images, motivating an image-size dependency experiment.The visual transformer can process arbitrary token counts, but the released models were trained at a fixed image size.
  • Metrics: Pixel-level average precision avoids dependence on a fixed threshold and is computed as the area under the precision-recall curve using Simpson integration.This is useful when new classes produce uncalibrated predictions.
  • Qualitative comparison: ViTSeg predictions show deficits relative to CLIPSeg: prompt details are not reflected in segmentation and many false positives occur.The comparison includes ViTSeg trained with visual samples, which shows worse performance.
  • Text prompts, object sizes and classes: The appendix analyzes different text prompts, object sizes, and object classes using randomly sampled prompt forms during training.These analyses are conducted on 5,000 PhraseCut test samples.
  • Text prompts, object sizes and classes: Alternative text prompts have a small effect on performance, larger objects perform better, and performance across classes is fairly balanced.The evaluation uses a pre-trained CLIPSeg model and 5,000 PhraseCut test samples.
Loading 2112.10003v2…