Image Segmentation Using Text and Image Prompts

Motivation

• Classical segmentation models are limited to categories seen during training, motivating methods that handle unseen categories, image supports, or complex text queries.The related settings include generalized zero-shot, one-shot, and referring expression segmentation.Source: §1 · p. 1

Approach

• CLIPSeg uses a pretrained CLIP backbone and a thin conditional decoder to segment an image from an arbitrary text query or example image.The shared text-visual embedding space conditions the decoder on either modality.Source: §1 · p. 1

Unified setting

• The model provides a unified framework for referring expression, zero-shot, and one-shot segmentation tasks, with dynamic target adaptation at inference time.The task comparison identifies dynamic adaptation as a property of the approaches beyond classic segmentation.Source: §1 · p. 1, Table 1 · p. 2

Results

• CLIPSeg achieves competitive performance across three low-shot segmentation tasks and generalizes to classes and expressions without prior segmentation examples.The introduction reports this as a central result without specifying a numerical score.Source: §1 · p. 2

Contribution

• The model accepts multimodal prompts and extends beyond task-specific training toward flexible segmentation queries.Its contribution includes text and image target specification and a unified model trained for several benchmarks.Source: §1 · pp. 1–2

Foundation Models and Segmentation

• Foundation models transfer representations from large-scale pretraining to multiple downstream tasks, and CLIP is a prominent image-text foundation model.CLIP has demonstrated strong performance on several image-classification tasks.Source: §2 · p. 2
• CLIPSeg extends CLIP with a transformer decoder rather than convolutional layers for segmentation.This distinguishes its decoder design within the paper’s related-work discussion.Source: §2 · p. 2

Referring Expression Segmentation

• Referring expression segmentation maps a natural-language phrase to all matching pixels without requiring unseen-category or visual-support generalization.The paper uses PhraseCut because it is larger in images and classes than other referring-expression datasets.Source: §2 · p. 2

Zero-Shot Segmentation

• Zero-shot segmentation targets categories absent from training, including generalized settings containing both seen and unseen categories.Existing methods address bias toward seen classes using mechanisms such as synthesized pixel-wise features from label embeddings.Source: §2 · p. 2

One-Shot Semantic Segmentation

• One-shot semantic segmentation supplies a single class example, usually a masked support image, at test time.Subsequent methods add increasingly complex feature-processing and attention mechanisms.Source: §2 · pp. 2–3

CLIP Extensions

• Prior CLIP extensions show that prompt design affects downstream behavior and apply CLIP across image editing, robotics, and model analysis.This motivates studying prompt construction beyond object names alone.Source: §2 · p. 3

Backbone and decoder

• CLIPSeg uses frozen ViT-B/16 CLIP features with a small, parameter-efficient transformer decoder trained for segmentation.The larger ViT-L/14@336px variant was not used because its weights were unavailable.Source: §3 · p. 3

Decoder Architecture

• The decoder uses U-Net-inspired skip connections from selected CLIP layers to maintain a compact, purely transformer-based architecture.Projected CLIP activations are added before decoder transformer blocks.Source: §3 · p. 3

Conditional prompts

• FiLM conditioning supplies the segmentation target through either a CLIP text embedding or a visual embedding from a feature-engineered prompt image.The decoder outputs a binary segmentation map from the conditioned activations.Source: §3 · p. 3

Image-Text Interpolation

• Image-text interpolation combines support-image and text embeddings in CLIP’s shared embedding space to condition segmentation.The method treats text and image queries as alternative or interpolated target representations.Source: §3 · p. 3

PhraseCut + Visual prompts (PC+)

• PhraseCut+ extends PhraseCut with visual support samples, negative samples, phrase augmentation, and crops that preserve partial object visibility.This enables training with image-text interpolation and a joint text-and-image model.Source: §3.1 · pp. 3–4

Visual prompt analysis

• Visual prompt construction strongly changes CLIP’s target-object predictions, so the study compares masking and highlighting variants systematically.The analysis uses probability differences between the original image and masked prompt variants.Source: Figure 3 · p. 4

• The study compares visual and text embeddings using cosine alignment, with modified images highlighting the target object.Alignments across highlighting techniques are softmax-normalized to compare target-object probabilities.Source: §4 · p. 4
• Visual prompt engineering combines a support image and mask into a new image before visual-transformer processing.This avoids applying the mask directly inside the model and treats prompt design as an input-engineering problem.Source: §4 · p. 5
• Cropping, background blurring, and background darkening each improve alignment between object text prompts and images.The evaluation covers 1,600 samples and measures average improvement in object probability.Source: §4 · p. 5, Table 2 · p. 5
• The combination of cropping, background blurring, and background darkening performs best and is used thereafter.The paper therefore adopts this combined visual-prompt variant for subsequent one-shot segmentation experiments.Source: §4 · p. 5

Experimental setup

• The experiments evaluate flexible few-shot segmentation with either text or image prompts across three established segmentation benchmarks.Models generate binary query-matching maps, with a multi-label adaptation used when required for Pascal zero-shot segmentation.Source: §5 · p. 5
• The universal PC+ model uses extended PhraseCut training with 20% negative samples, visual samples, and image-text interpolation.PC is trained on original PhraseCut, whereas PC+ is the extended universal variant.Source: §5 · p. 5

5.1 Referring Expression Segmentation

• CLIPSeg outperforms HULANet on PhraseCut, although MDETR performs better under full-resolution processing and two rounds of PhraseCut fine-tuning.ViTSeg generally performs worse than CLIPSeg, indicating that CLIP pre-training is helpful in this comparison.Source: §5.1 · p. 6

5.2 Generalized Zero-Shot Segmentation

• In generalized zero-shot segmentation, the model predicts one binary map per Pascal class and selects the highest-probability class for each pixel.The evaluation uses Pascal-VOC with unseen classes removed from PhraseCut+ training through WordNet-based filtering.Source: §5.2 · p. 6
• CLIPSeg’s zero-shot results are more balanced between seen and unseen classes than models trained specifically on Pascal-VOC seen classes.The paper presents this experiment as a reference rather than a direct competition because the training setup differs.Source: §5.2 · p. 6, Table 4 · p. 6

5.3 One-Shot Semantic Segmentation

• On Pascal-5i, CLIPSeg achieves competitive one-shot performance, with only HSNet performing better; it also works when trained on COCO-20i.For zero-shot evaluation using text input under the one-shot protocol, CLIPSeg outperforms the cited PFENet extension by a large margin.Source: §5.3 · p. 7, Table 6 · p. 7, Table 7 · p. 7

5.4 One Model For All: Generalized Prompts

• Generalized affordance-like prompts and rare object-class queries work surprisingly well despite differing from the PhraseCut training prompts.Prompt details affect segmentation, while color information strongly influences predicted object probabilities; false positives occur rarely.Source: §5.4 · p. 7

• CLIPSeg adapts to new tasks using text or image prompts at inference time instead of expensive retraining on new data.Source: §6 · p. 8
• The model achieves competitive performance on referring expression, zero-shot, and one-shot image segmentation tasks.Source: §6 · p. 8
• CLIPSeg generalizes to novel prompts involving affordances and properties, demonstrated both qualitatively and quantitatively.Source: §6 · p. 8
• The experiments cover only a small number of benchmarks and depend on large-scale CLIP pre-training, while using a non-best-performing CLIP model because of weight availability.Source: §6 · p. 9
• Image-size variation is supported only within certain limits, and video applications may suffer from missing temporal consistency.Source: §6 · p. 9
• The approach may replicate biases from PhraseCut and especially from the unpublished CLIP training dataset.Source: §6 · p. 9

Implementation

• CLIPSeg experiments use PyTorch with CLIP ViT-B/16 and train on PhraseCut for 20,000 iterations using binary cross entropy.Training uses batches of 64, an initial learning rate of 0.001, cosine decay, and automatic mixed precision.Source: Experimental Setup · p. 12

Image-size dependency

• The publicly available CLIP ViT-B/16 and ViT-B/32 models were trained on 224 × 224 pixel images, motivating an image-size dependency experiment.The visual transformer can process arbitrary token counts, but the released models were trained at a fixed image size.Source: Image-size Dependency of CLIP · p. 12

Metrics

• Pixel-level average precision avoids dependence on a fixed threshold and is computed as the area under the precision-recall curve using Simpson integration.This is useful when new classes produce uncalibrated predictions.Source: Average Precision Computation · p. 12

Qualitative comparison

• ViTSeg predictions show deficits relative to CLIPSeg: prompt details are not reflected in segmentation and many false positives occur.The comparison includes ViTSeg trained with visual samples, which shows worse performance.Source: Qualitative Predictions · p. 12, Figure 6 · p. 13

Text prompts, object sizes and classes

• The appendix analyzes different text prompts, object sizes, and object classes using randomly sampled prompt forms during training.These analyses are conducted on 5,000 PhraseCut test samples.Source: Text prompts, object sizes and classes · p. 14
• Alternative text prompts have a small effect on performance, larger objects perform better, and performance across classes is fairly balanced.The evaluation uses a pre-trained CLIPSeg model and 5,000 PhraseCut test samples.Source: Text prompts, object sizes and classes · p. 14