CRIS: CLIP-Driven Referring Image Segmentation

1. Introduction

• Referring image segmentation finds a particular image region from a linguistic expression rather than a predetermined category set.The task supports applications including interactive image editing and human-object interaction.Source: §1 · p. 1
• Different properties of image and language modalities make explicit alignment between textual features and pixel-level activations difficult.Earlier methods concatenate text with visual activations or use language-vision attention to improve cross-modal interaction.Source: §1 · pp. 1–2
• Existing pretrained approaches mainly transfer single-modal knowledge, while CLIP provides multimodal correspondence learned from large-scale image-text pairs.Direct CLIP usage can remain sub-optimal for pixel-level segmentation.Source: §1 · p. 2
• CRIS transfers CLIP knowledge for text-to-pixel alignment with a vision-language decoder and text-to-pixel contrastive learning.The decoder propagates fine-structured textual information to pixel-level features, while contrastive learning pulls related text and pixel features together and pushes irrelevant features apart.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

• Vision-language pretraining learns visual representations from language supervision and supports multimodal downstream tasks.Examples include MIL-NCE for instructional videos and SimVLM for large-scale weak supervision.Source: §2 · p. 2
• Contrastive learning trains representations by contrasting positive pairs against negative pairs, including approaches targeting pixel-level features for dense prediction.Pixel-level contrastive learning addresses the gap between self-supervised representation learning and dense prediction tasks.Source: §2 · p. 3
• Attention-based methods model cross-modal relationships by selecting informative words, visual regions, or language-conditioned visual contents.Examples include vision-guided linguistic attention, Cross-Modal Self-Attention, and bi-directional relationship inference.Source: §2 · p. 3
• CRIS combines multimodal information to boost cross-modal matching for referring image segmentation.This connects vision-language pretraining and contrastive learning with pixel-level segmentation.Source: §2 · pp. 2–3

Framework Overview

• CRIS extracts image and text features, fuses them through a cross-modal neck, and processes them with a vision-language decoder and projectors.The framework takes an image I and referring expression T as inputs.Source: §3 · p. 3, §3.1 · p. 3, Figure 3 · p. 4

Image & Text Feature Extraction

• The image encoder uses ResNet stages 2–4 to obtain multiple visual feature maps.These features have progressively different spatial resolutions and feature dimensions.Source: §3.1 · p. 3
• The text encoder uses a Transformer over BPE tokens to produce token-level features and a global representation from the [EOS] activation.The sequence is bracketed by [SOS] and [EOS] tokens.Source: §3.1 · p. 3
• The cross-modal neck fuses visual and global textual representations, using transformed features, elementwise multiplication, ReLU, and upsampling.Additional multi-modal features are then obtained from the fused representations.Source: §3.1 · p. 3

Vision-Language Decoder

• The vision-language decoder applies self-attention and cross-attention to propagate fine-grained textual semantics into pixel-level visual features.It adds fixed sine spatial positional encodings and produces evolved multi-modal features for the final segmentation mask.Source: §3.2 · pp. 4–5

Text-to-Pixel Contrastive Learning

• Text-to-pixel contrastive learning transforms global text and pixel-level features into a common dimension, encouraging similarity for corresponding pixels and dissimilarity for irrelevant pixels.The final segmentation result is obtained by reshaping the sigmoid of text-pixel similarities and upsampling it to the original image size.Source: §3.3 · p. 5

Experimental setup

• CRIS is evaluated on RefCOCO, RefCOCO+, and G-Ref using IoU and Precision@X, with CLIP initialization and no post-processing at inference.The framework uses ResNet-50 for ablation studies, images resized to 416 × 416, and binarizes predictions at threshold 0.35.Source: §4 · p. 5, §4.2 · p. 5

Ablation study

• 1.98%, 2.98%, and 3.43% IoU gains result from adding text-to-pixel contrastive learning over the baseline on the three datasets, respectively.The loss pulls linguistic and relevant pixel-level representations together while pushing away irrelevant representations.Source: §4.3 · p. 6
• Combining contrastive learning with the vision-language decoder improves IoU and Precision over either module alone, with margins of about 4%–8% across three datasets.The paper attributes this complementarity to contrastive learning guiding the decoder toward more informative emphasis.Source: §4.3 · p. 6

Main results

• CRIS surpasses prior state-of-the-art methods on every split of all three datasets despite using a shallow ResNet-50.Reported gains include 4.82%, 4.89%, and 3.37% over Vision Language Transformer on RefCOCO, about 4%–8% on RefCOCO+, and around 5% IoU over Locate then Segmentation on G-Ref.Source: §4.4 · p. 7

Qualitative analysis

• Qualitative results show the full CRIS model produces higher-quality masks than the baseline and either ablated component, while failures include ambiguous expressions, wrong labels, boundary errors, and occlusion.Boundary errors may be alleviated by a refine module, according to the paper.Source: §4.5 · pp. 7–8

• CRIS leverages CLIP in an end-to-end framework to achieve text-to-pixel alignment for referring image segmentation.Source: §5 · p. 8
• The vision-language decoder adaptively propagates language semantics into pixel-level visual features, promoting cross-modal consistency.Source: §5 · p. 8
• Text-to-pixel contrastive learning interweaves text representations with relevant pixel-level features to learn fine-grained multimodal correspondence.Source: §5 · p. 8
• CRIS significantly outperforms previous state-of-the-art methods without post-processing, with ablations validating each proposed component.Source: §5 · p. 8