Source-linked AI summary
CRIS: CLIP-Driven Referring Image Segmentation
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, Tongliang Liu
TL;DR
Referring image segmentation requires aligning linguistic expressions with pixel-level features, while existing pretrained approaches often omit multimodal correspondence. CRIS uses a vision-language decoder and text-to-pixel contrastive learning to transfer CLIP knowledge, significantly outperforming prior state-of-the-art methods on three benchmarks. Its failures include ambiguity, incorrect labels, inaccurate boundaries, and occlusion.
Problem
Different image and language properties make text-to-pixel alignment difficult, and existing pretrained methods mainly transfer single-modal knowledge without multimodal correspondence.
Method
CRIS transfers CLIP’s multimodal knowledge through a vision-language decoder and text-to-pixel contrastive learning.
Results
CRIS significantly outperforms previous state-of-the-art methods on RefCOCO, RefCOCO+, and G-Ref, including gains of +4.89, +8.88, and +5.47 IoU, respectively.
Takeaways & Limitations
The framework transfers CLIP’s cross-modal matching ability while learning fine-structured visual representations for text-to-pixel alignment.
Takeaways & Limitations
Failures remain under ambiguous expressions, wrong labels, inaccurate referent boundaries, and occlusion.
Abstract
from arXiv · showhide
Referring image segmentation aims to segment a referent via a natural linguistic expression.Due to the distinct data properties between text and image, it is challenging for a network to well align text and pixel-level features. Existing approaches use pretrained models to facilitate learning, yet separately transfer the language/vision knowledge from pretrained models, ignoring the multi-modal corresponding information. Inspired by the recent advance in Contrastive Language-Image Pretraining (CLIP), in this paper, we propose an end-to-end CLIP-Driven Referring Image Segmentation framework (CRIS). To transfer the multi-modal knowledge effectively, CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. More specifically, we design a vision-language decoder to propagate fine-grained semantic information from textual representations to each pixel-level activation, which promotes consistency between the two modalities. In addition, we present text-to-pixel contrastive learning to explicitly enforce the text feature similar to the related pixel-level features and dissimilar to the irrelevances. The experimental results on three benchmark datasets demonstrate that our proposed framework significantly outperforms the state-of-the-art performance without any post-processing. The code will be released.
1. Introduction
Referring image segmentation must align language with pixel-level visual features, but existing approaches often lack multimodal correspondence. CRIS transfers CLIP knowledge through a vision-language decoder and text-to-pixel contrastive learning, achieving strong benchmark gains.
- 1. Introduction: Referring image segmentation finds a particular image region from a linguistic expression rather than a predetermined category set.The task supports applications including interactive image editing and human-object interaction.
- 1. Introduction: Different properties of image and language modalities make explicit alignment between textual features and pixel-level activations difficult.Earlier methods concatenate text with visual activations or use language-vision attention to improve cross-modal interaction.
- 1. Introduction: Existing pretrained approaches mainly transfer single-modal knowledge, while CLIP provides multimodal correspondence learned from large-scale image-text pairs.Direct CLIP usage can remain sub-optimal for pixel-level segmentation.
- 1. Introduction: CRIS transfers CLIP knowledge for text-to-pixel alignment with a vision-language decoder and text-to-pixel contrastive learning.The decoder propagates fine-structured textual information to pixel-level features, while contrastive learning pulls related text and pixel features together and pushes irrelevant features apart.
2. Related Work
Related work spans vision-language pretraining, contrastive representation learning, and attention-based cross-modal modeling. CRIS extends these directions toward pixel-level referring image segmentation and multimodal correspondence.
- 2. Related Work: Vision-language pretraining learns visual representations from language supervision and supports multimodal downstream tasks.Examples include MIL-NCE for instructional videos and SimVLM for large-scale weak supervision.
- 2. Related Work: Contrastive learning trains representations by contrasting positive pairs against negative pairs, including approaches targeting pixel-level features for dense prediction.Pixel-level contrastive learning addresses the gap between self-supervised representation learning and dense prediction tasks.
- 2. Related Work: Attention-based methods model cross-modal relationships by selecting informative words, visual regions, or language-conditioned visual contents.Examples include vision-guided linguistic attention, Cross-Modal Self-Attention, and bi-directional relationship inference.
- 2. Related Work: CRIS combines multimodal information to boost cross-modal matching for referring image segmentation.This connects vision-language pretraining and contrastive learning with pixel-level segmentation.
3. Methodology
CRIS combines image and text encoders with a cross-modal neck, vision-language decoder, and projectors to align referring expressions with pixels. Its decoder propagates textual semantics to visual features, while contrastive learning aligns corresponding text and pixel features.
- Framework Overview: CRIS extracts image and text features, fuses them through a cross-modal neck, and processes them with a vision-language decoder and projectors.The framework takes an image I and referring expression T as inputs.
- Image & Text Feature Extraction: The image encoder uses ResNet stages 2–4 to obtain multiple visual feature maps.These features have progressively different spatial resolutions and feature dimensions.
- Image & Text Feature Extraction: The text encoder uses a Transformer over BPE tokens to produce token-level features and a global representation from the [EOS] activation.The sequence is bracketed by [SOS] and [EOS] tokens.
- Image & Text Feature Extraction: The cross-modal neck fuses visual and global textual representations, using transformed features, elementwise multiplication, ReLU, and upsampling.Additional multi-modal features are then obtained from the fused representations.
- Vision-Language Decoder: The vision-language decoder applies self-attention and cross-attention to propagate fine-grained textual semantics into pixel-level visual features.It adds fixed sine spatial positional encodings and produces evolved multi-modal features for the final segmentation mask.
- Text-to-Pixel Contrastive Learning: Text-to-pixel contrastive learning transforms global text and pixel-level features into a common dimension, encouraging similarity for corresponding pixels and dissimilarity for irrelevant pixels.The final segmentation result is obtained by reshaping the sigmoid of text-pixel similarities and upsampling it to the original image size.
4. Experimental results
Experiments on RefCOCO, RefCOCO+, and G-Ref evaluate CRIS against prior methods and isolate the contributions of contrastive learning and the vision-language decoder. CRIS improves segmentation across benchmarks, while failures remain associated with ambiguous expressions, incorrect labels, inaccurate boundaries, and occlusion.
- Experimental setup: CRIS is evaluated on RefCOCO, RefCOCO+, and G-Ref using IoU and Precision@X, with CLIP initialization and no post-processing at inference.The framework uses ResNet-50 for ablation studies, images resized to 416 × 416, and binarizes predictions at threshold 0.35.
- Ablation study: 1.98%, 2.98%, and 3.43% IoU gains result from adding text-to-pixel contrastive learning over the baseline on the three datasets, respectively.The loss pulls linguistic and relevant pixel-level representations together while pushing away irrelevant representations.
- Ablation study: Combining contrastive learning with the vision-language decoder improves IoU and Precision over either module alone, with margins of about 4%–8% across three datasets.The paper attributes this complementarity to contrastive learning guiding the decoder toward more informative emphasis.
- Main results: CRIS surpasses prior state-of-the-art methods on every split of all three datasets despite using a shallow ResNet-50.Reported gains include 4.82%, 4.89%, and 3.37% over Vision Language Transformer on RefCOCO, about 4%–8% on RefCOCO+, and around 5% IoU over Locate then Segmentation on G-Ref.
- Qualitative analysis: Qualitative results show the full CRIS model produces higher-quality masks than the baseline and either ablated component, while failures include ambiguous expressions, wrong labels, boundary errors, and occlusion.Boundary errors may be alleviated by a refine module, according to the paper.
5. Conclusion
CRIS is an end-to-end framework that leverages CLIP for text-to-pixel alignment in referring image segmentation. Its vision-language decoder and text-to-pixel contrastive learning transfer cross-modal knowledge and improve fine-grained visual representations, outperforming prior state-of-the-art methods without post-processing.
- CRIS leverages CLIP in an end-to-end framework to achieve text-to-pixel alignment for referring image segmentation.
- The vision-language decoder adaptively propagates language semantics into pixel-level visual features, promoting cross-modal consistency.
- Text-to-pixel contrastive learning interweaves text representations with relevant pixel-level features to learn fine-grained multimodal correspondence.
- CRIS significantly outperforms previous state-of-the-art methods without post-processing, with ablations validating each proposed component.