PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-shot Referring Image Segmentation with Global-Local Context Features

Seonghoon Yu, Paul Hongsuck Seo, Jeany Son

arXiv:2303.17811v2cs.CVcs.AIcs.CL

TL;DR

RIS requires locating the image region described by an expression, but collecting its precise expressions and dense masks is costly. The paper uses pre-trained CLIP with global-local visual and text encoders, and reports outperforming several baselines and a weakly supervised method by substantial margins.

  • Problem

    RIS requires precise referring expressions and dense target masks, making dataset annotation costly and labor-intensive.

    Source: §1 · p. 1

  • Method

    The method uses pre-trained CLIP with mask-guided global-local visual features and text features combining whole-sentence context with a target noun phrase.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The method significantly outperforms all baseline methods and the weakly supervised method.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Global-local context encodings provide a unified framework for handling target-object semantics and relations between objects in zero-shot RIS.

    Source: §6 · p. 8

Abstract

from arXiv · show

Referring image segmentation (RIS) aims to find a segmentation mask given a referring expression grounded to a region of the input image. Collecting labelled datasets for this task, however, is notoriously costly and labor-intensive. To overcome this issue, we propose a simple yet effective zero-shot referring image segmentation method by leveraging the pre-trained cross-modal knowledge from CLIP. In order to obtain segmentation masks grounded to the input text, we propose a mask-guided visual encoder that captures global and local contextual information of an input image. By utilizing instance masks obtained from off-the-shelf mask proposal techniques, our method is able to segment fine-detailed Istance-level groundings. We also introduce a global-local text encoder where the global feature captures complex sentence-level semantics of the entire input expression while the local feature focuses on the target noun phrase extracted by a dependency parser. In our experiments, the proposed method outperforms several zero-shot baselines of the task and even the weakly supervised referring expression segmentation method with substantial margins. Our code is available at https://github.com/Seonghoon-Yu/Zero-shot-RIS.

1. Introduction

The introduction frames zero-shot referring image segmentation as a way to address costly annotation requirements. It proposes consistent global-local visual and textual encoders using CLIP and reports results above several baselines and a weakly supervised method.

1. Introduction

  • Task-specific dense annotations are costly, and weakly supervised RIS still requires paired high-level expressions and images for target datasets.RIS annotations require both precise referring expressions and dense target masks.Source: §1 · p. 1
  • RIS requires selecting the instance described by an expression, including distinguishing same-class objects through their relations and semantics.The example distinguishes two cats by the relation “lying on the seat.”Source: §1 · pp. 1–2
  • Zero-shot referring image segmentation uses pre-trained CLIP without additional training or supervision for this task.The paper presents this as a new task and, to the authors’ knowledge, the first study of it.Source: §1 · p. 2
  • The visual and textual encoders integrate global and local context consistently to capture target-object semantics and relations between objects in both modalities.The text encoder combines whole-sentence context with a target noun phrase; the visual encoder uses mask-guided context.Source: §1 · p. 2
  • The method outperforms several baselines and the weakly supervised referring image segmentation method with substantial margins.The introduction reports this comparison for the proposed global-local approach.Source: §1 · p. 2

2. Related Work

Prior work applies zero-shot transfer from image-text pretraining to downstream tasks, including dense prediction, while RIS methods have largely relied on supervised or weakly supervised training. This paper situates its approach among efforts to adapt CLIP to dense prediction and describes a global-local mask-scoring framework.

2. Related Work

  • CLIP-based zero-shot transfer has been applied to semantic segmentation [63], referring expression comprehension [49], phrase localization [25], and object localization.The cited approaches use CLIP encoders directly with small architectural modifications and no additional training.Source: §2 · p. 2
  • Dense prediction remains challenging with CLIP because its pretraining learns image-level rather than pixel-level features.Prior approaches crop regions or adapt CLIP to generate spatial features; this work extracts both global and local visual context.Source: §2 · p. 2
  • The framework extracts global-local visual features from mask proposals and scores them against a global-local text feature, selecting the mask with the highest cosine similarity.The image and expression are the framework’s inputs.Source: Figure 2 · p. 3
  • Fully supervised RIS methods require dense target masks and expressions, while TSEG [48] uses text-based image-level supervision but still needs paired high-level expressions for target datasets.The paper proposes a zero-shot baseline without training or supervision.Source: §2 · p. 3

3. Method

The method uses CLIP-based global-local visual and textual encoders to match a referring expression against image mask proposals. It combines scene-aware and target-focused features to select the proposal most similar to the expression.

3.1. Overall Framework

  • For each image, the method selects the mask proposal with the highest cosine similarity between its global-local visual feature and the expression’s global-local textual feature.The candidates are an unsupervised set of mask proposals.Source: §3.1 · pp. 3–4

3.2. Mask-guided Global-local Visual Features

  • The method uses unsupervised instance-level mask proposals to divide segmentation into proposal generation and masked image-text matching.This decomposition addresses CLIP’s limitation for pixel-level dense prediction and supports fine-detailed instance-level masks.Source: §3.2 · p. 4
  • The visual encoder combines a global feature that retains scene context with a local feature focused on the masked region.The global feature is extracted from the full image’s feature map, while the local feature comes from a masked, cropped image.Source: §3.2 · pp. 4–5
  • The global-local visual feature is a weighted sum of the global and local visual features, controlled by α.The resulting feature is compared with the textual feature to score each mask proposal.Source: §3.2 · p. 5

3.3. Global-local Textual Features

  • The text encoder combines a CLIP feature for the whole expression with a feature for the target noun phrase selected using spaCy dependency parsing.The local feature focuses on the target noun in complex expressions containing multiple clauses.Source: §3.3 · p. 5

4. Implementation Details

The implementation uses FreeSOLO mask proposals and CLIP visual encoders with ResNet-50 or ViT-B/32 backbones. For global context, the two architectures apply masking differently: to ResNet feature pooling or to ViT tokens.

4. Implementation Details

  • The implementation uses FreeSOLO for unsupervised instance masks, resizes the shorter image side to 800, and feeds CLIP images at 224x224.It sets α to 0.85 for RefCOCOg and 0.95 for RefCOCO and RefCOCO+, with β set to 0.5 for all datasets.Source: §4 · p. 5

4.1. Masking in Global-context Visual Encoder

  • The visual encoder uses CLIP with either a ResNet-50 or ViT-B/32 architecture.The two backbones use mostly similar, but slightly different, global-context masking strategies.Source: §4.1 · p. 5
  • For ResNet, the method masks feature maps before computing the query, key, and value supplied to attention pooling.The original CLIP ResNet attention-pooling layer replaces global average pooling.Source: §4.1 · p. 5
  • For ViT, the method masks patch tokens in the last three Transformer layers and uses the final CLS-token output as the global-context feature.The experiments use ViT-B/32; the paper reports that masking-layer locations are explored in supplementary materials.Source: §4.1 · p. 5

5. Experiments

The experiments evaluate the method on three RIS datasets, compare it with zero-shot baselines and supervised methods, and test its mask proposals and global-local features. The reported results favor the proposed method across the main comparisons and feature analyses.

5.1. Datasets and Metrics

  • The evaluation uses RefCOCO, RefCOCO+, and RefCOCOg, with shorter expressions in the first two datasets and longer, more relational expressions in RefCOCOg.RefCOCO and RefCOCO+ average 3.6 words and 1.6 nouns per expression; RefCOCOg averages 8.4 words and about 2.8 nouns.Source: §5.1 · pp. 5–6

5.3. Results

  • The method outperforms zero-shot baselines and TSEG on the reported benchmarks, although it remains below fully supervised methods; all baseline methods use FreeSOLO proposals.The TSEG comparison is on mIoU because TSEG reports only that metric; FreeSOLO upper-bound scores use the proposal with maximum overlap with each ground-truth mask.Source: Table 1 · p. 6, §5.3 · p. 7

5.4. Ablation Study

  • With COCO instance ground-truth masks as proposals, the method surpasses the baselines by over 3.5%, particularly on RefCOCOg.This setup evaluates segmentation performance without considering the quality of the mask generator.Source: Table 2 · p. 6, §5.4 · p. 8
  • Joint global-local visual and textual features perform best among the tested combinations; qualitative examples show local-only and global-only features can miss the target in different ways.With local visual features, predictions can favor a same-class instance; with global visual features, they can capture expression context but select a different class.Source: Table 3 · p. 6, Figure 5 · p. 7, Figure 6 · p. 7

5.2. Baselines

  • Baselines adapt Grad-CAM activation maps, dense score maps, ViT region tokens, and cropping to score shared FreeSOLO mask proposals.The evaluation reports overall and mean IoU; ablations use oIoU.Source: §5.2 · pp. 6–7

6. Conclusion

The conclusion presents a zero-shot RIS framework that transfers CLIP representations through global-local context encodings. It reports that the method outperforms baseline and weakly supervised methods.

  • The framework uses global-local context encodings to compare images and expressions, handling target-object semantics and inter-object relations together; it significantly outperforms baseline and weakly supervised methods.Source: §6 · p. 8

- Supplementary Materials -

The supplementary materials identify the paper’s authors, affiliations, and arXiv version.

A. Analysis on Global-local Textual Feature

The supplementary analysis finds that global-local textual features are most helpful for longer expressions, while shorter expressions often yield no distinct target noun phrase. It also describes the noun-phrase extraction procedure and lists several failure cases.

  • Global-local textual features outperform global-context features on longer RefCOCOg expressions, while RefCOCO and RefCOCO+ show no significant difference between them.Source: A. Analysis on Global-local Textual… · p. 11, Figure 9 · p. 11
  • The target noun phrase matches the full sentence in 48.61% of RefCOCO and 43.43% of RefCOCO+ cases, versus 7.46% of RefCOCOg cases.The paper links this difference to the smaller gains from global-local textual features on RefCOCO and RefCOCO+.Source: A. Analysis on Global-local Textual… · p. 11
  • SpaCy selects the noun phrase containing the sentence root, substitutes a child noun when the root is a verb, and falls back to the full sentence if needed.Source: A. Analysis on Global-local Textual… · p. 11
  • Reported failure cases include indistinguishable nearby instances in mask proposals, multiple plausible expression labels, and expressions too complex for CLIP.Source: Figure 10 · p. 12

B. Analysis on hyperparameters α and β.

The analysis selects separate visual and textual feature settings for combining global and local context, including dataset-specific visual α values.

B. Analysis on hyperparameters α and β.

C. Ablation Study on Token Masking in ViT

The ViT token-masking ablation selects different best-performing masking depths by model variant; additional visual-feature examples use COCO instance ground-truth masks as proposals.

C. Ablation Study on Token Masking in ViT

D. Additional Qualitative Results

On RefCOCOg, global-local features improve final oIoU while combining strengths in class accuracy and instance localization; additional examples show qualitative comparisons and failure cases.

D. Additional Qualitative Results

Loading 2303.17811v2…