PaperlayerView exact revision on arXiv

Source-linked AI summary

Text Augmented Spatial-aware Zero-shot Referring Image Segmentation

Yucheng Suo, Linchao Zhu, Yi Yang

arXiv:2310.18049v1cs.CV

TL;DR

Zero-shot referring image segmentation avoids the labor-intensive pixel-level annotations needed to identify objects described by referring expressions, while masked-region and text alignment remains a challenge. TAS combines SAM mask proposals, text-augmented matching, and spatial rectification. It outperforms previous methods on RefCOCO, RefCOCO+, and RefCOCOg.

  • Problem

    Pixel-level annotations for referring image segmentation are labor-intensive, and masked-image matching must bridge a visual-text domain gap while improving region-text alignment.

    Source: §1 · pp. 1–2

  • Method

    TAS uses SAM for instance-level mask proposals, augments visual-text matching with caption-based positive and negative text scores, and applies a spatial rectifier.

    Source: §1 · p. 2, §3.3 · p. 4, §5 · p. 9

  • Results

    TAS outperforms previous methods on RefCOCO, RefCOCO+, and RefCOCOg.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TAS facilitates CLIP prediction without modifying the CLIP architecture or further fine-tuning.

    Source: §1 · p. 2

    Performance is constrained by occasional failures to generate ideal SAM proposals and depends on the image-text contrastive model employed.

    Source: §6 · p. 9

Abstract

from arXiv · show

In this paper, we study a challenging task of zero-shot referring image segmentation. This task aims to identify the instance mask that is most related to a referring expression without training on pixel-level annotations. Previous research takes advantage of pre-trained cross-modal models, e.g., CLIP, to align instance-level masks with referring expressions. %Yet, CLIP only considers image-text pair level alignment, which neglects fine-grained image region and complex sentence matching. Yet, CLIP only considers the global-level alignment of image-text pairs, neglecting fine-grained matching between the referring sentence and local image regions. To address this challenge, we introduce a Text Augmented Spatial-aware (TAS) zero-shot referring image segmentation framework that is training-free and robust to various visual encoders. TAS incorporates a mask proposal network for instance-level mask extraction, a text-augmented visual-text matching score for mining the image-text correlation, and a spatial rectifier for mask post-processing. Notably, the text-augmented visual-text matching score leverages a $P$ score and an $N$-score in addition to the typical visual-text matching score. The $P$-score is utilized to close the visual-text domain gap through a surrogate captioning model, where the score is computed between the surrogate model-generated texts and the referring expression. The $N$-score considers the fine-grained alignment of region-text pairs via negative phrase mining, encouraging the masked image to be repelled from the mined distracting phrases. Extensive experiments are conducted on various datasets, including RefCOCO, RefCOCO+, and RefCOCOg. The proposed method clearly outperforms state-of-the-art zero-shot referring image segmentation methods.

1 Introduction

Zero-shot referring segmentation avoids pixel-level training but must match complex expressions to image regions despite CLIP’s global alignment and masked-image domain gap. TAS combines mask proposals, text-augmented matching, and spatial rectification, and outperforms previous methods on the evaluated datasets.

1 Introduction

  • Zero-shot referring segmentation avoids costly pixel-level annotations, but CLIP’s global image-text alignment limits fine-grained matching and masked-image alignment.Prior methods also overlook distracting objects within an image.Source: §1 · pp. 1–2
  • TAS combines instance-level mask proposals, a text-augmented visual-text matching score, and a spatial rectifier.The framework uses SAM for mask proposals.Source: §1 · p. 2
  • The matching score combines masked-image similarity to the referring expression, caption similarity, and negative-expression similarity to address domain gap and distracting objects.Negative expressions are mined from noun phrases in input-image captions.Source: §1 · p. 2
  • The spatial rectifier uses mask-center coordinates to select the highest-scoring mask in the region indicated by an orientation description.For “man to the left,” it selects from the left half of the masks.Source: §1 · p. 2
  • TAS outperforms previous zero-shot referring segmentation methods on RefCOCO, RefCOCO+, and RefCOCOg without modifying CLIP or further fine-tuning.The authors report experiments and ablation studies on these datasets.Source: §1 · p. 2

2 Related Work

Related work covers cross-modal models for dense prediction, referring segmentation methods with varying annotation requirements, and image captioning. TAS’s pipeline builds on instance-mask proposals and caption-generated text.

2.1 Zero-shot Segmentation

  • Cross-modal pretraining has been applied to dense prediction; SAM can generate prompted masks or masks for all instances without a prompt.The text used in related dense-prediction works is often limited to object classes or attributes.Source: §2.1 · p. 2

2.2 Referring Image Segmentation

  • Referring image segmentation requires sentence comprehension; supervised methods need labor-intensive annotations, while zero-shot work retrieves masks using CLIP.The cited zero-shot framework uses FreeSOLO masks, and TAS’s pipeline applies SAM proposals before V-, P-, and N-score computation and spatial rectification.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

2.3 Image Captioning

  • Image captioning generates text for images, and TAS adopts BLIP-2 as its captioning network.The passage also notes that recent large language models enrich caption diversity.Source: §2.3 · p. 3

3 Method

TAS selects a referring-expression mask using instance proposals, text-augmented visual-text matching, and spatial post-processing. Its method combines SAM proposals with complementary text signals and a spatial rectifier.

3.1 Overall Framework

  • TAS selects a referring-expression mask by combining instance-level proposals, text-augmented mask–expression matching, and spatial post-processing.The framework targets fine-grained region-text matching and the gap between masked and natural images without modifying the model architecture.Source: §3.1 · p. 3

3.2 Mask Proposal Network

  • Figure 3 visualizes each network’s largest-IoU mask; SAM separates instances better than FreeSOLO, including in occluded or clustered scenes.The paper reports that SAM’s stronger instance segmentation yields higher upper-bound performance.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 Text-augmented visual-text matching score

  • The score supplements masked-image CLIP similarity with per-mask captions, matching caption text to the referring expression to address the masked-image domain gap.The V-score measures masked-image/text similarity, while the P-score compares generated captions with the referring expression.Source: §3.3 · p. 4
  • The N-score penalizes masks matching unrelated objects, using image-caption noun phrases filtered against synonyms of the referring-expression subject.The remaining phrases are treated as negative expressions when scoring masked images.Source: §3.3 · pp. 4–5
  • TAS linearly combines the V-, P-, and N-scores and selects the mask with the highest combined score.The combined score is formed in a common CLIP feature space without changing the model’s feature space or structure.Source: §3.3 · p. 5

3.4 Spatial Rectifier

  • The spatial rectifier identifies orientation words, uses mask centers as locations, and selects the highest-scoring mask in the corresponding image region.When no orientation word is found, the method does not apply spatial rectification.Source: §3.4 · p. 5

4 Experiments

Experiments evaluate TAS on referring-segmentation datasets against activation-map and similarity-based baselines, then test its modules and design choices. TAS outperforms the baselines on the reported benchmarks, with ablations examining module contributions and input formats.

4.1 Dataset and Metrics

  • Evaluation covers RefCOCO, RefCOCO+, RefCOCOg, and PhraseCut, using oIoU and mIoU.RefCOCOg includes UMD and Google partitions in the Table 1 comparison.Source: §4.1 · p. 5, Table 1 · p. 6

4.4 Results

  • TAS outperforms all baselines on RefCOCO, RefCOCO+, and RefCOCOg in oIoU and mIoU, and also beats the previous method on PhraseCut.Qualitative comparisons report that TAS handles long, complex expressions and orientation-related descriptions; all methods use SAM proposals in Figure 4.Source: Table 1 · p. 6, §4.4 · pp. 6–7, Table 2 · p. 7

4.2 Implementation Details

  • Experiments use default ViT-H SAM, BLIP-2 OPT-2.7b, and RN50 or ViT-B/32 CLIP; α is 0.1 and λ varies by dataset.λ is 0.7 for RefCOCO and RefCOCO+, and 1 for RefCOCOg.Source: §4.2 · p. 6

4.3 Baselines

  • Baselines include activation-map methods, masked image-text similarity methods, Text-only, CLIP-only, and the weakly supervised TSEG method.Global-Local is also evaluated with SAM masks for a fair comparison.Source: §4.3 · p. 6

4.5 Ablation Study

  • Ablations find that all proposed modules improve performance, Spatial is especially important on orientation-rich RefCOCO, λ is insensitive, and large α harms performance.The module comparison reports RefCOCO validation oIoU and mIoU; the weight sweep varies α and λ separately.Source: Table 4 · p. 7, Table 3 · p. 7, §4.5 · p. 7
  • Masked-image input preferences differ by model: BLIP-2 benefits from blurring, while CLIP performs better with cropping.The authors suggest that blurring supports context-aware BLIP-2 descriptions, whereas the black background left by cropping helps CLIP focus on the foreground.Source: Table 5 · p. 8, §4.5 · p. 8
  • Captioning-model changes have little effect, TAS improves results with ALBEF and BLIP-2, and its reported runtime is 3.63 seconds per image.The full pipeline used about 22 GB of GPU memory and was timed on one RTX 3090; CLIP-Only took 1.88 seconds per image.Source: §4.5 · p. 8

5 Conclusion

The conclusion presents TAS as a zero-shot referring image segmentation framework combining instance-level masks, text-augmented matching, and spatial rectification. Experiments on RefCOCO, RefCOCO+, and RefCOCOg demonstrate its effectiveness, while hard expressions over non-salient instances remain a future challenge.

5 Conclusion

  • TAS combines a mask proposal network, text-augmented visual-text matching, and a spatial rectifier for zero-shot referring image segmentation.The framework uses off-the-shelf SAM to obtain instance-level masks.Source: §5 · pp. 8–9
  • Positive and negative text help bridge the visual-text domain gap and improve fine-grained region-text alignment with a caption model.A spatial rectifier enables TAS to deal with long sentences containing orientation descriptions.Source: §5 · p. 9
  • Experiments on RefCOCO, RefCOCO+, and RefCOCOg demonstrate the effectiveness of TAS.Source: §5 · p. 9
  • Future work may improve comprehension of hard expressions over non-salient instances, potentially using large language models' reasoning ability.Source: §5 · p. 9

6 Limitations

The authors report favorable results across datasets on mIoU and oIoU, but identify limitations in mask proposals, image-text model choice, and complex expressions.

6 Limitations

  • SAM occasionally fails to generate ideal mask proposals, restricting the potential for optimal performance.Source: §6 · p. 9
  • TAS effectiveness depends on the image-text contrastive model: BLIP-2 outperforms CLIP, while ALBEF performs poorly on masked images.Source: §6 · p. 9
  • TAS may struggle with complex scenarios, noisy proposals, semantic granularity, and expressions involving metaphor or antonomasia.The authors note that some expressions require human-level comprehension and suggest LLM reasoning as a possible future direction.Source: §6 · p. 9

Related papers

Loading 2310.18049v1…