PaperlayerView exact revision on arXiv

Source-linked AI summary

Ref-Diff: Zero-shot Referring Image Segmentation with Generative Models

Minheng Ni, Yabo Zhang, Kailai Feng, Xiaoming Li, Yiwen Guo, Wangmeng Zuo

arXiv:2308.16777v2cs.CV

TL;DR

Zero-shot referring image segmentation seeks instance masks from descriptions without paired training data, yet requires understanding relationships between text and visual elements. Ref-Diff uses generative models for fine-grained multimodal information and instance proposals. Generative-only Ref-Diff is comparable to SOTA weakly supervised methods, while combining generative and discriminative models significantly outperforms competing methods.

  • Problem

    Zero-shot referring image segmentation seeks instance masks from descriptions without paired training data, but requires deeper understanding of relationships between text and image elements.

    Source: §1 · p. 1

  • Method

    Ref-Diff leverages generative models' fine-grained multimodal information and can provide instance proposals without a third-party proposal generator.

    Source: §1 · p. 2

  • Results

    Generative-only Ref-Diff achieves comparable performance to SOTA weakly supervised methods, while combining generative and discriminative models significantly outperforms competing methods.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings indicate that generative models benefit zero-shot RIS and can complement discriminative models for better referring segmentation.

    Source: §1 · p. 2

    Inference has high computational overhead due to pretrained modules, and ambiguous referring expressions result in noticeable segmentation errors.

    Source: §5 · p. 9

Abstract

from arXiv · show

Zero-shot referring image segmentation is a challenging task because it aims to find an instance segmentation mask based on the given referring descriptions, without training on this type of paired data. Current zero-shot methods mainly focus on using pre-trained discriminative models (e.g., CLIP). However, we have observed that generative models (e.g., Stable Diffusion) have potentially understood the relationships between various visual elements and text descriptions, which are rarely investigated in this task. In this work, we introduce a novel Referring Diffusional segmentor (Ref-Diff) for this task, which leverages the fine-grained multi-modal information from generative models. We demonstrate that without a proposal generator, a generative model alone can achieve comparable performance to existing SOTA weakly-supervised models. When we combine both generative and discriminative models, our Ref-Diff outperforms these competing methods by a significant margin. This indicates that generative models are also beneficial for this task and can complement discriminative models for better referring segmentation. Our code is publicly available at https://github.com/kodenii/Ref-Diff.

1 Introduction

Zero-shot referring image segmentation avoids paired training annotations but demands deeper understanding of relationships between referring text and image elements. Ref-Diff uses generative models for fine-grained information and proposals, and reports comparable performance with generative models alone and stronger results when combined with discriminative models.

1 Introduction

  • Precise image, referring-text, and instance-mask annotations are costly, while zero-shot RIS requires deeper understanding of relationships between text and visual elements.Unlike weakly supervised RIS, zero-shot RIS requires no referring annotations or training on paired data.Source: §1 · p. 1
  • CLIP's global image–text similarity does not capture the specific visual elements associated with referring text well, and generative models' potential for this task has rarely been explored.The paper motivates generative models by their implicit capture of relationships between visual elements and text.Source: §1 · pp. 1–2
  • Ref-Diff leverages fine-grained multimodal information from generative models and can provide instance proposals without depending on third-party proposal generators.The paper also proposes combining generative and discriminative models, which it describes as complementary for referring segmentation.Source: §1 · p. 2
  • Ref-Diff with generative and discriminative models significantly outperforms competing methods; the generative model alone achieves comparable performance to SOTA weakly supervised methods.The generative-only result does not require an offline proposal generator.Source: §1 · p. 2

2 Related Work

Prior referring image segmentation methods rely on labor-intensive annotations, while open-vocabulary discriminative models have limitations in fine-grained relations and object delineation. Generative models show fine-grained understanding in other vision tasks, but their use in zero-shot referring segmentation has received little investigation.

Zero-shot Referring Segmentation

  • Fully supervised RIS methods require labor-intensive referring-expression and pixel-mask annotations, and limited large-scale training annotations constrain scalability and out-of-domain samples.Recent work explores discriminative vision-language pretraining for weakly or zero-shot RIS, but the cited models have limits in object delineation and fine-grained text–visual relationships.Source: §2 · p. 2
  • Discriminative vision-language models learn global text–image similarity, which limits their fine-grained understanding of object delineation and relationships between visual elements and descriptions.The passage contrasts this limitation with Ref-Diff's use of generative models for fine-grained understanding.Source: §2 · p. 2

Visual Generative Models for Non-Synthesized Tasks

  • Text-to-image generative models have shown fine-grained capabilities in segmentation, detection, dense prediction, and classification, but their performance in zero-shot referring segmentation has not been well investigated.Much existing research focuses on transfer learning or synthetic data for specific tasks.Source: §2 · p. 2

3 Methodology

REF-DIFF uses generative-model correlations to produce segmentation candidates, optionally combining them with discriminative-model candidates. Its pipeline derives attention-based proposals and ranks them using generative and discriminative matching.

3.1 Problem Formulation and Inference Pipeline

  • In 3.1, REF-DIFF generates a text-image correlation matrix for weight-free proposals and candidates, optionally combines discriminative candidates, and selects the highest-similarity proposal.The task outputs an instance mask for the referring text without access to referring-segmentation training data.Source: §3.1 · pp. 2–3, Figure 1 · p. 3

3.2 Generative Process

  • In 3.2, the model adds Gaussian noise to the image, then uses text-image cross-attention and the syntactic ROOT token’s attention map to locate referring regions.The token-region correlation map is resized to the image resolution.Source: §3.2 · pp. 3–4

3.3 Discriminative Process

  • In 3.3, REF-DIFF applies a direction-conditioned spatial bias and uses proposal-masked CLIP self-attention to represent each candidate region.The CLIP representation uses the penultimate layer, whose features focus more on the proposal than the last layer’s image-wide representation.Source: §3.3 · p. 4

3.4 Proposals Extracting and Matching

  • In 3.4, thresholded generative cross-attention supplies weight-free mask proposals, while an available pre-trained segmentor can also supply proposals for generative and discriminative matching.Generative matching compares proposals with the cross-attention map; discriminative matching compares proposal representations with text features.Source: §3.4 · pp. 4–5

4 Experiments

Experiments evaluate REF-DIFF across benchmarks, ablations, component analyses, attention visualizations, and case studies. They report strong generative-only performance and improved results when generative and discriminative models are combined.

4.1 Implemention Details; 4.2 Experimental Setup

  • Experiments use inference only, Stable Diffusion V1.5, and 1024 × 1024 test images; evaluation uses mIoU and oIoU on three RIS benchmarks.The comparison settings include zero-shot RIS with a pre-trained segmentor and CLIP, and without them; SAM-CLIP uses SAM proposals ranked by CLIP.Source: §4.1 · p. 5, §4.2 · pp. 5–6

4.2 Experimental Setup

  • Approximately 10 mIoU improvement is reported for REF-DIFF over competing methods on RefCOCO, RefCOCO+, and RefCOCOg.The mIoU comparison is reported as statistically significant at p < 0.01 under a t-test.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 Ablation Study

  • REF-DIFF/G matches weakly supervised TSEG [23] on RefCOCO+ and exceeds it on RefCOCOg, while REF-DIFF/GS outperforms it across all test sets.Tables 4 and 5 report the ablation results for oIoU and mIoU.Source: §4.3 · p. 7, Table 4 · p. 7, Table 5 · p. 7

4.4 Effect of Generative Model

  • REF-DIFF/G segments referring content without a segmentor or CLIP, including a case with two instances of the same object class.The authors attribute this capability to generative-model attention on relevant visual content.Source: §4.4 · p. 7

4.5 Effect of Discriminative Model

  • Combining generative and discriminative models achieves the best results for localization, where the discriminative model can miss the leftmost broccoli or fail to eliminate a redundant plate region.Tables 1 and 2 compare oIoU and mIoU, respectively, for settings using a pre-trained segmentor and CLIP; improvements are statistically significant at p < 0.01 under a t-test.Source: Table 1 · p. 6, Table 2 · p. 6, §4.4 · p. 7
  • The discriminative model mitigates errors when the generative model attends to salient parts, such as a laptop screen or a person’s face.The authors attribute this to the discriminative model’s robust categorization capability.Source: §4.5 · p. 7, Figure 4 · p. 8

4.6 Attention from Generative Model

  • Generative-model attention varies by token, and attention to the subject token closely aligns with final segmentation results.The authors report that this token captures the comprehensive information from the referring text, unlike the first token in their model.Source: Figure 5 · p. 8, §4.6 · p. 9

4.7 Case Study

  • Case studies show accurate segmentation amid similar objects and complex spatial relationships, but ambiguity in a referring expression causes a failure.In the failure case, REF-DIFF mistakes the leftmost hand for the first arm.Source: §4.7 · p. 9, Figure 6 · p. 9

5 Broader Impact and Limitations

REF-DIFF is presented as training-free, high-quality referring segmentation for data-scarce scenarios, with potential applications including image editing, robot control, and human-machine interaction. Its current limitations are high inference computational overhead and errors from ambiguous referring expressions.

5 Broader Impact and Limitations

  • REF-DIFF demonstrates the feasibility of training-free, high-quality referring segmentation in data-scarce scenarios, with applications such as image editing, robot control, and human-machine interaction.The authors state that this significantly reduces AI deployment costs in related fields.Source: §5 · p. 9
  • Pre-trained modules cause high computational overhead at inference, and ambiguity in referring expressions results in noticeable segmentation errors.The authors identify lower computational cost and greater robustness as future work.Source: §5 · p. 9
  • The authors identify image editing, robot control, and human-machine interaction as potential application domains for zero-shot referring segmentation.Source: §5 · p. 9

6 Conclusion

REF-DIFF leverages fine-grained multimodal information from generative models for zero-shot referring image segmentation. A generative model alone performs comparably to state-of-the-art weakly supervised methods without a proposal generator, while combining generative and discriminative models outperforms competing methods by a significant margin.

6 Conclusion

  • Combining generative and discriminative models lets REF-DIFF outperform competing methods by a significant margin.Source: §6 · p. 9
  • A generative model alone achieves performance comparable to state-of-the-art weakly supervised models without requiring a proposal generator.Source: §6 · p. 9
  • REF-DIFF uses fine-grained multimodal information from generative models, offering a promising direction for zero-shot referring image segmentation.The authors state that this work brings new insights for addressing the task's challenges.Source: §6 · p. 9

Related papers

Loading 2308.16777v2…