PaperlayerView exact revision on arXiv

Source-linked AI summary

Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors

Ryan Burgert, Kanchana Ranasinghe, Xiang Li, Michael S. Ryoo

arXiv:2211.13224v2cs.CVcs.CLcs.LG

TL;DR

Pretrained text-to-image diffusion models have rarely been explored for semantic localization, despite their language-conditioned image-generation capabilities. Peekaboo uses inference-time alpha-mask optimization to extract localization cues without segmentation-specific retraining. The paper evaluates it on Pascal VOC-C and RefCOCO and presents qualitative examples of open-vocabulary localization at varying granularities.

  • Problem

    Few works have explored text-to-image diffusion models for semantic localization or grounding, and unsupervised semantic segmentation methods struggle with complex language prompts.

    Source: §1 · p. 1

  • Method

    Peekaboo uses a pretrained image-language diffusion model and inference-time optimization of an alpha mask to segment image regions conditioned on natural-language prompts, without segmentation-specific retraining.

    Source: §1 · p. 1, §3.2 · p. 4

  • Results

    Peekaboo is evaluated on RefCOCO and Pascal VOC-C, with qualitative examples showing localization of cultural references and sub-regions of a person using different captions.

    Source: §1 · p. 2, §4.3 · pp. 7–8

  • Takeaways & Limitations

    The work reports that pretrained text-to-image diffusion models contain localization cues despite having no explicit localization modeling in their training process.

    Source: §6 · p. 9

    Peekaboo relies on a large diffusion model pretrained on internet-scale data, so its flaws and biases, along with those of the training data, are transferred to the method.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recently, text-to-image diffusion models have shown remarkable capabilities in creating realistic images from natural language prompts. However, few works have explored using these models for semantic localization or grounding. In this work, we explore how an off-the-shelf text-to-image diffusion model, trained without exposure to localization information, can ground various semantic phrases without segmentation-specific re-training. We introduce an inference time optimization process capable of generating segmentation masks conditioned on natural language prompts. Our proposal, Peekaboo, is a first-of-its-kind zero-shot, open-vocabulary, unsupervised semantic grounding technique leveraging diffusion models without any training. We evaluate Peekaboo on the Pascal VOC dataset for unsupervised semantic segmentation and the RefCOCO dataset for referring segmentation, showing results competitive with promising results. We also demonstrate how Peekaboo can be used to generate images with transparency, even though the underlying diffusion model was only trained on RGB images - which to our knowledge we are the first to attempt. Please see our project page, including our code: https://ryanndagreat.github.io/peekaboo

1 Introduction

The introduction asks whether pretrained diffusion models can localize language in images, then presents Peekaboo as an inference-time approach for semantic and referring segmentation without segmentation-specific retraining. It also highlights pixel-level localization, transparency-capable image generation, and evaluation on Pascal VOC-C and RefCOCO.

1 Introduction

  • Prior unsupervised semantic segmentation methods struggle with complex language prompts, while diffusion-based segmentation had required expensive training.The introduction asks whether pretrained diffusion models can associate natural language with relevant image regions.Source: §1 · p. 1
  • Peekaboo iteratively optimizes an alpha mask for an image and caption, using a novel alpha-compositing loss to produce semantic and referring segmentations without model training.The approach is based on a pretrained image-language Stable Diffusion model.Source: §1 · p. 1
  • The authors identify pixel-level localization information in pretrained text-to-image diffusion models as a contribution.Their stated goal is to extract that localization knowledge without changing the model weights.Source: §1 · p. 1
  • Figure 1 illustrates complex-prompt grounding without segmentation-specific training, including a case where the bald man's unseen arms are segmented.The authors note that Peekaboo has fairly strong shape priors.Source: Figure 1 · p. 2
  • Peekaboo is also presented as enabling end-to-end text-to-image generation of RGBA images with transparency.The underlying diffusion model was trained on RGB images.Source: §1 · p. 2
  • The approach is evaluated on RefCOCO and modified Pascal VOC (Pascal VOC-C).Figure 2 shows referring segmentation with custom prompts and examples using existing RefCOCO captions.Source: §1 · p. 2, Figure 2 · p. 2

2 Related Work

Related work spans vision-language grounding, diffusion-based image generation and optimization, unsupervised segmentation, and referring segmentation. Peekaboo is positioned as an off-the-shelf, zero-shot segmentation approach that does not require model training.

Vision-Language Models

  • Vision-language work includes zero-shot recognition, visual language generation, language-image pretraining, grounding, and unsupervised segmentation; Peekaboo handles compound phrases without segmentation-specific retraining.The passage describes contrastive language-image models [8] [9] and their extensions across tasks [26] [27] [28] [29] [30] [31] [32].Source: §2 · p. 2

Diffusion Models

  • Diffusion models have been used in language-vision generation and other applications, and recent work has sampled them through optimization; Peekaboo applies efficient latent diffusion to zero-shot segmentation.The authors characterize this as a cross-modal discriminative task.Source: §2 · p. 2
  • Figure 3 depicts alpha blending of an input image with a random background, followed by text-conditioned diffusion processing and iterative mask optimization.Figure 4 presents a timelapse of the inference-time alpha-mask optimization.Source: Figure 3 · p. 3, Figure 4 · p. 3

Score Distillation Loss

  • Peekaboo applies score distillation loss, previously used for 3D models and with Stable Diffusion, to alpha masks for image segmentation.The related-work passage reports better results than previous CLIP-based techniques [49].Source: §2 · p. 3

Unsupervised Segmentation

  • Prior unsupervised segmentation ranges from spatial-local affinity methods to self-supervised deep learning, while LSeg [7] uses ground-truth masks and CLIP embeddings to generalize dataset labels to language.Peekaboo also enables open-vocabulary grouping aligned to natural language.Source: §2 · p. 3

Referring Segmentation

  • Unlike large supervised segmentation models that require annotated datasets and an unsupervised referring method requiring 5.3 days of training on 32 NVIDIA V100 GPUs, Peekaboo performs unsupervised referring segmentation with no model training.The comparison is with method, which uses the same Stable Diffusion model.Source: §2 · p. 3

3 Proposed Method

Peekaboo optimizes a foreground alpha mask against image and text prompts using a pre-trained text-to-image Stable Diffusion model, without training a segmentation model. Its mask parametrization uses image-aware bilateral filtering, and the section describes comparisons among baselines and variants.

3 Proposed Method

  • Peekaboo formulates segmentation as foreground alpha-mask optimization conditioned on image and text prompts, using a pre-trained text-to-image Stable Diffusion model.The approach is presented as a zero-training method.Source: §3 · pp. 3–4

3.1 Background: Score Distillation Sampling

  • Score distillation sampling generates samples by optimizing a score distillation loss and can optimize learnable alpha masks in latent diffusion.The optimization applies where samples can be mapped back to images differentiably.Source: §3.1 · p. 4

3.2 Overview

  • Peekaboo optimizes a learnable alpha mask by blending an image with backgrounds and minimizing an objective that makes composites match the text prompt.The method adapts score distillation sampling to optimize masks in latent diffusion space.Source: §3.2 · p. 4, §3.1 · p. 4

3.3 Inference-time Objective

  • The inference-time Peekaboo loss combines latent score distillation with alpha regularization, which discourages unnecessary pixels and the all-ones mask.Latent score distillation measures cross-modal similarity through noise reconstruction conditioned on the text prompt.Source: §3.3 · p. 4

3.4 Alpha Mask Parametrization

  • The raster bilateral parametrization blurs a learnable alpha matrix and applies a sigmoid to produce a mask aligned with image content and boundaries.The paper reports this approach performs better than a simple learnable matrix.Source: §3.4 · p. 4

3.4.1 Peekaboo’s Bilateral Filter

  • A modified bilateral filter uses image color and spatial information to update the alpha mask at every optimization step, aiming to preserve edges and improve segmentation.Applying the filter during optimization, rather than only as post-processing, is intended to align masks to image content from early iterations.Source: §3.4.1 · p. 5

3.5 Baselines

  • The baselines include predicting the whole image as the mask, GroupViT, and LSeg, which was trained on the VOC dataset.GroupViT is described as a weakly supervised method trained on datasets similar to those used for Stable Diffusion.Source: §3.5 · pp. 5–6

3.6 Peekaboo Variants

  • Depth Bilateral uses MIDAS depth to guide the filter, performs better than the main algorithm, and outperforms Clippy on COCO and VOC-C.Raster skips bilateral filtering and performs worse than other parametrizations; Fourier improves on a basic matrix but has more hallucinations, while Fourier Bilateral improves on Fourier alone. Figure 5 compares variants qualitatively on one VOC-C image with the prompt “car”.Source: §3.6 · p. 6, Figure 5 · p. 6

4 Experiments

Peekaboo is evaluated on referring and semantic segmentation, with competitive results despite requiring no segmentation training. Qualitative examples show open-vocabulary localization and caption-dependent changes in segmentation granularity.

4.1 Referring Segmentation

  • On RefCOCO, Peekaboo outperforms GroupViT, which was trained specifically for segmentation, while LSeg outperforms Peekaboo.Table 1 reports RefCOCO precision and mIoU for Peekaboo and baselines; the supplied passages do not give the values.Source: §4.1 · p. 6, Table 1 · p. 7

4.2 Semantic Segmentation

  • On VOC-C, Peekaboo is fairly competitive but does not outperform Clippy, GroupViT, or LSeg; its Depth Bilateral variant outperforms Clippy.Peekaboo requires no training, unlike these baselines; Table 2 reports VOC-C results.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 Qualitative Results

  • Subtle caption variations can localize subregions of a person, while real-world object localizations overlap the objects’ centroids.Figure 6 notes that granularity behavior is sensitive to the caption.Source: Figure 6 · p. 7, §4.3 · p. 8
  • Examples show Peekaboo localizing regions associated with popular-cultural references, including a person’s face and body rather than her dress.The authors hypothesize that its broad vocabulary reflects the strength of the pretrained diffusion model.Source: §4.3 · p. 7, Figure 7 · p. 8

5 Generating Images with Transparency

Peekaboo’s loss can generate RGBA images with detailed transparency masks, although text-to-image models are trained on RGB images. The method iteratively optimizes an RGB image jointly with an alpha mask.

5 Generating Images with Transparency

  • Peekaboo can generate RGBA images with detailed alpha masks despite text-to-image models being trained on RGB images and not designed to generate alpha channels.The paper notes potential uses in graphic design assets and video game textures.Source: §5 · p. 8
  • The generation process iteratively optimizes a learnable RGB image and alpha mask together using Peekaboo loss and a text prompt.Algorithm 2 returns the optimized image and alpha mask.Source: §5 · p. 8
  • The optimization process is similar to score-distillation optimization of 2D images, and the authors note that it shares some caveats.Source: §5 · p. 8
  • Figures 8 and 9 show generated transparent images, including examples that separate high-frequency details such as rib bones from the background.Figure 8 presents a generation timelapse; Figure 9 shows examples on different backgrounds.Source: §5 · p. 8, Figure 9 · p. 9, Figure 8 · p. 9

6 Conclusion

The paper argues that text-to-image diffusion models contain localization cues despite lacking explicit localization modeling, and presents inference-time optimization to use those cues for segmentation without segmentation-specific retraining. Its intuition is that prompt-relevant regions receive stronger gradients, while alpha regularization leaves irrelevant regions transparent.

6 Conclusion

  • Peekaboo extracts localization cues in pretrained text-to-image diffusion models through inference-time optimization for referring and semantic segmentation, without segmentation-specific retraining.The diffusion model’s weights remain intact.Source: §6 · p. 9
  • The method relies on a large diffusion model pretrained on internet-scale data, so its flaws and biases, as well as those of its training data, transfer to Peekaboo.The authors note that resource constraints make publicly available pretrained models a dependency for approaches built on Peekaboo.Source: §6 · p. 9

A Intuition

  • Prompt-relevant regions receive stronger gradients from the conditional diffusion model, guiding the learnable alpha mask toward those regions.In the dog-on-a-couch example, the couch receives fewer gradients and alpha regularization leaves it out of the mask.Source: A Intuition · p. 14
  • Peekaboo can converge on the region most relevant to a text description, which the authors characterize as localizing the described object’s “essence.”For Emma Watson, the face can be prioritized over the dress because it is more essential to the prompt.Source: A Intuition · p. 14

B Ablations

The ablations examine alpha regularization, which encourages minimal masks by discouraging irrelevant image regions. Increasing its coefficient yields more transparent generated images and smaller segmentation masks.

B.1 Alpha Regularization

  • The alpha regularization term discourages irrelevant regions in the mask, encouraging Peekaboo to produce a minimal alpha mask.The experiments use a regularization coefficient λα = .05.Source: §B.1 · p. 14
  • The alpha-regularization ablation varies λα by scaling it relative to its normal value of .05.For example, “2x” denotes .05 × 2 = .1.Source: §B.1 · p. 14
  • Higher λα values correspond to more transparent generated images and smaller segmentation masks.The authors show these patterns in Figures 10 and 11.Source: §B.1 · p. 14

B.2 Implicit neural representations

Implicit neural representations produce less noisy outputs and converge more reliably than direct pixel representations in the reported ablation. The section also describes alpha regularization, bilateral filtering, and optimization settings used in Peekaboo.

B.2 Implicit neural representations

  • Neural-texture implicit representations parameterize learnable masks and RGB channels, while the ablation compares them with directly learned pixel tensors.The pixel-level representation is a learnable (H, W, C) tensor, with C = 1 for masks and C = 3 for RGB images.Source: §B.2 · p. 14
  • Increasing alpha regularization tends to reduce transparency and generated fries, while removing it produces more fries than normal.For segmentation, excessive alpha regularization can cause the image to be ignored, while too little can include many background details.Source: Figure 10 · p. 15, Figure 11 · p. 15
  • Around 50 iterations, implicit-representation variance converges; pixel-wise variance still fails after 350, with noisier outputs and mask non-convergence.The comparison measures pixel variance over iterations for generated images; the text also reports that pixel-level optimization can produce overly saturated images.Source: Figure 12 · p. 15, §B.2 · p. 16

B.3 Bilateral Filter

  • The modified bilateral filter is used to align masks with image boundaries; its application can leave vague skeleton outlines in a noisy mask.Figure 14 compares Fourier and Bilateral Fourier variants over optimization time.Source: §B.3 · p. 16

C Optimization Details

  • Peekaboo runs segmentation optimization for 200 iterations with learning rate 1e-5 and stochastic gradient descent.At initialization, it applies an image-conditioned modified bilateral blur to alpha masks using a size-3 kernel for 40 iterations, which the authors say results in faster and better convergence.Source: C Optimization Details · p. 16

D Limitations

Peekaboo can hallucinate prompted shapes from background texture, add inappropriate region parts, or fail to converge. The paper also presents open-vocabulary segmentation with arbitrary natural-language prompts as its contribution.

Limitations

  • Peekaboo can hallucinate an object's shape using random background texture, a failure illustrated especially in bird, cat, and dog examples.The text says this behavior is more common with simple, often one-word captions.Source: D Limitations · p. 16, Figure 15 · p. 18
  • The bilateral-filter ablation reports improved alignment with object boundaries and more accurate, precise segmentations.Figure 14 shows alpha-mask optimization over time for Fourier and Bilateral Fourier variants.Source: Figure 14 · p. 17
  • Masks can include inappropriate parts or fail to converge, retaining outlines of the foreground object despite localizing the prompt.The knife example gains a handle formed from background bread; in the eyes example, foreground outlines remain in the mask.Source: D Limitations · p. 17
  • The paper characterizes Peekaboo as an unsupervised method for open-vocabulary segmentation using arbitrary natural-language prompts.The authors state that they hope to address the limitations in future work.Source: D Limitations · p. 17

E Analagous Example

An analogous experiment uses Stable Diffusion to optimize foregrounds, backgrounds, and alpha masks, producing relevant masks despite unrealistic generated images. The authors distinguish this experiment from Peekaboo, which optimizes masks to segment stand-alone images.

E Analagous Example

  • Stable Diffusion's generated-image distribution rarely places objects away from the center, a bias the text links to Peekaboo's strength on centrally located objects.Figure 16 uses images generated with Stable Diffusion v1.4 and DDPM at guidance scale 7.Source: Figure 16 · p. 18
  • The experiment combines six learnable foregrounds, five backgrounds, and six alpha masks into 30 composite cells, optimizing all components.Each cell receives a caption combining its foreground and background prompts; this differs from Peekaboo, which optimizes only alpha masks.Source: Figure 17 · p. 19
  • The experiment explores whether pretrained diffusion-model knowledge of object boundaries and localization can support foreground segmentation through score-distillation optimization.The account describes it as analogous to, but not exactly, Peekaboo.Source: E Analagous Example · p. 19
  • The analogous experiment generates foreground-relevant masks, but the resulting images are unrealistic.The Stable Diffusion model was pretrained on LAION-5B; the experiment optimizes foreground, background, and alpha mask using score distillation loss.Source: E Analagous Example · pp. 19–20
  • The authors focus the rest of the work on segmenting stand-alone real-world images with free-form text captions, not generated-image quality.They state that image quality is indifferent to the segmentation component, which uses single-channel alpha masks.Source: E Analagous Example · p. 20

F More Results

The supplementary figures show segmentation outputs on RefCOCO-C and Pascal VOC-C, additional varied-image examples, and transparent-image generation timelapses.

F More Results

  • RefCOCO-C examples show each prompt, input image, output alpha mask, and image segmented by that mask.The figure presents these components for each sample.Source: Figure 20 · p. 21
  • Pascal VOC-C examples present prompt, input image, output alpha mask, and the image segmented by the mask for each sample.The figure uses the same visual layout as the RefCOCO-C examples.Source: Figure 21 · p. 22
  • Additional examples include pop-culture references and AI-generated imaginary objects, with prompts and inputs beside the resulting alpha masks.The figure names Avatar the Last Airbender and avocado armchairs as examples.Source: Figure 22 · p. 23
  • Transparent-image generation timelapses show generated images with transparency masks, with each prompt displayed alongside its sequence.Figures 23 and 24 continue the task shown in Figure 9.Source: Figure 23 · p. 24, Figure 24 · p. 25

Related papers

Loading 2211.13224v2…