PaperlayerView exact revision on arXiv

Source-linked AI summary

Localizing Object-level Shape Variations with Text-to-Image Diffusion Models

Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, Daniel Cohen-Or

arXiv:2303.11306v2cs.CVcs.GRcs.LG

TL;DR

Text-to-image generation lacks localized object-level shape control, making targeted exploration difficult. The paper combines prompt mixing across denoising stages with attention- and segmentation-based localization to generate diverse object variations while preserving surrounding content. Experiments and comparisons report effective shape variation and localization, with the method remaining faithful to original generated or real images.

  • Problem

    Global text-to-image generation changes entire images unpredictably, limiting focused exploration and refinement of individual objects.

  • Method

    The method mixes prompts across denoising stages and uses self-attention injection with segmentation-based preservation to localize object shape edits.

  • Results

    The method generates more diverse shape changes with better content preservation than alternatives, while supporting faithful variations of generated and real images.

  • Takeaways & Limitations

    The approach supports object-level shape exploration and extends its localization techniques to generic local image editing.

  • Takeaways & Limitations

    Automatic proxy words may sometimes fail, and token-level proxy words may lack standalone semantic meaning despite producing meaningful variations.

Abstract

from arXiv · show

Text-to-image models give rise to workflows which often begin with an exploration step, where users sift through a large collection of generated images. The global nature of the text-to-image generation process prevents users from narrowing their exploration to a particular object in the image. In this paper, we present a technique to generate a collection of images that depicts variations in the shape of a specific object, enabling an object-level shape exploration process. Creating plausible variations is challenging as it requires control over the shape of the generated object while respecting its semantics. A particular challenge when generating object variations is accurately localizing the manipulation applied over the object's shape. We introduce a prompt-mixing technique that switches between prompts along the denoising process to attain a variety of shape choices. To localize the image-space operation, we present two techniques that use the self-attention layers in conjunction with the cross-attention layers. Moreover, we show that these localization techniques are general and effective beyond the scope of generating object variations. Extensive results and comparisons demonstrate the effectiveness of our method in generating object variations, and the competence of our localization techniques.

1. Introduction

Text-to-image sampling produces global, uncontrolled changes that hinder object-level exploration. The paper introduces automatic shape variation generation with prompt mixing and attention-based localization while preserving the rest of the image.

  • Random initial noise changes object shape, appearance, location, and composition, making focused refinement of a specific object difficult.
  • Inpainting and SDEdit mainly support texture changes and struggle to alter object shape without affecting other image regions.Figure 2 illustrates these limitations for basket variations.
  • The method automatically generates a gallery of shape variations for a selected object without requiring additional user input.
  • Prompt mixing switches prompts across denoising stages, exploiting coarse-to-fine synthesis to control layout, object shape, and fine details.
  • Attention-based localization preserves other objects and background appearance through self-attention injection, segmentation, and late-stage blending.
  • Extensive experiments report more diverse shape changes and better content preservation than alternative methods, without costly optimization.

2. Related Work

Prior text-to-image systems improved synthesis quality but provided limited object-level control. Existing editing methods offer spatial or layout control, while related prompt-switching work changes appearance and preserves shape rather than enabling geometric variation.

  • Large-scale text-to-image models generate image galleries, but random initial noise can change composition, shape, color, and texture unpredictably.
  • Spatial conditions such as segmentation maps, bounding boxes, and keypoints provide spatial control but not object-level control.
  • Text-guided editing methods use noise-denoising, user masks, attention manipulation, or text-latent operations to preserve or localize image changes.
  • Prior prompt switching changes object semantics through appearance modifications while explicitly preserving shape, unlike this work’s focus on geometric modifications.
  • The paper’s pipeline combines prompt mixing with attention-based localization to generate object shape variations while retaining layout and background structure.

3. Preliminaries

The paper builds on latent diffusion models, where cross-attention injects text guidance and self-attention models spatial relationships. These mechanisms provide complementary controls for generating and localizing image edits.

  • Stable Diffusion uses a latent diffusion architecture in which diffusion operates in the latent space of a pretrained autoencoder.
  • The denoising network is a UNET with self-attention and cross-attention layers operating on noised spatial codes.
  • Cross-attention forms token-specific attention maps from text-derived keys and image-derived queries, with values carrying token information.
  • The text prompt directly affects the cross-attention keys and values, rather than the query computation.
  • Self-attention models relationships among image patches and helps preserve image layout and object shapes during editing.

4. Prompt-Mixing

The method mixes prompts across denoising intervals to generate object-shape variations while preserving selected aspects of the original image. It combines this interval-based control with proxy words and localized editing techniques.

  • 4. Prompt-Mixing: The method manipulates denoising at inference time to obtain object-level variations from a prompt and a selected object word.It requires neither optimization nor model training.
  • 4. Prompt-Mixing: Prompt-mixing uses different prompts across three denoising intervals, based on the process’s coarse-to-fine synthesis stages.The intervals are [T, T3], [T3, T2], and [T2, 0].
  • 4. Prompt-Mixing: The localization component uses self-attention injection for rough alignment and segmentation maps to apply edits selectively.It is introduced because preserving other objects and image appearance is important for localized manipulation.
  • 4.1. Denoising Diffusion Process Stages: The first interval controls layout, the second controls object shape, and the third controls fine visual details such as texture.Experiments combine prompts while using the same initial noise to analyze these controls.
  • 4.2. Object Variations: Mix-and-Match keeps the original prompt in the first and third intervals while inserting a proxy word in the shape-controlling second interval.This produces the original layout and fine details with the proxy word’s shape.
  • 4.2. Object Variations: Proxy words are semantically close tokens with substantially different shapes, selected using distances in CLIP’s text embedding space.Candidates are first ranked by token-level semantic similarity and then by their distance in the full prompt context.
  • 4.2. Object Variations: For each selected proxy word, Mix-and-Match generates an image containing a variation of the object of interest.The approach therefore turns proxy-word selection into a collection of object variations.
  • 4.2. Object Variations: Some token-level proxy words lack standalone semantic meaning, yet still produce meaningful variations in Mix-and-Match.Their usefulness is attributed to proximity in CLIP’s embedding space.

5. Edit Localization

The paper localizes object-shape edits by selectively injecting self-attention information and using cross-attention to identify object regions. It then uses self-attention clustering and noun-based labeling to preserve selected background and object appearance while generating localized variations.

  • 5.1. Attention-Based Shape Localization: Selective self-attention injection preserves shapes of chosen objects without preventing shape changes in the target object.The mask selects rows and columns associated with pixels of objects to preserve; injecting the full map would also suppress target-object changes.
  • 5.1. Attention-Based Shape Localization: The localization mask marks self-attention entries when either indexed pixel belongs to a preserved object region.For a self-attention map entry [i,j], the mask is one if i or j belongs to O_t^(l), and zero otherwise.
  • 5.1. Attention-Based Shape Localization: Cross-attention maps identify preserved-object pixels by selecting high-activation pixels for the corresponding prompt noun.The object-pixel set is obtained using a fixed threshold on the cross-attention map of the relevant noun.
  • 5.2. Controllable Background Preservation: Self-attention clustering produces semantic segments from noised latent codes, which are labeled by matching normalized cross-attention maps of prompt nouns.The method averages self-attention maps, clusters deep pixels with K-Means, and assigns a noun when its segment score exceeds a fixed threshold; otherwise the segment is background.
  • 5.2. Controllable Background Preservation: The blending procedure retains original pixels for regions labeled background or user-defined objects in both images, while taking edited regions from the generated image.This preserves selected appearance while allowing the target object's shape to change.
  • 5.2. Controllable Background Preservation: Across synthetic and real scenes, the method generates diverse object-shape variations while preserving the rest of the image.The object-variation gallery presents the original image first and marks the modified object through the emphasized prompt word.

6. Experiments

The experiments evaluate object-level shape variation against sampling-based and text-guided editing methods, using diversity, object faithfulness, and image preservation. Results show that the method produces varied shapes while preserving object identity and surrounding content, with localization improving editing outcomes.

  • Evaluation: The evaluation measures shape diversity, object faithfulness, and preservation of regions outside the object of interest.Diversity is defined as 1 − IoU between object masks, faithfulness uses CLIP similarity, and preservation uses LPIPS.
  • Object Shape Variations: The method generates diverse shape variations while preserving the original image, unlike inpainting and SDEdit, which mainly change texture or make small shape changes.SDEdit also fails to preserve the background and other objects.
  • Text-Guided Editing: The method remains faithful to the chair class and preserves the rest of the image while Mix-and-Match transfers alternative shapes without replacing fine chair details.For example, wheel shapes can be borrowed while retaining the chair’s fine visual details.
  • Text-Guided Editing: 40% self-attention injection prevents changes to the object of interest, whereas 10% struggles to preserve the dog and background.Other baselines trade off diversity, preservation, or faithfulness: higher-diversity methods lose preservation or object fidelity, while stronger-preservation methods barely change shape.
  • Ablation Studies: Ablations show that Mix-and-Match alone fails to preserve the dog and background, motivating the attention-based localization and segmentation components.The localization techniques were also integrated with existing editing methods to improve their results.

7. Discussion and Conclusion

The paper concludes that object-level shape exploration requires both shape variation and localized editing. It combines Mix-and-Match, self-attention preservation, and self-segmentation, while acknowledging proxy-word failures and future continuous-word exploration.

  • Discussion and Conclusion: The method addresses object shape changes and localization through Mix-and-Match, self-attention injection, and self-segmentation.Self-segmentation uses geometric information in self-attention maps to guide background preservation.
  • Discussion and Conclusion: Automatic proxy words can fail, and future work aims to explore a continuous word space instead of the current discrete one.

Appendices

The appendices detail prompt-mixing, cross-attention injection, proxy-word selection, and additional comparisons. They show that modifying cross-attention Values preserves layouts better, while closer proxy words generally yield more successful shape variations.

  • Prompt-Mixing: Prompt-mixing switches prompts during denoising, with colored bars indicating which word is used at each stage.
  • Cross-Attention Injection: Using the altered prompt for Values while retaining the original prompt for Keys preserves the original layout while changing object appearance.Using altered Keys and original Values produces less desirable results and can resemble generation from the altered prompt alone.
  • Proxy Words: Proxy words navigate prompt-embedding space to produce shape variations, although occasional unexpected words can still generate convincing results.Adding noise to prompt embeddings or cross-attention Values performed significantly worse and reduced diversity.
  • Proxy Words: Proxy words closer to the original word generally produce more successful variations than words with larger CLIP distance.
  • Additional Comparisons: Additional comparisons include different self-attention masks integrated with P2P and examples contrasting original prompts with object-level variations.

A.2. Attention-Based Shape Localization

Attention-based localization selectively injects self-attention information for objects to preserve while allowing the target object to change. Self-segmentation derives object regions from diffusion features, supporting localization without an external segmentation model.

  • Self-Attention Injection Mask Controls: The self-attention mask contains rows and columns corresponding to pixels of the object being preserved.The paper reports that the mask containing both rows and columns is the most robust in experiments.
  • Self-Attention Injection Mask Controls: Injecting only rows preserves the object’s influence on image pixels while allowing its pixels to change their effects on other regions.This can allow pixels formerly occupied by another object, such as a hat, to become part of the preserved object in the new image.
  • Self-Attention Injection Mask Controls: Removing the target object’s pixels from the preservation mask releases that object from its connections to the object being preserved.Whether to remove those pixels depends on the user’s preferences.
  • Self-Segmentation: Self-segmentation uses the generative prior and internal diffusion features instead of an external segmentation model.It is demonstrated on challenging cases including a transparent fish fin and a non-realistic cup sketch.
  • Implementation: The method uses 50 denoising steps, with prompt-mixing boundaries near T3 = 44 ± 1, T2 = 34 ± 1, and T1 = 15 ± 1, while optimal mixing ranges vary slightly by prompt and seed.Segmentation experiments use five clusters and σ = 0.3 for cluster labeling; the method requires no optimization.

B.2. Evaluation Setup for Alternative Methods

The evaluation compares non-deterministic editing methods and text-guided methods using naïve seed changes and automatically generated proxy words. Text-guided baselines create variations by refining prompts or replacing the explored object.

  • The study compares SDEdit and Inpainting as non-deterministic baselines, alongside Prompt-to-Prompt and Instruct-Pix2Pix as text-guided editing methods.
  • For non-deterministic methods, the evaluation uses both different seeds and proxy words replacing the explored object in the original prompt.
  • Text-guided baselines generate variations by adding adjectives, replacing objects with proxy words, or issuing explicit edit prompts that request preservation of other objects.
  • The dataset contains 150 initial images from 15 prompts, with 20 object variations generated for each initial image and method.

C.1. Comparison to Other Methods

Compared with alternative editing methods, the proposed method produces larger shape changes while preserving image content and object semantics. Baselines often make texture-only changes, lose background fidelity, or drift from the target domain.

  • The method produces diverse tent shapes while preserving the tent class and the rest of the image, whereas Plug-n-Play and Instruct-Pix2Pix mainly change texture.
  • Zero-shot Image2Image Translation struggles to change tent shape, while Imagic struggles to preserve the hamster and background and drifts from the tent domain.

C.2. Our Method Results

Additional examples show object-level shape variations across real and synthetic scenes and across multiple explored objects. The method changes the selected object's shape while preserving the surrounding image content.

  • The pipeline generates lamp shape variations in both a real image and a different synthetic scene.
  • Figure 20 compares results produced by replacing the original “tent” prompt word with different words.
  • Across various scenes, the method generates different shape variations for each emphasized, explored object.
Loading 2303.11306v2…