PaperlayerView exact revision on arXiv

Source-linked AI summary

Directed Diffusion: Direct Control of Object Placement through Attention Guidance

Wan-Duo Kurt Ma, J. P. Lewis, Avisek Lahiri, Thomas Leung, W. Bastiaan Kleijn

arXiv:2302.13153v3cs.CVcs.GRcs.LG

TL;DR

Text-to-image diffusion models often struggle to place multiple objects in specified positional relationships, limiting their use for visual storytelling. Directed Diffusion optimizes cross-attention maps during early denoising to place prompt-specified objects within user-provided bounding boxes. It provides coarse multi-object positional control with contextual blending while using a pre-trained model and requiring only lightweight implementation changes.

  • Problem

    Text-to-image diffusion models often struggle to synthesize multiple objects with desired positional relationships, a capability needed for image-based storytelling.

  • Method

    Directed Diffusion edits selected prompt-word cross-attention maps during early denoising to inject activation into user-specified bounding boxes, then uses conventional denoising.

  • Results

    Directed Diffusion provides coarse positional control over multiple objects while maintaining contextual interaction with the background using a pre-trained model.

  • Takeaways & Limitations

    The approach is a step toward visual storytelling with positioned objects and can be added to an existing text-driven diffusion model with only a few lines of code.

  • Takeaways & Limitations

    The method inherits Stable Diffusion’s trial-and-error dependence, can fail for some random seeds, may produce distorted subjects, and does not strictly constrain objects to bounding boxes.

Abstract

from arXiv · show

Text-guided diffusion models such as DALLE-2, Imagen, eDiff-I, and Stable Diffusion are able to generate an effectively endless variety of images given only a short text prompt describing the desired image content. In many cases the images are of very high quality. However, these models often struggle to compose scenes containing several key objects such as characters in specified positional relationships. The missing capability to ``direct'' the placement of characters and objects both within and across images is crucial in storytelling, as recognized in the literature on film and animation theory. In this work, we take a particularly straightforward approach to providing the needed direction. Drawing on the observation that the cross-attention maps for prompt words reflect the spatial layout of objects denoted by those words, we introduce an optimization objective that produces ``activation'' at desired positions in these cross-attention maps. The resulting approach is a step toward generalizing the applicability of text-guided diffusion models beyond single images to collections of related images, as in storybooks. Directed Diffusion provides easy high-level positional control over multiple objects, while making use of an existing pre-trained model and maintaining a coherent blend between the positioned objects and the background. Moreover, it requires only a few lines to implement.

1 Introduction

Text-to-image diffusion models can generate diverse images but often fail to compose multiple objects in specified positional relationships, limiting their use for visual storytelling. Directed Diffusion addresses this gap with coarse bounding-box control using cross-attention guidance, while preserving contextual interactions and requiring minimal implementation effort.

  • Text-to-image diffusion models often fail on complex prompts containing multiple objects, including missing objects, incorrect relationships, or unspecified placement.
  • Directed Diffusion edits cross-attention maps for selected prompt words during early denoising to concentrate activation within user-specified bounding boxes.
  • Directed Diffusion uses the available pre-trained Stable Diffusion model and is intended to complement methods that provide specific object or character identities.
  • The method provides coarse positional control over multiple objects for storytelling and compositionality without requiring exact pixel-level placement.
  • Positioned objects blend consistently with their environment because bounding boxes guide placement while subsequent diffusion fills details and jointly produces lighting and interactions.
  • Approximate bounding boxes replace detailed masks, and the method requires no training or fine-tuning, only a small optimization over a weight vector with d < 77.

2 Related Work

Prior work extends diffusion models through denoising, text conditioning, editing, layout guidance, and storytelling systems, but these approaches differ in supervision, guidance requirements, and positional control. Directed Diffusion targets open-set, zero-shot object placement with a pre-trained model and lightweight guidance.

  • Diffusion models generate images by progressively removing Gaussian noise, while text-to-image systems condition generation on text representations through mechanisms such as cross-attention.
  • Existing image-editing approaches use masks, blending, or related guidance mechanisms, but detailed shape guidance can be laborious and may produce alignment or interaction problems.
  • Layout-to-image methods position objects but often rely on closed-set training supervision, limiting placement of categories absent from training.
  • Directed Diffusion aims for high-level open-set, zero-shot placement of several objects using a pre-trained model without requiring computational resources beyond those available to typical users.
  • Related systems provide robust multi-object placement, but differ in training requirements, masks, or model design; Directed Diffusion intentionally avoids detailed shape masks.
  • Storytelling approaches can condition later images on previous captions and images, yet do not necessarily control subject positions needed for position-based storytelling.

3 Method

Directed Diffusion controls object placement by editing cross-attention maps during early denoising and then completing generation with conventional Stable Diffusion denoising. It uses prompt-word indices and bounding boxes, with Gaussian activation and optimization designed to preserve learned text-image associations.

  • The method exploits the observation that object position and coarse shape emerge early in denoising, while later steps add object-identifying details.
  • Cross-attention maps have a spatial interpretation linking image locations to prompt-word meanings, allowing selected words to guide objects toward desired regions.
  • Directed Diffusion takes a prompt and region information R = {B, I}, where B specifies bounding boxes and I identifies directed prompt words, and uses a pre-trained latent diffusion model without fine-tuning.
  • Attention Editing: Attention Editing modifies cross-attention during the first N denoising steps, amplifying activation inside each box and weakening surrounding regions; N was set to 10 in most experiments.
  • Conventional SD Denoising: After attention editing, Conventional SD Denoising runs classifier-free guidance over the remaining reverse-diffusion steps, differing only in whether cross-attention is edited.
  • Cross-Attention Map Guidance: Bounding boxes use Gaussian activation windows, so the resulting object shape is determined jointly with the U-net activation and may extend beyond the box.
  • Cross-Attention Map Guidance: The optimization adjusts attention-related weights rather than directly forcing latent variables, because direct latent-energy guidance was unstable and could leave the training range.

4 Applications

The applications extend Directed Diffusion with scene compositing for multiple objects and placement finetuning for repositioning objects while preserving identity and interactions.

  • 4.2 Placement Finetuning: Placement finetuning lets artists move an object after generating an image while preserving its identity, without model finetuning or additional optimization.The method is intended for experimenting with positions, not for producing video sequences requiring control of pose or viewpoint.
  • 4.2 Placement Finetuning: Placement finetuning thresholds a final cross-attention map inside the bounding box, transforms the resulting object mask and latent, fills background holes, and composites the transformed latent during denoising.A single noise-and-denoising iteration blends inpainted areas with the background before subsequent compositing steps.
  • 4.2 Placement Finetuning: With N generally set to 10, larger N better preserves foreground and background while smaller N encourages more interaction between them.This parameter controls the trade-off between preserving original regions and blending the moved object with its environment.

5 Experiments and Comparisons

Experiments compare Directed Diffusion with existing methods for object visibility, placement, compositing, and interactions. The reported results show comparable or better CLIP similarity and qualitatively coherent object-background and object-object relationships, while language-driven interactions remain limited by CLIP understanding.

  • 5 Experiments and Comparisons: Directed Diffusion achieves CLIP similarities similar to or better than the compared methods across the reported experimental categories.Table 1 reports CLIP scores for the categories corresponding to Figures 4a, 4, and 6.
  • 5.1 Comparison: Scene compositing: Compared with CD, BOXDIFF, and GLIGEN, Directed Diffusion makes directed objects more visible, better preserves prompt fidelity, and reduces information blending between objects.Examples include clearer castles, cherry blossoms, churches, and trees, with fewer incorrect color mixtures.
  • 5.1 Comparison: Scene compositing: Against BLD, Directed Diffusion produces realistic object-background interactions such as contact, shadows, occlusion, and hand-object interaction.The comparison uses BLD images reproduced from its original paper and a guessed prompt for roughly similar images.
  • 5.4 Placement Finetuning: Placement finetuning preserves object identity while moving castles, dogs, birds, and spheres, including coherent reflections and waves in the surrounding environment.The castle experiment compares sliding bounding boxes with the placement-finetuning alternative.
  • 5.5 Comparison: object interactions from prompt verbs: For two-object scenes, Directed Diffusion produces natural relations such as a running dog touching a ball and a cube and sphere supported in a shopping basket.Figure 6 evaluates prompts describing both spatial and action relationships against Stable Diffusion, GLIGEN, and BOXDIFF.
  • 5.5 Comparison: object interactions from prompt verbs: The “chasing” prompt conveys action more effectively than a prompt merely naming a dog and ball, but this capability is bounded by CLIP’s limited language understanding and may fail for complex sentences.Both prompts consistently produce the dog and ball, while the action prompt more often depicts the dog running rather than sitting.

6 Limitations and Conclusion

Directed Diffusion aims to control object placement for storytelling while remaining simple to implement. The method can convey interactions, but retains trial-and-error requirements and limitations that constrain broader applications.

  • Conclusion: Directed Diffusion requires only a few lines of modification to the widely used Diffusers library.
  • Conclusion: The “chasing” prompt more frequently depicts the dog running than sitting, showing that wording can sometimes convey interaction between objects.
  • Limitations: Directed Diffusion inherits trial-and-error exploration, can produce missing or distorted objects, and requires choosing the number of editing steps.
  • Limitations: Objects may extend outside their specified bounding boxes, while forcing strict box adherence can produce unnatural appearances.
  • Limitations: Significant additional advances are needed before video storytelling, although the method may support storybooks and comic books with other tools.

8 Implementation

The implementation uses pretrained Stable Diffusion with standard diffusion components and switches from attention editing to refinement during denoising. The switching point is selected from latent-change behavior and is usually set empirically to N=10.

  • Implementation: The implementation uses PyTorch 2.01, Diffusers 0.14, pretrained stable-diffusion-v1-4, guidance scale 7.5, and a Linear Multistep Scheduler.
  • Editing schedule: The gradient norm identifies an initial phase of rapidly changing latents, followed by slower refinement as fine details are filled in.
  • Editing schedule: Editing operates during the initial yellow phase, while conventional Stable Diffusion refinement operates during the subsequent green phase.
  • Editing schedule: N=10 is empirically selected for nearly all experiments as the switch from editing to refinement.
  • Optimization: Adam optimizes the main objective with learning rate 0.0005, while c=0.15 generally initializes the vector a.

9 Analysis and Ablation

The analysis explains why Directed Diffusion edits trailing attention maps and evaluates their role through ablations and placement examples. Optimization avoids the instability and grid search required by direct activation injection, while experiments demonstrate quadrant and horizontal placement control.

  • Analysis and Ablation: Trailing attention maps are the non-prompt-token maps among the 77 possible CLIP tokens and are edited by injecting Gaussian-falloff activations before optimization.
  • Analysis and Ablation: Cross-attention maps compare latent queries with token keys, so manipulating trailing maps changes the conditioning used for classifier-free text guidance.
  • Analysis and Ablation: With only five edited trailing maps, removing optimization barely modifies the generated image, while intermediate editing settings produce the best results.
  • Analysis and Ablation: Optimization over trailing attention maps replaces unstable direct injection and avoids the grid search previously needed to obtain good results.
  • Placement experiments: Directed objects can be placed in four quadrants or the image center, and can be moved from left to right using a bounding box spanning the full height and 40% width.
  • Placement experiments: Examples include directing a large cabin, insect robot, sun, stone castle, white cat, and white dog, with contextual variations such as sun reflections.

10 Placement finetuning

Placement finetuning refines a generated image after moving a directed object while preserving subject content and environmental consistency. It uses attention-derived foreground and background masks, moves the foreground, inpaints the displaced region, and denoises the composite.

  • 10 Placement finetuning: Placement finetuning refines Directed Diffusion or Stable Diffusion results without changing subject content while preserving environmental context consistency.
  • 10 Placement finetuning: The method derives background, original-foreground, and transformed-foreground masks from the directed object’s cross-attention map.
  • 10 Placement finetuning: The background mask inpaints the area left by the moved object, while the foreground masks extract and shift the object latent by translation.
  • 10 Placement finetuning: The composited latent is denoised after a single edit at time step T−N to keep the repositioned object fixed while blending it with the background.

11 Scene compositing

Directed Diffusion composites separately directed objects into scenes while preserving contextual backgrounds and object interactions. The scene-compositing examples show control over two objects without modifying the pretrained Stable Diffusion model.

  • 11 Scene compositing: DD directs two objects, such as a red cube and blue sphere or a bear and bird, using compositing and subsequent conventional SD denoising.The compositing procedure combines directed-object latents with their bounding boxes before denoising the resulting latent.
  • 11 Scene compositing: The method retains prompt-consistent backgrounds and natural object-environment relationships, including textures and shadows.The blue sphere’s shadow is cited as an example of contextual alignment.
  • 11 Scene compositing: DD preserves relationships between directed objects, including the “watching” interaction between the bear and bird.
  • 11 Scene compositing: Stable Diffusion comparison examples include missing objects, incorrect relations, and ignored positional terms such as “upper left.”The supplement also illustrates user difficulty producing realistic placement masks, motivating a simpler guidance mechanism.

12 Qualitative Evaluations

Directed Diffusion is qualitatively compared with BoxDiff, GLIGEN, and Multidiffusion across scene compositing, one-object, and two-object tasks. The methods are broadly comparable overall, while Directed Diffusion sometimes performs better on object attributes, composition, missing or incorrect objects, and interactions.

  • 12 Qualitative Evaluations: Directed Diffusion and three concurrently developed competing methods produce broadly comparable qualitative results, with Directed Diffusion sometimes performing better on several specific failure modes.Reported advantages include reduced color bleeding, missing or incorrect objects, implausible composition and layout, incorrect object color, and object-object interaction.
  • 12 Qualitative Evaluations: The qualitative comparison covers scene compositing, one-object interactions with undirected subjects, and two-object control with an optionally specified background.These tasks examine interactions among directed objects, other subjects, and the environment.
  • 12 Qualitative Evaluations: The Scene Compositing task tests whether descriptive attributes remain associated with their intended objects within a holistic prompt.The example assigns “white” to the church and “pink” to the sky.
  • 12 Qualitative Evaluations: The One Object task evaluates whether a directed object remains spatially controlled while other subjects retain realistic positioning, interaction, and lighting.The bonfire-and-man example requires both the specified bonfire placement and a realistic relationship with the man.
  • 12 Qualitative Evaluations: Each experiment presents multiple random seeds for one prompt, enabling visual comparison of methods’ quality and failures.The comparison includes reproduced results from public GLIGEN, BoxDiff, and Multidiffusion repositories.

13 Quantitative Evaluation and Limitations

The paper discusses why common generative-model metrics are imperfect for position-directed generation. CLIP score is favorable but incomplete, IOU can conflict with natural object interactions, and FID is poorly aligned with placement-focused innovation.

  • 13 Quantitative Evaluation and Limitations: Common quantitative metrics are not fully appropriate for evaluating position-directed generation because the task emphasizes object placement, interactions, and attribute binding.The section explicitly examines limitations of standard metrics rather than treating them as sufficient evidence.
  • 13 Quantitative Evaluation and Limitations: CLIP score results are favorable, but CLIP score is not considered the best measure because Stable Diffusion struggles with multi-object prompts and attribute binding.The metric evaluates whether the generated image reflects the prompt content, yet may not capture the task’s specific failures adequately.
  • 13 Quantitative Evaluation and Limitations: Good IOU can conflict with natural object interactions when objects overlap or occlude one another.In a dog-chasing-a-ball example, partial occlusion can reduce estimated dog segmentation overlap even while producing a more natural interaction.
  • 13 Quantitative Evaluation and Limitations: FID is poorly aligned with this task because the method changes object placement using a fixed pretrained model rather than modifying generative-model weights.FID also compares representation distributions from a position-invariant classifier, which does not directly target placement quality.

14 Societal Impact

The paper acknowledges misuse risks from generating misleading images and notes that generative models can reflect biases in their training data. The supplied examples also document how competing systems and Directed Diffusion behave across varied object-placement scenes.

  • Generative models reflect biases in their training data and may be misused to generate misleading images.The authors state that these risks should be addressed, including through measures such as penalizing malicious behavior.
  • Directed Diffusion produces natural-looking placements in several challenging scenes, including church, tree, gravestone, dog-and-ball, and cube-and-sphere examples.The reported advantages include natural church orientations, consistently placed gravestones, natural dog-and-ball results, and strong alignment for the cube-and-sphere prompt.
  • The comparison includes failures and trade-offs: excessive direction can remove surrounding context, while some outputs show color leakage or missing backgrounds.The paper links these behaviors to overly strong guidance and reports occasional red-color leakage and missing backgrounds in competing or directed results.
  • Intersection over Union favors GLIGEN numerically, but the paper cautions that this metric does not correlate with natural composition or object-environment interaction.Average IoU is 0.51 for BoxDiff, 0.87 for GLIGEN, and 0.53 for Directed Diffusion.
Loading 2302.13153v3…