Source-linked AI summary
Diffusion Self-Guidance for Controllable Image Generation
Dave Epstein, Allan Jabri, Ben Poole, Alexei A. Efros, Aleksander Holynski
TL;DR
Text prompts cannot precisely express many desired image properties, motivating more direct control of diffusion generation. Self-guidance steers pretrained-model attention and activations without additional models or training, enabling composed edits of generated and real images. Its scope is bounded because cross-attention requires target objects to be named in the prompt and can hinder disentanglement between interacting objects.
Problem
Text descriptions struggle to convey precise image properties, while existing controls may require expensive fine-tuning, optimization, or auxiliary resources.
Method
Self-guidance constrains intermediate activations and attention maps inside a pretrained text-to-image diffusion model to steer prompt-named entities using user-specified or transferred properties.
Results
Self-guidance enables granular, disentangled zero-shot manipulation of object shape, position, appearance, attributes, interactions, and real images.
Takeaways & Limitations
A small set of internal representation properties can be composed for complex image manipulations, including appearance transfer, layout rearrangement, object composition, and real-image editing.
Takeaways & Limitations
Cross-attention restricts control to objects described in the text prompt and can hinder fully disentangled control between interacting objects.
Abstract
from arXiv · showhide
Large-scale generative models are capable of producing high-quality images from detailed text descriptions. However, many aspects of an image are difficult or impossible to convey through text. We introduce self-guidance, a method that provides greater control over generated images by guiding the internal representations of diffusion models. We demonstrate that properties such as the shape, location, and appearance of objects can be extracted from these representations and used to steer sampling. Self-guidance works similarly to classifier guidance, but uses signals present in the pretrained model itself, requiring no additional models or training. We show how a simple set of properties can be composed to perform challenging image manipulations, such as modifying the position or size of objects, merging the appearance of objects in one image with the layout of another, composing objects from many images into one, and more. We also show that self-guidance can be used to edit real images. For results and an interactive demo, see our project page at https://dave.ml/selfguidance/
1 Introduction
Large-scale text-to-image models make realistic images, but text alone cannot precisely communicate many desired image properties. Self-guidance addresses this gap with zero-shot, disentangled control over object shape, position, appearance, and complex image compositions.
- Text prompts can be surprisingly difficult for specifying image details that are hard or impossible to convey precisely.
- Existing control methods often require expensive paired-data fine-tuning or costly optimization, limiting the scope of edits.
- Modern large-scale diffusion models had not convincingly demonstrated precise edits such as moving or resizing one object while preserving the rest of a scene.
- Self-guidance uses pretrained diffusion-model internals to provide zero-shot, direct control over object shape, position, and appearance.
- Composing a small set of extracted properties enables complex manipulations, including object replacement, appearance transfer, scene rearrangement, and real-image editing.
2 Background
Diffusion models generate images by iteratively denoising noise, and guidance modifies this process to impose conditioning or other objectives. Self-guidance identifies useful control signals inside the pretrained denoiser, using attention and activations to steer named entities without auxiliary models or annotated data.
- 2.1 Diffusion generative models: A diffusion model transforms random noise into an image through sequential sampling that reverses a noise-adding process.
- 2.1 Diffusion generative models: The denoising network estimates the noise added to a noisy image, with text or another signal supplied as conditioning.
- 2.1 Diffusion generative models: Sampling alternates between predicting noise and updating the noisy image, using methods such as DDPM or DDIM.
- 2.2 Guidance: Guidance alters the sampling update direction by composing score functions or adding an energy gradient to impose auxiliary conditions.
- 2.3 Where can we find signal for controlling diffusion?: Self-guidance avoids auxiliary models and annotated control data by using structural information in attention maps and coarse appearance information in network activations.
- 2.3 Where can we find signal for controlling diffusion?: The method constrains intermediate activations and attention maps, with user-specified or image-transferred signals controlling entities named in prompts.
3 Self-guidance
Self-guidance extracts object properties from diffusion-model attention and activations, then constrains these internal representations during sampling. Position, size, shape, and appearance targets can be user-specified or transformed and transferred from other images.
- Core representation: Self-guidance manipulates attention channels and intermediate activations from the denoising pass to control objects named in text prompts.The guidance terms are added to the denoising objective using representations internal to the pretrained diffusion model.
- Object properties: Object position is represented by an attention-channel centroid, enabling absolute or relative translations.For example, “burger” can be guided toward (0.3, 0.5) or shifted right by (0.1, 0.0).
- Object properties: Object size is computed by spatially summing a differentiably thresholded attention map, supporting absolute or relative resizing.Soft thresholding reduces background-noise effects before size computation.
- Object properties: Thresholded attention maps directly represent object shape, allowing finer control than position and size alone.A specified binary mask can serve as the target, including a mask extracted from another image’s attention.
- Composed manipulation: Arbitrary scale, rotation, or translation of a shape target enables object manipulation while maintaining its silhouette.Figure 3 applies isolated property changes to move or resize individual objects without modifying the rest of the image.
- Object properties: Object-level appearance combines thresholded attention for object extent with spatial activation maps for local appearance.The resulting representation supports appearance guidance alongside spatial control.
4 Composing self-guidance properties
Self-guidance properties can be composed to manipulate individual objects, transfer layouts and appearances across images, combine objects from multiple sources, and edit real images. These controls extend from spatial properties such as shape, position, and size to appearance and broader captioned concepts.
- Adjusting specific properties: Self-guidance can move or resize one object while preserving other objects’ shapes and appearances.The method translates attention masks to move objects and changes the transformation to up- or down-sample shape matrices for resizing.
- Sampling new appearances: Constraining object layout while leaving appearance unconstrained samples new styles for the same scene.Figure 4 guides object shapes toward reconstruction of a given image’s layout, producing new appearances.
- Composition between images: Properties from different images can be combined to transfer one image’s layout and another’s object appearance, or to sample new arrangements for the same objects.Dropping the shape-guidance term preserves appearances while allowing new layouts.
- Compositional generation: Objects from multiple images can be collaged in place, or their appearances can be combined with a layout supplied by another image when source layouts are incompatible.The in-place operation preserves each object’s shape, size, position, and appearance; an alternative uses only appearances and a new target layout.
- Editing with real images: Self-guidance transfers appearances from one real image into new contexts and enables spatial manipulation of real images.Real-image editing extracts intermediates from noised versions and controls appearance, position, or size without fine-tuning.
- Attributes and interactions: The method also manipulates captioned verbs and adjectives and can help enforce attribute binding during generation.Its scope extends beyond objects to any concept appearing in the caption.
5 Discussion
Self-guidance uses internal attention maps and activations to satisfy properties during diffusion sampling, enabling composed image manipulations. The discussion identifies entanglement, information leakage, and prompt-dependent control as important limitations.
- Self-guidance constrains intermediate activations and attention maps to steer diffusion sampling using properties derived from the denoising model itself.
- The method can manipulate non-object words, including adjectives and verbs, and additional guidance can correct improper attribute binding.
- High appearance guidance can leak object position, while strongly guiding all token shapes can leak appearance information.
- The proposed properties have limitations, including leakage between appearance and position and entanglement among objects in attention space.
- Cross-attention restricts control to objects described in the text prompt and hinders fully disentangled control when interacting objects have correlated attention maps.
- Appearance features often encode spatial layout, while shapes of multiple tokens can sometimes reveal appearance through hidden interaction patterns between attention channels.
Broader impact
Self-guidance enables precise image manipulation, but its use with real-image appearance or layout transfers may create harmful content risks.
- Self-guidance can enable harmful image manipulations by transferring appearance or layout from real images into arbitrary generated content.
- The authors mitigated this risk experimentally by refraining from generating images containing humans.
A. Implementation details
The implementation applies self-guidance during selected portions of diffusion sampling and extracts different control signals from attention maps and decoder activations. Figures demonstrate object movement and resizing with fixed guidance settings.
- Self-guidance is applied during the first 3N⁄16 steps, while later steps alternate guided and unguided updates before the final N⁄32 unguided steps.The implementation uses N = 1024 steps.
- The overall gradient scale is v = 7500 because per-pixel gradients are often only 10^-7 to 10^-6.
- Centroid, size, and shape terms are applied across 36 cross-attention interactions spanning encoder, bottleneck, and decoder resolutions.
- Attention maps are differentiably binarized before size and shape computation to reduce background-noise effects, unlike centroid guidance.
- Figure 11 moves onion, pineapple, owl, and pig objects using Eqn. 9 with weights w0 = 1.5, w1 = 0.25, and w2 = 2.
- Figure 12 resizes objects using up- or down-sampled attention maps, reducing a punching bag’s height 0.5× and enlarging chicken 2.5× and boombox 2×.
B. Using self-guidance
Self-guidance supports consistency-preserving edits, appearance and layout resampling, cross-image composition, and transfer from real images. The approach relies on tunable guidance weights and can trade appearance diversity or reliability for stronger control.
- Using self-guidance: Applying a transformation directly to an original shape preserves object silhouettes more effectively during resizing and repositioning than equivalent centroid and size changes.
- Using self-guidance: Abstract prompt words often carry layout-correlated message-passing patterns, so avoiding their layout guidance can improve compositionality.
- Using self-guidance: Corrupting target shapes with Gaussian noise can mitigate appearance-layout entanglement, whereas mean-pooling attention across multi-token concepts does not improve results.
- Using self-guidance: Object movement can fix all other object shapes and appearances while changing one object’s position and preserving its size through separate guidance terms.
- Using self-guidance: Higher weights for sampling new appearances preserve layout more precisely but reduce diversity in appearance.
- Using self-guidance: Self-guidance can sample new layouts for fixed appearances and compose object properties across multiple source images or into a target layout.
- Using self-guidance: Object appearance can be transferred from a source image to a new prompt when a mapping relates source and generated object indices.
- Using self-guidance: The method can guide generation from attention and activations extracted through forward-process denoising of real images, enabling appearance transfer and edits such as moving or resizing objects.
C. Additional results
Additional experiments report non-cherry-picked results across the paper’s demonstrated image-editing tasks, using varied prompts and seeds.
- C. Additional results: The protocol evaluates the first eight generated images without further filtering after selecting prompts and testing several guidance-weight configurations.Typically, the authors start with their usual edit weights and try around 3–5 alternatives.
- C. Additional results: Results cover moving and resizing objects, generating new appearances for fixed layouts, and generating new layouts for given objects.
- C. Additional results: The additional experiments also transfer real-object appearances into new contexts.
- C. Additional results: Different random seeds are used to demonstrate guidance without shared DDPM noise.