Source-linked AI summary
Universal Guidance for Diffusion Models
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, Tom Goldstein
TL;DR
Typical diffusion models are tied to a particular conditioning modality, so supporting another modality can require costly retraining. This paper proposes universal guidance that uses off-the-shelf differentiable guidance functions with a fixed diffusion model, achieving promising results across segmentation, face recognition, object detection, and combined guidance tasks.
Problem
Typical diffusion models are trained for particular conditioning modalities, and adapting them to other modalities can require costly retraining.
Method
The method samples with a fixed foundation diffusion model while evaluating off-the-shelf guidance and loss functions on denoised images, without retraining.
Results
The algorithm generates promising guided images for segmentation, face recognition, object detection, classifier labels, and combinations of multiple guidance functions.
Takeaways & Limitations
Universal guidance lets one fixed diffusion model support diverse and simultaneous image-generation constraints using off-the-shelf guidance functions.
Takeaways & Limitations
Generation is typically slower because recurrence steps and, in some scenarios, inner-loop multistep gradient descent are required.
Abstract
from arXiv · showhide
Typical diffusion models are trained to accept a particular form of conditioning, most commonly text, and cannot be conditioned on other modalities without retraining. In this work, we propose a universal guidance algorithm that enables diffusion models to be controlled by arbitrary guidance modalities without the need to retrain any use-specific components. We show that our algorithm successfully generates quality images with guidance functions including segmentation, face recognition, object detection, and classifier signals. Code is available at https://github.com/arpitbansal297/Universal-Guided-Diffusion.
1. Introduction
Diffusion conditioning usually locks a model to one input modality, whereas universal guidance adapts a fixed diffusion model using off-the-shelf guidance functions without retraining. The proposed sampler evaluates guidance on denoised images, supporting diverse and simultaneous constraints.
- Motivation: Standard conditioning requires training a diffusion model for a particular modality, making new modalities expensive because retraining may be needed from scratch.The paper identifies text, segmentation maps, and class labels as examples of conditioning modalities.
- Guidance: Guidance pairs a generic diffusion generator with a function that measures whether a user-specified criterion has been met.Sampling iterates are nudged along the guidance-function gradient toward satisfying the criterion.
- Contribution: Universal guidance uses any off-the-shelf model or loss function without retraining or modifying the guidance function or diffusion model.The fixed diffusion network can therefore support many commonplace and bespoke use cases.
- Contribution: The proposed sampler evaluates guidance models on denoised images rather than noisy latent states, closing the domain gap between sampling inputs and guidance-model training data.This design keeps the underlying diffusion model fixed and requires no fine-tuning.
- Results: The approach is demonstrated with classifier labels, human identities, segmentation maps, object-detector annotations, and inverse linear-problem constraints.These examples cover multiple guidance modalities and constraint types.
2. Background
Diffusion models generate images by reversing a noise-adding process, while controlled generation seeks in-distribution samples matching differentiable prompts or constraints. Prior conditional methods retrain models for each prompt type; this work instead studies universal guidance with off-the-shelf functions.
- 2.1. Diffusion Models: Diffusion models combine a forward process that adds Gaussian noise with a reverse process that progressively denoises toward a clean data point.The model is trained around a learned denoising network.
- 2.1. Diffusion Models: DDIM-style sampling predicts a clean data point, substitutes it for the unknown original, and samples the preceding noisy state.Sampling methods can be abstracted as z_t−1 = S(z_t, ˆϵ, t).
- 2.2. Controlled Image Generation: Controlled generation applies a differentiable guidance function to an image and seeks an in-distribution sample whose output matches a specified prompt under a closeness loss.Examples include CLIP feature extractors and segmentation networks.
- 2.2. Controlled Image Generation: Prior controlled diffusion methods divide into conditional and guided image generation.Conditional methods train new diffusion models to accept prompts, whereas guided methods modify sampling around a frozen foundation model.
- 2.2. Controlled Image Generation: Conditional methods succeed across constraint types but require computationally intensive diffusion-model retraining, while guided methods use feedback from external guidance functions.The paper positions its method within guided image generation using off-the-shelf functions such as object detection and segmentation networks.
3. Universal Guidance
Universal guidance augments a frozen diffusion sampler with forward and backward guidance computed from predicted clean images, while self-recurrence helps balance prompt satisfaction with image realness. The method supports generic and multiple guidance functions without retraining, but can be slower and may produce artifacts when guidance conflicts with natural-image sampling.
- Universal Guidance: Universal guidance extends diffusion sampling with off-the-shelf guidance functions while keeping the underlying diffusion model fixed and requiring no fine-tuning.The algorithm can also be adapted to multiple guidance-function and loss pairs.
- 3.1. Forward Universal Guidance: Directly applying standard classifier-style guidance fails because auxiliary guidance models trained on clean images provide weak feedback on noisy diffusion states.The method therefore computes guidance using predicted clean images rather than noisy latent states.
- 3.1. Forward Universal Guidance: Forward guidance nudges the denoising prediction using the gradient of the guidance loss evaluated on the predicted clean image, with s(t) controlling per-step strength.This keeps the generation trajectory in the data manifold while bringing images closer to the prompt.
- 3.2. Backward Universal Guidance: Forward guidance can over-prioritize image realness, while increasing its strength may cause instability and still fail to satisfy the prompt.Backward guidance supplements it by prioritizing constraint enforcement.
- 3.2. Backward Universal Guidance: Backward guidance optimizes a clean-space change that directly reduces the guidance loss, then translates that change back into the noisy data space.Multiple gradient steps are affordable because each backward-guidance step is computationally cheaper than forward guidance.
4. Experiments
Experiments evaluate universal guidance across Stable Diffusion and unconditional ImageNet diffusion using text, segmentation, face, object-location, style, and CLIP guidance. The method produces high-quality generations that satisfy these constraints, including combinations of guidance signals and out-of-distribution prompts.
- Experimental setup: Experiments span Stable Diffusion and an unconditional ImageNet diffusion model across diverse guidance functions.The tested functions include CLIP, segmentation, face recognition, object detection, classifier labels, and combined inpainting guidance.
- Text guidance: The universal algorithm generates high-quality images matching text descriptions comparably to specialized text-conditioning models.The comparison uses identical prompts for universal guidance and classical text-conditional Stable Diffusion.
- Segmentation guidance: Segmentation guidance yields realistic images whose object-background separation nearly perfectly matches the supplied maps while preserving descriptive text.The text describes combinations of segmentation maps with new prompts and text-conditional sampling.
- Additional guidance modalities: Face, object-location, and style guidance preserve prompt content while matching identities, bounding-box placement and size, or reference-image style.Face characteristics are reproduced almost perfectly, objects appear in designated locations with appropriate sizes, and generated styles match the provided style images.
- ImageNet guidance: CLIP guidance on the unconditional ImageNet model produces quality images matching text prompts even for prompts expected to be out of distribution.The experiments include unrealistic art styles and objects outside ImageNet class labels.
- ImageNet guidance: Using both forward and backward guidance produces realistic images with correctly matched object categories and locations, unlike forward guidance alone.Forward-only generations remain realistic but can contain mismatching categories and locations; the results validate the necessity of backward guidance.
5. Limitations
Universal guidance is typically slower than standard conditional generation, especially when complex guidance requires repeated denoising, backward optimization, or large guidance networks.
- Generation is typically slower than standard conditional generation because complex guidance requires multiple denoising iterations at each noise level.
- Runtime scales linearly with the number of recurrence steps k, so larger k slows image generation.
- Backward guidance can require a multistep gradient-descent minimization loop, increasing computation time.
- Backward guidance takes longer when the guidance function is a very-large neural network.
- Optimal results require sampling hyper-parameters to be chosen individually for each guidance network.
6. Conclusion
The paper proposes universal guidance for guided image generation with differentiable, off-the-shelf guidance functions and a fixed foundation diffusion model. It reports promising results across complex guidance tasks and supports combining multiple guidance functions.
- The algorithm uses any off-the-shelf guidance function with a fixed foundation diffusion model, without retraining either component for a specific prompt type.
- Guidance and loss functions must be differentiable for the algorithm to operate.
- The method produces promising results for segmentation, face recognition, and object detection guidance.
- Multiple guidance functions can be combined and used together.
A. More results
Additional examples illustrate image generation guided by segmentation maps, human identity, object locations, and styling images, each paired with a caption as the text prompt.
- Segmentation maps guide image generation, with each map paired with its caption as the text prompt.
- Human identity images guide face-related image generation, with each image paired with its caption as the text prompt.
- Object locations guide image generation, with each location image paired with its caption as the text prompt.
- Styling images guide style transfer, with each image paired with its caption as the text prompt.