PaperlayerView exact revision on arXiv

Source-linked AI summary

FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model

Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, Jian Zhang

arXiv:2303.09833v1cs.CV

TL;DR

Conditional diffusion models often require costly retraining for new conditions, while existing training-free methods have limited generality. FreeDoM uses off-the-shelf time-independent networks to build energy functions that guide diffusion without training. Experiments show effectiveness across conditions and image and latent-code domains, with faster generation than UGD in the reported comparison.

  • Problem

    Training-required conditional diffusion models are costly and inconvenient to transfer across conditions, while many training-free methods are difficult to generalize beyond specific applications.

  • Method

    FreeDoM uses off-the-shelf pre-trained time-independent networks to construct flexible energy functions whose gradients guide unconditional or latent diffusion sampling.

  • Results

    FreeDoM is effective across varied conditions and diffusion domains, and synthesizes one 512×512 image in about 84 seconds, nearly 30× faster than UGD's 40 minutes.

  • Takeaways & Limitations

    FreeDoM supports a broad range of conditions and can combine training-free energy interfaces with existing training-required conditional interfaces.

  • Takeaways & Limitations

    Sampling remains slower than training-required methods, fine-grained control in large data domains can be difficult, and multi-condition control assumes independent conditions.

Abstract

from arXiv · show

Recently, conditional diffusion models have gained popularity in numerous applications due to their exceptional generation ability. However, many existing methods are training-required. They need to train a time-dependent classifier or a condition-dependent score estimator, which increases the cost of constructing conditional diffusion models and is inconvenient to transfer across different conditions. Some current works aim to overcome this limitation by proposing training-free solutions, but most can only be applied to a specific category of tasks and not to more general conditions. In this work, we propose a training-Free conditional Diffusion Model (FreeDoM) used for various conditions. Specifically, we leverage off-the-shelf pre-trained networks, such as a face detection model, to construct time-independent energy functions, which guide the generation process without requiring training. Furthermore, because the construction of the energy function is very flexible and adaptable to various conditions, our proposed FreeDoM has a broader range of applications than existing training-free methods. FreeDoM is advantageous in its simplicity, effectiveness, and low cost. Experiments demonstrate that FreeDoM is effective for various conditions and suitable for diffusion models of diverse data domains, including image and latent code domains.

1 Peking University Shenzhen Graduate School 2 KAUST https://github.com/vvictoryuki/FreeDoM

Figure 1 presents FreeDoM as a training-free controller for diffusion generation across face editing, text guidance, combined conditions, and latent diffusion interfaces.

  • FreeDoM guides an unconditional diffusion model using segmentation maps, sketches, landmarks, and face IDs for face editing.
  • CLIP-based text guidance controls image synthesis and editing, with segmentation masks limiting the edited areas.
  • Combined conditions can jointly control the generation process.
  • Training-free guidance also augments Stable Diffusion and ControlNet with face-ID and style-image energy controls.

1. Introduction

The introduction identifies a flexibility and cost gap in conditional diffusion models and proposes FreeDoM, which uses adaptable energy guidance with off-the-shelf networks across conditions and diffusion domains.

  • Training-required conditional diffusion models need retraining or finetuning when new target conditions are introduced, making them inconvenient and expensive to adapt.
  • Existing training-free methods are effective for individual applications but difficult to generalize to style, face ID, and segmentation-mask conditions.
  • FreeDoM constructs flexible energy functions from off-the-shelf pre-trained time-independent models to support varied conditions without training.
  • FreeDoM inserts an energy-gradient derivative step into unconditional sampling, while its energy functions and base diffusion models need no retraining on desired conditions.
  • The method supports text, segmentation maps, sketches, landmarks, face IDs, style images, multiple conditions, and image or latent diffusion domains.

2. Related Work

Related work contrasts strong but training-expensive conditional interfaces with training-free methods that remain narrower in task and condition coverage, motivating broader training-free CDMs.

  • Training-required methods use supervised data pairs to provide strong control, including text-to-image, edge-map, segmentation-map, and depth-map interfaces.
  • Their training cost is especially relevant for complex multi-condition control, where they are not the cheapest or most convenient practical solution.
  • Training-free approaches exploit diffusion denoising through cross-attention control, intermediate-result modification, or initial-noise adjustment.
  • The paper positions FreeDoM as faster and applicable to a broader range of applications than recent efforts.

3. Preliminaries

The preliminaries formulate conditional diffusion through score correction and energy guidance: an unconditional score is combined with a condition-dependent correction derived from an energy function.

  • Score-based diffusion models estimate ∇x_t log p(x_t) and use the estimated score to predict x_t−1 from noisy x_t step by step.
  • The DDPM sampling formula combines the current noisy state, its score, predefined β_t, and Gaussian noise to produce the previous timestep.
  • Conditional Score Function: Conditional diffusion replaces the unconditional score with ∇x_t log p(x_t|c), decomposing it into an unconditional score and a correction gradient.
  • Conditional Score Function: The correction gradient points x_t toward data compatible with condition c, and classifier-based methods compute it using a time-dependent classifier.
  • Energy Diffusion Guidance: Energy guidance models condition compatibility with E(c, x_t), whose negative gradient supplies the correction used in conditional sampling.
  • Energy Diffusion Guidance: The resulting sampling update subtracts a scaled energy gradient, with ρ_t acting as the correction-term learning rate.

4. The Proposed FreeDoM Method

FreeDoM approximates time-dependent energy guidance with time-independent distance functions applied to estimated clean images, then improves guidance in large data domains using selective time travel. Its energy functions are constructed from off-the-shelf networks and can support varied conditions.

  • 4.1. Approximate Time-Dependent Energy: FreeDoM uses off-the-shelf time-independent distance networks for clean data to approximate time-dependent energy functions without training condition-specific noisy-image networks.Available classification, segmentation, and face-ID networks can measure distances on clean images.
  • 4.1. Approximate Time-Dependent Energy: The method estimates a clean image x0|t from each noisy state xt using the pre-trained unconditional score estimator before evaluating the energy function.The estimate is x0|t = 1/sqrt(ᾱt)(xt + (1 − ᾱt)s(xt,t)).
  • 4.1. Approximate Time-Dependent Energy: FreeDoM guides each sampling update by subtracting a learning-rate-scaled gradient of the distance between the condition and estimated clean image.This inserts the energy-gradient correction into unconditional diffusion sampling.
  • 4.2. Efficient Time-Travel Strategy: On large domains such as ImageNet, ordinary energy guidance can fail because unconditional score directions have greater freedom, producing condition-mismatched or random results.The baseline process works effectively on smaller domains such as human faces but often fails on large domains.
  • 4.2. Efficient Time-Travel Strategy: Time travel resamples an intermediate state after moving it backward by j steps, improving guidance but increasing sampling cost.The experiments use j = 1 and repeat resampling rt times at timestep t.
  • 4.2. Efficient Time-Travel Strategy: FreeDoM applies time travel only during the semantic stage, where generated content changes substantially, avoiding inefficient use during chaotic and refinement stages.The semantic-stage range is selected experimentally for each diffusion model, and rt = 1 means no time travel at timestep t.

5. Experiments

Experiments evaluate FreeDoM across multiple pretrained diffusion models, domains, and conditional tasks. Results show effective condition control, diversity, style alignment, and compatibility with latent diffusion models, with learning-rate control over guidance intensity.

  • Implementation Details: FreeDoM is evaluated with unconditional face and ImageNet diffusion models, a classifier-based ImageNet model, Stable Diffusion, and ControlNet.Experiments cover image resolutions of 256 × 256 and latent diffusion outputs including Stable Diffusion at 512 × 512.
  • Qualitative Results: Single-condition face and ImageNet generations meet the given conditions while retaining good quality and diversity.The evaluated face conditions include text, parsing maps, sketches, landmarks, and face IDs; ImageNet experiments include text and style images.
  • Qualitative Results: Multiple-condition experiments demonstrate conditional synthesis with unconditional human-face and ImageNet diffusion models.The experiments use the efficient time-travel strategy to improve condition matching where needed.
  • Comparison with TediGAN: FreeDoM generates images matching conditions better than TediGAN, with comparable or better image quality across segmentation, sketch, and text tasks.The comparison uses condition distance and FID on 1000 generated results for each task.
  • Comparison with UGD: FreeDoM offers better style alignment than UGD and synthesizes one 512 × 512 image in about 84 seconds versus UGD’s 40 minutes.The reported inference time is nearly 30× faster than UGD on a GeForce RTX 3090 GPU.
  • Further Studies: Increasing the energy-function learning rate from 0 changes control intensity, allowing users to adjust guidance strength.The study examines learning rates across unconditional ImageNet and human-face diffusion models.

6. Conclusions & Limitations

FreeDoM provides training-free energy-guided conditional diffusion across image and latent domains using pretrained time-independent networks. Its limitations include higher sampling cost, weaker fine-grained control in large data domains, and an independence assumption for multiple conditions.

  • Conclusions: FreeDoM uses pretrained time-independent networks to approximate energy functions whose gradients guide diffusion sampling without training.The framework supports both image and latent diffusion models.
  • Conclusions: The method supports diverse conditions and diffusion models, while the presented applications represent only a subset of its stated scope.Future work proposes exploring additional energy functions for broader tasks.
  • Limitations: Sampling remains slower than training-required methods because energy derivatives and time-travel add computation and sampling steps.This is identified as the first limitation of FreeDoM.
  • Limitations: Fine-grained structural control can be difficult in large data domains, where Canny edge conditions may produce poor guidance.The paper states that training-required methods can be a better alternative in this case.
  • Limitations: Multi-condition control assumes independent conditions, so conflicting conditions may yield subpar generations.The independence assumption is not necessarily true in practice.

Appendix A. More Results

Additional results demonstrate FreeDoM across text, segmentation, sketch, landmark, face-ID, style, and latent diffusion applications. The generated outputs are reported as condition-consistent and diverse, including combinations with ControlNet models.

  • Human-Face Conditions: Text-guided human-face generations use short and long prompts, including characteristics described as seldom seen in the training set, while remaining diverse and condition-consistent.The results are shown in Figure 11.
  • Human-Face Conditions: Segmentation, sketch, and landmark guidance are assessed by comparing the input conditions with corresponding outputs extracted from generated faces.Figures 12–14 report consistency with the provided conditions and good diversity.
  • Human-Face Conditions: Face-ID guidance uses six celebrity identities as references, and the generated results are reported as consistent with the given IDs.These results are shown in Figure 15.
  • Latent Diffusion Applications: Style guidance is demonstrated with Stable Diffusion and Scribble ControlNet, whose generated images are reported to match the provided styles.These experiments are shown in Figures 16 and 17.
  • Latent Diffusion Applications: Face-ID guidance is combined with Human-pose ControlNet, and fixed random seeds expose the effect of adding ID guidance.Figure 19 visualizes decoding, face detection, landmark alignment, and ID-energy computation in this process.

Appendix B. Relationship between FreeDoM and Zero-Shot Image Restoration Methods

FreeDoM provides a framework for interpreting zero-shot image restoration as conditional generation guided by data-consistency energies. Existing restoration updates can therefore be viewed as distance-based or analytically derived gradient guidance within this framework.

  • Framework Relationship: Zero-shot image restoration methods are presented as special cases of FreeDoM because they update clean intermediate results to satisfy y = A(x0|t).Here, y is the degraded image and A is a linear or nonlinear degradation operator.
  • Examples: DPS uses the gradient of a measurement-space distance between the restored intermediate image and the degraded observation.Its update is interpreted as improving matching without learned parameters.
  • Examples: DDNM derives an update direction for linear noiseless tasks through Range-Null Space Decomposition, interpretable as an approximate distance-gradient solution.The direction is −A†(Ax0|t −y).
Loading 2303.09833v1…