PaperlayerView exact revision on arXiv

Source-linked AI summary

Guiding a Diffusion Model with a Bad Version of Itself

Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, Samuli Laine

arXiv:2406.02507v3cs.CVcs.AIcs.LGcs.NEstat.ML

TL;DR

Diffusion guidance improves quality but can entangle quality, prompt alignment, and variation, while CFG is limited to conditional generation and may distort sampling. The paper introduces autoguidance, which guides a model with a smaller or less-trained version of itself under unchanged conditioning. It improves ImageNet generation, reaching FIDs of 1.01 at 64×64 and 1.25 at 512×512, while also applying to unconditional synthesis.

  • Problem

    Classifier-free guidance improves quality and prompt alignment at the cost of variation, leaving these effects entangled and limiting guidance for unconditional generation.

  • Method

    Autoguidance uses an inferior, capacity- or training-limited version of the main model as the guiding model with unchanged conditioning.

  • Results

    Autoguidance considerably improves ImageNet generation, setting record FIDs of 1.01 for ImageNet-64 and 1.25 for ImageNet-512.

  • Takeaways & Limitations

    The method provides quality guidance without compromising variation and extends guidance to unconditional diffusion models.

  • Takeaways & Limitations

    CFG guidance can distort sampling trajectories, exaggerate truncation, and cause mode dropping because its implied density is not a valid heat diffusion.

Abstract

from arXiv · show

The primary axes of interest in image-generating diffusion models are image quality, the amount of variation in the results, and how well the results align with a given condition, e.g., a class label or a text prompt. The popular classifier-free guidance approach uses an unconditional model to guide a conditional model, leading to simultaneously better prompt alignment and higher-quality images at the cost of reduced variation. These effects seem inherently entangled, and thus hard to control. We make the surprising observation that it is possible to obtain disentangled control over image quality without compromising the amount of variation by guiding generation using a smaller, less-trained version of the model itself rather than an unconditional model. This leads to significant improvements in ImageNet generation, setting record FIDs of 1.01 for 64x64 and 1.25 for 512x512, using publicly available networks. Furthermore, the method is also applicable to unconditional diffusion models, drastically improving their quality.

1 Introduction

Diffusion models generate images by reversing noise through iterative denoising, typically conditioned on labels or prompts. Classifier-free guidance improves quality and alignment but entangles these benefits with reduced variation, motivating autoguidance.

  • Diffusion models: Diffusion models generate synthetic images by reversing a stochastic corruption process through successive denoising steps.The denoiser, equivalent to a score function, is central to generation and can be conditioned on labels or text prompts.
  • Motivation: Training aims to cover the entire conditional data distribution, including low-probability regions that limited models may represent poorly.These regions can produce low-quality samples despite being included by the training objective.
  • Classifier-free guidance: Classifier-free guidance focuses sampling on well-learned regions but is limited to conditional generation and can overshoot the desired conditional distribution.Its conditional and unconditional denoisers solve different tasks, contributing to skewed or simplified compositions.
  • Autoguidance: Autoguidance uses an inferior version of the main model with unchanged conditioning, separating quality improvement from prompt alignment.The guiding model can be produced by limiting capacity or training time and also supports unconditional synthesis.
  • Results: Autoguidance considerably improves FID and FDDINOv2, setting new ImageNet-512 and ImageNet-64 records.The implementation and pretrained models are publicly available.

2 Background

Diffusion samples are generated by following learned score fields from high noise to zero noise. Guidance extrapolates between two denoisers, and CFG uses conditional and unconditional models to increase class alignment, though its implied density can distort sampling trajectories.

  • Denoising diffusion: Diffusion models smooth the data distribution with Gaussian noise, sample approximately from the highest-noise density, then reverse the process with a probability-flow ODE.At zero noise, the intended output distribution is the original data distribution.
  • Score estimation: A denoiser trained on noisy data approximates the score as (Dθ(x; σ) − x)/σ2, up to finite-capacity and training-time errors.The network can therefore be interpreted as predicting either a denoised sample or a score vector.
  • Conditional generation: Conditional generation trains a denoiser that accepts a class label or text prompt and seeks samples from p(x|c; σ=0).The conditioning input controls the desired outcome during generation.
  • Guidance: General guidance extrapolates between denoisers as Dw(x; σ, c) = wD1(x; σ, c) + (1 − w)D0(x; σ, c).Setting w=0 or w=1 recovers D0 or D1, while w>1 over-emphasizes D1.
  • Classifier-free guidance: CFG uses an unconditional denoiser alongside a conditional denoiser, guiding samples toward stronger alignment with the specified class.Its extrapolated score adds a term involving the class likelihood gradient.
  • Limitations: CFG guidance does not define a valid heat diffusion of its zero-noise implied density, so solving the ODE can distort trajectories, exaggerate truncation, and drop modes.Despite these drawbacks, image quality often improves substantially.

3 Why does CFG improve image quality?

CFG improves image quality partly by exploiting a quality gap between conditional and unconditional denoisers, pulling samples inward toward better-learned regions. Autoguidance preserves diversity by using a degraded version of the same conditional model rather than relying on naive isotropic truncation.

  • Score matching leads to outliers: Small conditional diffusion models produce unlikely outliers because limited score networks and score matching encourage coverage of poorly learned distribution extremities.Sampling can also hand the network unlikely inputs that differ from those seen during training.
  • Autoguidance: Autoguidance concentrates samples in high-probability regions without reducing diversity by using a weaker version of the model itself as the guide.The toy example visualizes this behavior alongside conditional sampling, CFG, and naive truncation.
  • CFG eliminates outliers: CFG removes outliers and pulls samples toward the manifold core, but it also drops entire branches and cannot explain the concentration through class-likelihood boosting alone.This motivates examining differences between the conditional and unconditional models.
  • Quality difference: The unconditional denoiser faces the harder all-class task and therefore fits the data worse than the class-specific denoiser.The resulting density is more spread out, creating a quality difference that guides samples inward.
  • Density-ratio guidance: Guidance follows gradients of log[p1(x|c; σ)/p0(x|c; σ)], whose inward direction arises because the higher-quality density is more concentrated than the spread-out guiding density.The additional force increases the relative likelihood of samples under the higher-quality implied distribution.
  • Naive truncation: Naive truncation uniformly lengthens score vectors, concentrating samples isotropically while emptying outer branches and reducing variation, detail, and texture diversity.This differs from the adaptive, model-disagreement-based correction attributed to CFG.

4 Our method

Autoguidance improves diffusion sampling by guiding a high-quality model with an inferior version trained on the same task, conditioning, and data distribution. Its effectiveness depends on compatible degradations between the two models, while mismatched or synthetic degradations limit practical utility.

  • Core method: Autoguidance guides a high-quality model D1 with an inferior model D0 sharing its task, conditioning, and data distribution.D0 can be produced through reduced capacity or under-training.
  • Core method: In a 2D toy example, using a smaller D0 with fewer training iterations pulls samples toward the distribution without systematically dropping parts of it.
  • Mechanism and assumptions: Compatible degradations are expected to make autoguidance effective because weaker models broadly reproduce the stronger model’s systematic errors.The specific errors depend on architecture, dataset, and training details and cannot be identified a priori.
  • Synthetic degradation study: 2.55 FID is achieved with dropout autoguidance at w = 2.25, matching the base model’s FID.D1 and D0 use 5% and 10% dropout, with FIDs of 4.98 and 15.00, respectively.
  • Synthetic degradation study: 2.56 FID is achieved with input-noise autoguidance at w = 2.00, matching the base model’s ImageNet-512 conditional FID.The less-corrupted D1 has FID = 3.96 and the guiding D0 has FID = 9.73.
  • Synthetic degradation study: Mismatched degradations do not improve results; the best FID occurs at w = 1, disabling guidance and using D1 alone.
  • Limitations: Synthetic dropout and input-noise degradations are not proposed as practically useful guiding-model constructions for realistic diffusion models.They would not yield consistent truncation toward the data manifold.

5 Results

Across ImageNet resolutions and unconditional generation, autoguidance substantially improves diffusion-model quality using a smaller, less-trained guiding model. It also preserves broader image-composition variation than CFG in qualitative examples, while performance depends on guiding-model degradations and EMA settings.

  • ImageNet results: Autoguidance improves ImageNet generation across 512×512, 64×64, and unconditional settings.The evaluation uses EDM2 models on ImageNet-512 and ImageNet-64, including an unconditional setting.
  • ImageNet results: 1.34 FID replaces 2.56 for EDM2-S on ImageNet-512, while EDM2-XXL reaches a record 1.25.The EDM2-S result also beats the 1.68 FID reported for CFG + Guidance Interval.
  • ImageNet results: 1.01 FID and 31.85 FDDINOv2 set new ImageNet-64 records.The reported improvement is described as larger than on ImageNet-512.
  • Unconditional generation: 3.86 FID replaces 11.67 for unconditional EDM2-S generation with autoguidance.The corresponding FDDINOv2 improvement is also reported as significant.
  • Ablations: Reduced capacity and shorter guiding-model training are both beneficial, but reduced training contributes more to the improvement.For EDM2-S on ImageNet-512, isolating shorter training gives 1.51 FID, while isolating reduced capacity gives 2.13; both remain better than standard CFG.
  • Hyperparameter sensitivity: Guiding-model training at 1/16 of the main model is best, guidance weight is relatively insensitive, and EMA lengths remain important.An XS guiding model performs best for EDM2-S; further capacity reduction increases guidance-weight sensitivity, while post-hoc EMA makes parameter search feasible.
  • Qualitative results: CFG tends toward canonical class images, whereas autoguidance retains a wider gamut of compositions as guidance increases.The qualitative comparison includes Tree frog, Palace, Mushroom, and Castle examples from ImageNet-512.
  • Text-conditional results: DeepFloyd IF combines CFG and autoguidance through blending, with autoguidance using an XL model guided by an M model.Autoguidance is applied only to the base model; the two subsequent super-resolution stages continue using CFG.

6 Discussion and Future work

The paper argues that classifier-free guidance entangles quality, variation, and conditioning, while autoguidance opens a broader design space. It identifies formal analysis, guiding-model selection, and comparisons with other guidance strategies as future work.

  • Autoguidance separates effects that classifier-free guidance combines and removes its unnecessary dependence on conditioning.
  • Current large-scale generators may lack suitable early snapshots of smaller models, and staged training can change the data distribution.
  • Future work includes proving when autoguidance helps, developing guiding-model selection rules, and evaluating combinations and alternative metrics.
  • Noise-dependent CFG schedules suppress guidance at high noise levels, whereas autoguidance is not expected to reduce variation there because both models target the same distribution.

A Additional results

Additional experiments examine autoguidance across DeepFloyd IF results, guidance-weight sweeps, variation, and implied density evolution. They report lower sensitivity to guidance weight and greater variation than CFG.

  • Figure 6 adds DeepFloyd IF results comparing CFG, autoguidance, and blends across guidance weights w ∈{1, 2, 3, 4}.
  • At w = 4, autoguidance preserves much greater variation than CFG across Tree frog, Minibus, and Mushroom classes, despite visible saturation and artifacts.
  • FID and FDDINOv2 sweeps show that autoguidance is less sensitive to the exact guidance weight than CFG.
  • Figure 9 visualizes how implied learned densities evolve during sampling under CFG and autoguidance.

B Implementation details

The experiments build on publicly available EDM2 code and require only a small sampling-loop modification to use a differently conditioned guiding network. Reproduction uses existing scripts and specified model and hyperparameter configurations.

  • The experiments use the publicly available EDM2 codebase with NVIDIA A100 GPUs, Python 3.11.7, PyTorch 2.2.0, CUDA 11.8, and CuDNN 8.9.7.
  • Reproduction required changing one sampling-loop line to pass the class label to the guiding network alongside the main network.
  • Algorithm 1 demonstrates reproducing one Table 1 result, while other results use the indicated models and hyperparameters.

B.1 Hyperparameter search

Autoguidance parameters were optimized separately for FID and FDDINOv2 using a pruned local grid search. The search covered model capacity, training time, guidance weight, and EMA length, but required substantial computation.

  • Autoguidance parameters were grid-searched separately for FID and FDDINOv2 across five parameters.
  • The search varied model capacity, training time, guidance weight from 1.00 to 3.50, and EMA lengths from 0.010 to 0.250.
  • A local-neighborhood strategy progressively refined parameters around the best result and re-evaluated nearby choices twice to account for random noise.
  • Approximately 30,000 metric evaluations consumed an estimated 60–150 MWh overall, with each evaluation generating 50,000 images.

B.2 DeepFloyd IF experiments

The experiments combine classifier-free guidance with autoguidance by using a main model, an unconditional model, and a reduced-capacity conditional model. The method interpolates between the two guidance mechanisms while applying autoguidance only to the base model in the DeepFloyd IF cascade.

  • Three models combine CFG and autoguidance: the main model Dm, unconditional model Du, and reduced-capacity conditional model Dc.The guided denoising formulation extends to multiple guiding models.
  • wu and wc are the guidance weights for CFG and autoguidance, respectively.
  • α interpolates between CFG and autoguidance through wu := (1−α)(w−1)+1 and wc := α(w−1)+1.Here, w is the desired total guidance amount and α ∈[0, 1].
  • DeepFloyd IF applies the method only to its base model, while both super-resolution stages use CFG with default weights 4 and 9.The cascade uses 100, 75, and 50 sampling steps for the base and subsequent stages, respectively.

C Details of the 2D toy example

The toy example uses analytically tractable tree-like Gaussian mixtures and scalar-output denoisers to study the method. Models are trained by exact score matching and evaluated with a deliberately high-step EDM sampler, while the accompanying procedures reproduce model and FID experiments.

  • Dataset: Gaussian-mixture construction enables analytical ground-truth scores and probability densities without additional assumptions.The class-conditional distribution is represented as a mixture whose components have weights, means, and 2×2 covariance matrices.
  • Models: The denoiser models output scalar logarithms of unnormalized probability densities, from which scores and denoised samples are obtained by automatic differentiation.Alternative score-vector and denoised-sample formulations produced qualitatively similar behavior.
  • Training: The models are trained with exact score matching using the analytically available ground-truth score function.Denoising score matching produced no noticeable differences in model behavior or training dynamics.
  • Sampling: The EDM sampler uses N = 32 Heun steps, NFE = 63, σmin = 0.002, σmax = 5, and ρ = 7 for the toy experiments.The chosen N and σmax exceed what the dataset requires to reduce potential discretization effects.
Loading 2406.02507v3…