Source-linked AI summary
Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models
Tuomas Kynkäänniemi, Miika Aittala, Tero Karras, Samuli Laine, Timo Aila, Jaakko Lehtinen
TL;DR
Constant classifier-free guidance is widely used, but its effects vary across noise levels and can reduce variation early while adding little benefit late. The paper restricts guidance to a middle noise interval, improving quality and sampling cost, including a record ImageNet-512 FID result from 1.81 to 1.40.
Problem
Constant guidance throughout sampling is sub-optimal because guidance is harmful at high noise levels, beneficial in the middle, and largely unnecessary at low noise levels.
Method
The paper limits guidance to an interval of middle sampling steps while leaving the guidance weight otherwise unchanged.
Results
1.81 to 1.40: the optimal guidance interval improves ImageNet-512 FID, with benefits also reported across samplers, architectures, datasets, and Stable Diffusion XL.
Takeaways & Limitations
The authors recommend exposing the guidance interval as an additional sampler parameter because it improves numerical and visual results while reducing sampling cost.
Takeaways & Limitations
Future work is needed to determine whether optimal guidance intervals can be derived automatically from the ODE and how trained-denoiser non-idealities affect them.
Abstract
from arXiv · showhide
Guidance is a crucial technique for extracting the best performance out of image-generating diffusion models. Traditionally, a constant guidance weight has been applied throughout the sampling chain of an image. We show that guidance is clearly harmful toward the beginning of the chain (high noise levels), largely unnecessary toward the end (low noise levels), and only beneficial in the middle. We thus restrict it to a specific range of noise levels, improving both the inference speed and result quality. This limited guidance interval improves the record FID in ImageNet-512 significantly, from 1.81 to 1.40. We show that it is quantitatively and qualitatively beneficial across different sampler parameters, network architectures, and datasets, including the large-scale setting of Stable Diffusion XL. We thus suggest exposing the guidance interval as a hyperparameter in all diffusion models that use guidance.
1 Introduction
Diffusion models generate high-quality images across modalities, with classifier-free guidance commonly used to trade variation for perceptually higher quality. The paper argues that constant guidance is sub-optimal and proposes limiting it to middle noise levels, improving both quality and cost.
- 1 Introduction: Diffusion models repeatedly denoise noise into images and have advanced high-quality synthesis across text, video, 3D shapes, and audio.They also scale to large datasets.
- 1 Introduction: Classifier-free guidance uses unconditional and conditional denoising results to steer samples toward the condition, trading variation for perceptually higher quality.The guidance weight controls the strength of this steering.
- 1 Introduction: Constant guidance is sub-optimal because high noise levels reduce variation toward template images, middle levels sharpen feature choices, and low levels add little benefit.These effects differ substantially across the sampling chain.
- 1 Introduction: The paper limits guidance to a middle interval where its net effect is positive, avoiding detrimental effects while reducing computational cost.The guidance weight itself is otherwise unchanged.
- 1 Introduction: 1.81 to 1.40: an optimal guidance interval improves ImageNet-512 state-of-the-art FID and visual quality.Benefits are reported across sampler parameters, network architectures, datasets, and Stable Diffusion XL.
2 Background
The paper formulates diffusion sampling as an ODE and views classifier-free guidance as modifying its trajectory through conditional and unconditional denoising. A synthetic example shows that guidance is harmful at high noise levels, while disabling it there and at low noise can restore modes and reduce cost.
- 2 Background: A synthetic 1D example uses ideal analytic denoisers to visualize guidance harms without approximation errors from learned denoisers.In this scenario, classifier-free guidance is strictly harmful.
- 2 Background: Diffusion sampling evolves an initial high-noise sample toward the data distribution by following an ODE as σ decreases to zero.The initial distribution at σmax is chosen to approximate pure Gaussian noise.
- 2 Background: Classifier-free guidance constructs a modified ODE by linearly combining conditional and unconditional ODEs, with guidance weight w and condition information c.For w > 1, the condition is extrapolated relative to the unconditional result.
- 2 Background: Guidance extrapolation can push trajectories away from the data distribution and cause mode drops by oversteering toward the condition.The modified trajectory is interpreted as concentrating probability mass in regions agreeing with the condition.
- 2 Background: Sampling discretizes the ODE into N steps, and computational cost is directly proportional to N.Different solvers and discretization schemes can be used.
3 Our method
The method limits classifier-free guidance to a middle interval of noise levels, avoiding distribution drift at high noise and reducing evaluations at low noise. The interval improves sample quality and applies across model variants and samplers without changing model complexity.
- Applying guidance at every noise level can skew intermediate distributions and almost entirely drop one mode.The toy example attributes this drift to unconditional trajectories repelling guided trajectories.
- Disabling guidance at high noise recovers both modes, while disabling it at low noise has little effect on the resulting distribution.This also reduces sampling cost with minimal effect on outputs.
- Limiting CFG improves FID and FDDINOv2 across EDM2 variants and diffusion transformers while slightly reducing sampling cost without altering model complexity.The table reports results on ImageNet-512 across small and XXL EDM2 models and diffusion transformers.
- The proposed method applies guidance only within a continuous middle interval of noise levels and disables it elsewhere.The ODE is redefined by replacing the constant guidance weight with a piecewise constant function.
- σhi enables guidance and σlo turns it off; traditional CFG is recovered when σlo = 0 and σhi = ∞.
- The interval boundaries must align with sampler step boundaries so the guidance weight remains constant within each Runge–Kutta step.The method rounds σlo and σhi appropriately to satisfy the smoothness requirement.
4 Results
Across ImageNet, diffusion transformers, and Stable Diffusion XL, limiting guidance to an intermediate noise interval improves quality while reducing guidance-related artifacts and computation. The method remains effective across sampling settings and guidance weights, with qualitative gains in detail, diversity, composition, and color realism.
- Main results: FID improves from 2.23 to 1.68 with EDM2-S and reaches 1.40 with EDM2-XXL; FDDINOv2 improves from 33.09 to 29.16 with EDM2-XXL.These ImageNet-512 results use the models with default 32-step deterministic Heun sampling.
- Main results: DiT-XL/2 also benefits, with best FID using guidance weight w = 2.5 for 75 of 250 sampling steps.The corresponding FID interval is σ ∈(0.34, 1.02], while the best FDDINOv2 uses σ ∈(0.45, 1.23] and w = 4.0.
- Main results: Limiting guidance makes FID and FDDINOv2 less sensitive to guidance weight, allowing substantially higher weights than standard classifier-free guidance.Standard guidance becomes harmful at excessive weights because the output distribution is overly truncated.
- Ablations: The method improves FDDINOv2 mainly through higher recall without significantly changing precision, consistent with more varied generated results.Precision-recall curves are evaluated in DINOv2 feature space over guidance weights from 1.0 to 4.0.
- Ablations: Including excessive high-noise levels or choosing too narrow an interval worsens FID, while the upper and lower limits can be optimized separately.The proposed search procedure first determines the upper limit with the lower limit at zero, then determines the lower limit.
5 Conclusions
The paper concludes that limiting classifier-free guidance to an interval improves diffusion-model results while reducing sampling cost. It identifies automatic interval derivation and the role of trained-denoiser non-idealities as future research directions.
- Limiting classifier-free guidance improves image results numerically and visually while reducing sampling cost.
- Future work could investigate deriving the optimal guidance interval automatically from the ODE.
- Future work could examine how non-idealities in trained denoisers affect the optimal guidance interval.
A Characterizing sampling steps in noise levels
The appendix characterizes how sampling-step indices correspond to noise levels under the discretizations used for different models. It specifies parameter settings for EDM2, Stable Diffusion XL, and DiT.
- For EDM2 models and Stable Diffusion XL, the ith sampling step is mapped to a noise level using the discretization from [22].
- Stable Diffusion XL uses ρ = 3, while EDM2 models use ρ = 7.
- The discretization uses σmin = 0.002 and σmax = 80.
- For DiT, the ith sampling step is mapped to its corresponding noise level using the iDDPM discretization from [22].
- The default parameters are C1 = 0.001, C2 = 0.008, M = 1000, and j0 = 0.
B Additional qualitative results
Additional figures compare classifier-free guidance with the proposed method and show examples using increased guidance weights. The figures also compare both methods at higher guidance weights.
- Figures 10 and 11 provide further comparisons between classifier-free guidance and the proposed method.
- Figures 12 and 13 show additional examples from the proposed method with increased guidance weight.
- Figures 14, 15, and 16 compare classifier-free guidance with the proposed method when the guidance weight is increased.
C Broader impacts
The method improves result quality and decreases sampling cost for large-scale diffusion models, but these models may amplify societal harms and continue to require substantial computing power.
- The method improves the result quality of large-scale diffusion models, potentially magnifying disinformation, harmful biases, and stereotypes.
- In the large-scale setting, the method decreases sampling cost, but diffusion models still require substantial computing power.
- The continuing computational demands of diffusion models may contribute to wider issues such as climate change.
D Licenses
The paper reports licensing information for the pre-trained models and ImageNet dataset, alongside qualitative comparisons of standard CFG and the proposed limited-guidance method.
- The pre-trained EDM2 models are licensed under CC BY-NC-SA 4.0 by NVIDIA.
- The pre-trained SD-XL model is available under Stability AI’s CreativeML Open RAIL++-M License.
- ImageNet uses a custom non-commercial license.
- Standard CFG with high guidance can produce crisp details but low diversity, whereas limited guidance produces crisp details with high diversity.