PaperlayerView exact revision on arXiv

Source-linked AI summary

CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling

Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, Romann M. Weber

arXiv:2310.17347v4cs.CV

TL;DR

Conditional diffusion models can lose output diversity under high guidance or limited training data, despite strong quality and coverage. CADS anneals Gaussian noise on the conditioning signal during inference without retraining, and improves diversity across tasks while achieving state-of-the-art ImageNet FIDs of 1.70 and 2.31 at 256×256 and 512×512.

  • Problem

    Conditional diffusion models can produce low-diversity outputs under high classifier-free guidance or when trained on small datasets, while lower guidance harms image quality and larger datasets may be infeasible or insufficient.

    Source: §1 · p. 2, §3 · p. 4

  • Method

    CADS adds scheduled, monotonically decreasing Gaussian noise to the conditioning vector during inference and can be applied without retraining across diffusion samplers.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    CADS improves diversity and quality across conditional tasks and achieves state-of-the-art ImageNet FIDs of 1.70 at 256×256 and 2.31 at 512×512.

    Source: §4.1 · p. 7

  • Takeaways & Limitations

    CADS preserves the quality benefits of high guidance while improving diversity through inference-time conditioning control.

    Source: §1 · p. 2, §5 · p. 9

    Larger datasets only partially mitigate limited diversity, and CADS’s benefits are less pronounced for Stable Diffusion because it already generates diverse samples.

    Source: §3 · p. 4, §4.1 · p. 7

Abstract

from arXiv · show

While conditional diffusion models are known to have good coverage of the data distribution, they still face limitations in output diversity, particularly when sampled with a high classifier-free guidance scale for optimal image quality or when trained on small datasets. We attribute this problem to the role of the conditioning signal in inference and offer an improved sampling strategy for diffusion models that can increase generation diversity, especially at high guidance scales, with minimal loss of sample quality. Our sampling strategy anneals the conditioning signal by adding scheduled, monotonically decreasing Gaussian noise to the conditioning vector during inference to balance diversity and condition alignment. Our Condition-Annealed Diffusion Sampler (CADS) can be used with any pretrained model and sampling algorithm, and we show that it boosts the diversity of diffusion models in various conditional generation tasks. Further, using an existing pretrained diffusion model, CADS achieves a new state-of-the-art FID of 1.70 and 2.31 for class-conditional ImageNet generation at 256$\times$256 and 512$\times$512 respectively.

1 INTRODUCTION

Conditional diffusion models offer strong quality and coverage but can produce insufficiently diverse outputs under high guidance or limited training data. CADS addresses this by annealing noisy conditioning during inference, improving diversity while preserving condition alignment and sample quality.

1 INTRODUCTION

  • High classifier-free guidance can reduce diversity, while smaller training datasets can limit variation even at low guidance scales.Lowering guidance harms image quality, and expanding datasets may be infeasible or only partially effective.Source: §1 · p. 2
  • CADS adds scheduled Gaussian noise to the conditioning signal, decreasing it to zero during inference to diversify samples while restoring condition dependence late.The method is intended to let the broader data distribution influence early sampling while preserving alignment with the input condition.Source: §1 · p. 2
  • CADS requires no retraining, adds minimal computational overhead, and integrates with all diffusion samplers.The overhead consists only of an additive operation.Source: §1 · p. 2
  • CADS resolves the diversity–quality trade-off in conditional diffusion models, outperforms standard DDPM sampling across several conditional tasks, and achieves state-of-the-art FID on class-conditional ImageNet at 256×256 and 512×512 resolutions.These results use higher guidance values.Source: §1 · p. 2

2 RELATED WORK

Prior diffusion research has advanced architectures, training, guidance, and sampling efficiency across many domains, while diversity remains a comparatively underaddressed concern. CADS builds on this literature as a sampler-agnostic method focused directly on diversity.

2 RELATED WORK

  • Diffusion models have achieved strong fidelity and diversity across image, text-to-image, translation, motion, and audio generation.The related work situates diffusion models as state-of-the-art across these domains.Source: §2 · pp. 2–3
  • Architectural, training, and guidance advances have improved diffusion models while exposing a quality–diversity balancing problem similar to GAN truncation.Guidance mechanisms provide a way to balance sample quality and diversity.Source: §2 · p. 3
  • Sampling-efficiency methods reduce the number of diffusion steps but do not necessarily address the output-diversity issue.Standard DDPM sampling uses 1000 steps and can be computationally challenging.Source: §2 · p. 3
  • CADS targets similarity among outputs under a fixed condition rather than direct memorization of training data or sampling specifically from low-density regions.Unlike Sehwag et al. (2022), CADS is also compatible with latent diffusion models.Source: §2 · p. 3
  • Unlike one-time input-noising schemes in cascaded models, CADS uses scheduled noise during inference and does not require retraining with noisy inputs.The paper presents CADS as applicable to any architecture and sampling algorithm.Source: §2 · p. 3

3 DIVERSITY IN DIFFUSION MODELS

The paper defines diversity as variation across outputs for a fixed condition and attributes low diversity to conditioning-driven concentration in common modes. CADS progressively corrupts and restores the condition during sampling, while Dynamic CFG provides a less effective guidance-only comparison.

3 DIVERSITY IN DIFFUSION MODELS

  • Diversity means generating varied outputs for a fixed condition as the initial random sample changes.Low-diversity models produce similar outputs across different random inputs and cover a narrower subset of the data distribution.Source: §3 · p. 3
  • High guidance on large datasets and limited variation in small-data models both encourage near one-to-one mappings from conditions to images.High guidance is often used for quality, while DeepFashion-scale training can limit variation even at low guidance.Source: §3 · pp. 3–4
  • Reducing guidance can compromise image quality, and larger datasets such as SHHQ only partially mitigate the diversity problem.These alternatives therefore do not fully resolve the observed trade-off.Source: §3 · p. 4

3.1 CONDITION-ANNEALED DIFFUSION SAMPLER (CADS)

  • CADS perturbs the condition with Gaussian noise during inference, then decreases the corruption so condition alignment is restored near the end.The method is designed to break conditioning dependence early and regain it late, and the paper reports direct integration into samplers.Source: §3.1 · p. 4
  • CADS uses a piecewise linear annealing schedule with user-defined thresholds, applying high corruption early and none when t ≤ τ1.The schedule is defined over reverse-time inference from t = 1 to t = 0.Source: §3.1 · p. 4
  • Rescaling the corrupted condition toward its prior mean and standard deviation improves stability at high noise scales but slightly reduces diversity.The mixing factor ψ controls the trade-off between stable sampling and more diverse generations.Source: §3.1 · p. 4

3.2 INTUITION BEHIND CONDITION ANNEALING

  • Condition annealing makes early sampling follow mainly the unconditional score, increasing exploration before the conditional influence grows later.This progression is intended to produce diverse samples while retaining high-quality condition alignment.Source: §3.2 · pp. 4–5

3.3 DYNAMIC CLASSIFIER-FREE GUIDANCE

  • Dynamic Classifier-Free Guidance scales the guidance weight by γ(t), improving diversity over DDPM but underperforming CADS.The paper attributes CADS’s advantage to additional stochasticity and different sampling dynamics.Source: §3.3 · p. 5

4 EXPERIMENTS AND RESULTS

CADS is evaluated across conditional generation tasks, samplers, guidance scales, and ablations, showing broader diversity gains with limited quality loss. The experiments also examine condition alignment, sampler compatibility, and how noise and rescaling choices affect the quality–diversity balance.

4.1 MAIN RESULTS

  • Across conditional tasks at high guidance, CADS consistently improves FID, Recall, and similarity scores while preserving Precision, with smaller gains for already-diverse Stable Diffusion.The comparison covers pose-to-image, identity-conditioned face generation, class-conditional ImageNet, and text-to-image settings.Source: Figure 3 · p. 6, Figure 4 · p. 6, Table 1 · p. 7
  • FID 1.70 and 2.31 establish state-of-the-art ImageNet results at 256×256 and 512×512 without retraining DiT-XL/2.CADS uses higher guidance values while maintaining diversity, surpassing the previous best FID at 256×256 through improved sampling alone.Source: Table 2 · p. 7, §4.1 · p. 7
  • CADS reduces the quality–diversity trade-off as guidance increases, with FID and Recall deteriorating less than under DDPM.Higher guidance scales use more CADS noise by lowering τ1 and increasing s; the gap between CADS and DDPM expands as w_CFG increases.Source: §4.1 · pp. 7–8, Figure 5 · p. 8
  • CADS remains compatible with popular diffusion samplers and consistently improves the base sampler’s diversity without compromising quality.The evaluation includes DDPM and additional off-the-shelf samplers, with improvements reflected in FID and Recall.Source: Table 3 · p. 8, §4.1 · p. 8
  • CADS generally produces more diversity than Dynamic CFG, outperforming it in FID with Recall values of 0.62 versus 0.39.Both methods increase diversity in class-conditional ImageNet generation, while the comparison supports condition annealing beyond simply modulating guidance weight.Source: §4.1 · p. 8, Table 4 · p. 8, Figure 6 · p. 9

4.2 ABLATION STUDIES

  • Noise-scale and cut-off ablations show that too little noise yields minimal diversity gains, whereas excessive noise degrades image quality.For rescaling, stronger regularization typically improves FID but lowers Recall; the authors recommend ψ = 1 unless more diversity is needed.Source: §4.2 · p. 9, Table 6 · p. 9

5 CONCLUSION AND DISCUSSION

CADS preserves the quality benefits of high classifier-free guidance while improving diversity through decreasing Gaussian noise on the conditioning signal during inference. The method requires no retraining, applies across diffusion samplers, and is theoretically linked to score smoothing and Langevin-like dynamics.

5 CONCLUSION AND DISCUSSION

  • CADS maintains high-quality, diverse outputs without retraining pretrained models and can be applied on top of all diffusion samplers.The conclusion also reports a new benchmark in FID for class-conditional ImageNet generation.Source: §5 · p. 9

A BACKGROUND ON DIFFUSION MODELS

  • Diffusion models generate samples by reversing a noise-adding process with a learned denoiser that can accept conditioning signals such as class labels or text prompts.The reverse process can be formulated through an ODE or SDE and solved backward from noisy samples.Source: A BACKGROUND ON DIFFUSION MODELS · p. 15
  • Classifier-free guidance mixes conditional and unconditional denoiser predictions, increasing individual-image quality at the expense of diversity as guidance strength rises.The unconditional branch is trained by randomly replacing conditions with a null condition.Source: A BACKGROUND ON DIFFUSION MODELS · p. 15

B CONDITION ANNEALING AND LANGEVIN DYNAMICS

  • CADS perturbs the condition with Gaussian noise during inference, producing a Langevin-like update whose noise is guided by the diffusion ODE.The condition perturbation introduces an additive term involving the condition gradient, while ordinary reverse-process noise alone is empirically insufficient for diversity.Source: B CONDITION ANNEALING AND LANGEVIN… · p. 16

C CONDITION ANNEALING AS SCORE SMOOTHING

  • In a linear-condition analysis, condition noise smooths the conditional score similarly to ℓ2 regularization, reducing strong-mode dominance and increasing generation diversity.The analysis approximates the posterior over clean inputs as Gaussian and applies the linear condition transformation.Source: C CONDITION ANNEALING AS SCORE SMOOTHING · p. 16
  • On a toy mixture, high-guidance DDPM converges toward component modes, whereas CADS improves coverage without sampling outside the real data support.The sampler also shows behavior similar to the base sampler as NFEs increase while outperforming DDIM at each tested NFE.Source: Figure 8 · p. 17, Figure 9 · p. 17

D ADDITIONAL EXPERIMENTS

Additional experiments show that CADS improves coverage in a controlled toy problem and remains effective across sampling-step counts. Its behavior tracks the base sampler as NFEs change, although a fixed annealing schedule can suit some NFEs better than others.

D.1 A TOY EXAMPLE

  • CADS improves coverage of each component in a 25-component Gaussian mixture without generating samples outside the real data support.High-guidance DDPM instead concentrates samples near each component mode.Source: §D.1 · p. 17

D.2 PERFORMANCE OF CADS VS THE NUMBER OF SAMPLING STEPS

  • CADS maintains a consistent advantage over sampling without condition annealing across different numbers of neural function evaluations.The comparison switches to DDIM at low NFEs because DDPM performs poorly there.Source: §D.2 · p. 17, Table 7 · p. 18
  • A fixed annealing schedule produces a U-shaped FID trend over part of the ImageNet NFE range, suggesting schedule suitability varies with the number of steps.Source: §D.2 · p. 17

D.3 ADDITIONAL EXPERIMENTS WITH DIFFERENT SAMPLERS

Across alternative diffusion samplers, CADS consistently improves diversity without compromising the base sampler’s quality.

D.3 ADDITIONAL EXPERIMENTS WITH DIFFERENT SAMPLERS

  • CADS consistently improves diversity without compromising quality when combined with DDIM, PNDM, or DPM++.The experiments extend pose-to-image comparisons and include Stable Diffusion sampling with PNDM and DPM++.Source: §D.3 · p. 18

D.4 TRAINING WITH NOISY CONDITIONS

Additional experiments show that inference-time condition annealing remains preferable to noisy-condition training or simply lowering guidance. CADS improves diversity while preserving quality, generally outperforming Dynamic CFG, although combining the methods helps mainly at high guidance scales.

D.4 TRAINING WITH NOISY CONDITIONS

  • Training with noisy pose images does not resolve low diversity and can reduce CADS effectiveness because the model associates noisy conditions with specific images.Source: §D.4 · p. 18, Figure 10 · p. 18

D.5 ADDING NOISE TO THE JOINT LOCATIONS

  • Adding noise to joint locations causes blurry boundaries and increases sampling overhead because the pose image must be rendered at every step.Adding noise directly to the pose image produces better diversity and quality.Source: §D.5 · p. 19, Figure 11 · p. 19

D.6 REDUCING THE GUIDANCE SCALE

  • Reducing classifier-free guidance lowers image quality and does not resolve limited diversity in small-dataset models, whereas CADS improves diversity while preserving high-quality images at higher guidance scales.Source: §D.6 · p. 19, Figure 12 · p. 20, Figure 13 · p. 20

D.7 MORE DISCUSSION ON DYNAMIC CFG

  • CADS outperforms Dynamic CFG on pose-to-image generation, with FID 7.73 for CADS versus 9.58 for Dynamic CFG and fewer visual artifacts.Both methods improve diversity over DDPM, but Dynamic CFG produces more artifacts.Source: §D.7 · p. 19, Figure 14 · p. 21

D.8 COMBINING DYNAMIC CFG AND CADS

  • Combining Dynamic CFG with CADS helps at high guidance scales but worsens FID at lower guidance values, so CADS alone is used for consistent performance.Source: §D.8 · p. 20, Table 8 · p. 21

E ADDITIONAL ABLATIONS

Additional ablations examine how CADS performance depends on the initial noise duration, annealing schedule, and noise distribution. Higher guidance benefits from more initial noise, while piecewise-linear schedules perform best and Gaussian noise is not uniquely required.

E ADDITIONAL ABLATIONS

  • The additional ablations evaluate the effects of τ2, alternative annealing schedules, and alternative noise distributions on CADS performance.These experiments cover the main tunable components of the sampler.Source: E ADDITIONAL ABLATIONS · p. 21, §E.2 · p. 22

E.1 EFFECT OF τ2

  • Reducing τ2 improves FID and Recall at wCFG = 5 but worsens both metrics at wCFG = 2.5, with similar benefits in pose-to-image generation.The results suggest that more initial noise is useful when low diversity is more pronounced.Source: §E.1 · p. 21, Table 9 · p. 22

E.2 VARYING THE ANNEALING SCHEDULE

  • The piecewise-linear annealing schedule with τ1 = 0.6 and τ2 = 0.8 outperforms the tested polynomial schedules, motivating its use for performance and interpretability.FID and Recall change only slightly as the polynomial degree increases.Source: §E.2 · p. 22, Table 10 · p. 22

E.3 CHANGING DISTRIBUTION OF THE NOISE

  • CADS is insensitive to whether the added noise is Uniform, Gaussian, Laplace, or Gamma; only its standard deviation matters and can be absorbed into the noise scale.The experiments therefore use Gaussian noise throughout.Source: §E.3 · p. 22

F LIMITATION OF IS AND PRECISION

The paper finds that IS and Precision can favor low-diversity samples and should therefore be interpreted cautiously. Comparisons with real images indicate that FID and Recall better reflect the desired combination of realism and diversity.

F LIMITATION OF IS AND PRECISION

  • IS and Precision can appear artificially high for DDPM because limited diversity improves these scores, whereas FID and Recall perform better on real data.The paper uses this comparison to argue that FID better captures realism and diversity than IS and Precision.Source: F LIMITATION OF IS AND PRECISION · p. 22, Table 12 · p. 23
  • CADS lowers IS and Precision relative to DDPM without falling below real-image values, indicating greater diversity while maintaining high image quality.This metric behavior motivates cautious interpretation of IS and Precision for diverse outputs.Source: F LIMITATION OF IS AND PRECISION · p. 22

G IMPLEMENTATION DETAILS

The implementation details provide pseudocode, sampling settings, and training procedures for reproducing CADS experiments. CADS adds the annealed conditioning operation within the sampling loop while using task-specific evaluation budgets and LDM-based training for two datasets.

G IMPLEMENTATION DETAILS

  • CADS sampling requires classifier-free guidance strength, an input condition, an annealing schedule γ(t), and an initial noise scale s.The algorithm initializes a Gaussian latent, iterates through reverse-diffusion steps, and returns the final sample.Source: G IMPLEMENTATION DETAILS · p. 23
  • FID uses 10k samples in most experiments and 50k for the state-of-the-art comparison, with 250 sampling steps for Table 2 and 100 elsewhere.Stable Diffusion evaluation uses 1,000 COCO captions and 10 samples per prompt.Source: G IMPLEMENTATION DETAILS · p. 23
  • DeepFashion and SHHQ models use publicly released LDM architectures and hyperparameters, with VQGAN autoencoders trained from scratch before latent-space velocity-prediction diffusion training.The implementation follows the cited LDM and VQGAN procedures.Source: G IMPLEMENTATION DETAILS · p. 23

H MORE VISUAL RESULTS

Additional visual results cover ImageNet, pose-to-image, and identity-conditioned face generation, while the appendix supplies CADS and Dynamic CFG pseudocode. The examples vary guidance settings, resolutions, conditions, and random seeds to illustrate the reported sampling behavior.

H MORE VISUAL RESULTS

  • Figures 16–18 provide additional uncurated CADS samples for class-conditional ImageNet at 256×256 and 512×512 under several guidance scales and classes.The examples include wCFG = 5, 7, and 6 for the respective ImageNet visualizations.Source: H MORE VISUAL RESULTS · p. 23, Figure 16 · p. 26, Figure 17 · p. 27
  • The appendix includes CADS inputs and sampling-loop operations, including condition handling, classifier-free guidance, reverse diffusion, and Dynamic CFG.The pseudocode specifies the annealing schedule and noise addition used by CADS.Source: Figure 15 · p. 25, H MORE VISUAL RESULTS · p. 25
  • Figures 19–21 compare DDPM and CADS on fixed poses using eight identical random seeds per row for DeepFashion and SHHQ.The fixed-pose, matched-seed layout supports visual comparison of variation across generated samples.Source: Figure 19 · p. 29, Figure 20 · p. 30, Figure 21 · p. 31
  • Figures 22–23 show identity-conditioned face samples arranged in 4×4 grids with a fixed identity vector as input.These examples extend the qualitative evaluation to identity-conditioned synthesis.Source: H MORE VISUAL RESULTS · p. 24, Figure 22 · p. 32, Figure 23 · p. 33

Related papers

Loading 2310.17347v4…