PaperlayerView exact revision on arXiv

Source-linked AI summary

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

Zhisheng Xiao, Karsten Kreis, Arash Vahdat

arXiv:2112.07804v2cs.LGstat.ML

TL;DR

Generative models struggle to achieve high quality, broad mode coverage, and fast sampling together, while diffusion models are especially slow because Gaussian denoising suits only small steps. The paper introduces denoising diffusion GANs, which use multimodal conditional GANs for large denoising steps, and reports competitive quality and diversity with substantially faster sampling. Its scope includes discrete-time diffusion assumptions and leaves other expressive conditional generators for future work.

  • Problem

    Generative models struggle to achieve high sample quality, mode coverage, and fast sampling simultaneously, while diffusion models combine quality and diversity with costly sampling.

  • Method

    Denoising diffusion GANs model each reverse-process denoising distribution with a multimodal conditional GAN, allowing larger denoising steps.

  • Results

    Denoising diffusion GANs achieve sample quality and diversity competitive with diffusion models, about 2000× faster sampling on CIFAR-10, and greater diversity than traditional GANs.

  • Takeaways & Limitations

    The reported speed-ups make diffusion models applicable to interactive, real-world image generation and editing at low computational cost.

  • Takeaways & Limitations

    The paper focuses on discrete-time diffusion models and leaves exploration of other expressive conditional generators to future work.

Abstract

from arXiv · show

A wide variety of deep generative models has been developed in the past decade. Yet, these models often struggle with simultaneously addressing three key requirements including: high sample quality, mode coverage, and fast sampling. We call the challenge imposed by these requirements the generative learning trilemma, as the existing models often trade some of them for others. Particularly, denoising diffusion models have shown impressive sample quality and diversity, but their expensive sampling does not yet allow them to be applied in many real-world applications. In this paper, we argue that slow sampling in these models is fundamentally attributed to the Gaussian assumption in the denoising step which is justified only for small step sizes. To enable denoising with large steps, and hence, to reduce the total number of denoising steps, we propose to model the denoising distribution using a complex multimodal distribution. We introduce denoising diffusion generative adversarial networks (denoising diffusion GANs) that model each denoising step using a multimodal conditional GAN. Through extensive evaluations, we show that denoising diffusion GANs obtain sample quality and diversity competitive with original diffusion models while being 2000$\times$ faster on the CIFAR-10 dataset. Compared to traditional GANs, our model exhibits better mode coverage and sample diversity. To the best of our knowledge, denoising diffusion GAN is the first model that reduces sampling cost in diffusion models to an extent that allows them to be applied to real-world applications inexpensively. Project page and code can be found at https://nvlabs.github.io/denoising-diffusion-gan

1 INTRODUCTION

The generative learning trilemma is the difficulty of achieving high sample quality, broad mode coverage, and fast sampling simultaneously. The paper addresses diffusion models’ slow sampling by replacing Gaussian denoising with multimodal conditional GANs for large denoising steps.

  • Generative learning trilemma: The generative learning trilemma requires high-quality sampling, mode coverage and diversity, and fast, inexpensive sampling.Existing frameworks often trade some of these requirements against others.
  • Existing generative models: GANs generate high-quality samples rapidly but have poor mode coverage, whereas VAEs and normalizing flows cover modes faithfully but often produce lower-quality samples.
  • Diffusion models: Diffusion models achieve strong sample quality and mode coverage, but sampling often requires thousands of denoising steps.
  • Proposed direction: The paper attributes diffusion models’ slow sampling to a Gaussian denoising assumption that is justified only for small denoising steps.
  • Proposed model and results: Denoising diffusion GANs use conditional GANs to model multimodal denoising distributions, taking as few as two steps and achieving about 2000× speed-up on CIFAR-10.Their sample quality and mode coverage are competitive with diffusion models, while diversity significantly outperforms state-of-the-art GANs and fidelity remains competitive.
  • Practical consequence: The authors report several orders of magnitude speed-up for image generation and editing, enabling low-cost interactive real-world applications.

2 BACKGROUND

Diffusion models add noise through a predefined forward process and learn a reverse denoising process by matching parameterized and true denoising distributions. Their standard Gaussian parameterization is tied to small steps and typically hundreds to thousands of denoising iterations.

  • Forward process: Diffusion models gradually add noise to data x0 drawn from q(x0) over T forward steps using a predefined variance schedule βt.
  • Reverse process: The reverse process denoises xt into xt−1 using a parameterized distribution pθ(xt−1|xt).
  • Training objective: Training maximizes the data likelihood through an evidence lower bound that matches q(xt−1|xt) with pθ(xt−1|xt) using KL divergence.
  • Denoising distributions: For small denoising steps, the true denoising distribution is close to Gaussian, whereas larger steps produce more complex and multimodal distributions.The figure illustrates this change by showing the evolving data distribution and denoising distributions conditioned on a fixed noisy state.
  • Alternative formulation: The ELBO can also be expressed using tractable distributions and is equivalent to score-based training with denoising score matching.
  • Common assumptions: Standard diffusion models commonly assume Gaussian denoising distributions and use hundreds to thousands of denoising steps.The paper focuses on discrete-time models and notes analogous small-timestep assumptions during continuous-time discretization.

3 DENOISING DIFFUSION GANS

The paper replaces Gaussian denoising distributions with expressive multimodal conditional GANs so diffusion models can take larger steps and sample faster. This approach is designed to preserve mode coverage and sample quality while reducing denoising steps.

  • 3.1 MULTIMODAL DENOISING DISTRIBUTIONS FOR LARGE DENOISING STEPS: Large denoising steps can make the true denoising distribution complex and multimodal when the data distribution is non-Gaussian.
  • 3.2 MODELING DENOISING DISTRIBUTIONS WITH CONITIONAL GANS: The denoising diffusion GAN models each denoising distribution with a conditional GAN.
  • 3.2 MODELING DENOISING DISTRIBUTIONS WITH CONITIONAL GANS: The model uses a small number of diffusion steps, with T ≤8 and larger βt values than conventional diffusion models.
  • 3.2 MODELING DENOISING DISTRIBUTIONS WITH CONITIONAL GANS: The discriminator judges whether xt−1 is a plausible denoised version of xt, contrasting generated samples with samples from the true denoising distribution.
  • 3.2 MODELING DENOISING DISTRIBUTIONS WITH CONITIONAL GANS: The generator samples x0 from xt using a random latent variable, allowing the denoising distribution to become multimodal rather than unimodal.
  • 3.2 MODELING DENOISING DISTRIBUTIONS WITH CONITIONAL GANS: Conditional VAEs performed poorly in early experiments, so the paper focuses on conditional GANs and leaves other expressive conditional generators for future work.

4 RELATED WORK

Prior work addresses diffusion-model sampling costs through distillation, schedules, alternative processes, or improved solvers, but these approaches can retain many steps or degrade sample quality. Related expressive denoising models can also remain slow because of expensive sampling procedures.

  • Diffusion and score-based models commonly sample slowly because generation requires many iterative steps.
  • Existing acceleration methods include knowledge distillation, adaptive noise schedules, non-Markovian processes, and improved SDE solvers.
  • These methods either suffer significant sample-quality degradation or still require many sampling steps in the paper’s evaluation.
  • Gao et al. model single-step denoising with a conditional energy-based model, but MCMC sampling requires about 180 network evaluations.
  • The paper is also related to GAN methods that improve sample quality and diversity through augmentation, consistency regularization, or entropy regularization.

5 EXPERIMENTS

The experiments evaluate denoising diffusion GANs across sample fidelity, diversity, sampling speed, mode coverage, resolution, and synthesis applications. Results show competitive quality and diversity with substantially fewer denoising steps, while ablations support multimodal denoising and multi-step generation.

  • CIFAR-10 evaluation: The CIFAR-10 evaluation measures sample fidelity with FID and IS, diversity with improved recall, and sampling cost with NFE and batch-generation clock time.Sampling time is measured for 100 images on a V100 GPU.
  • CIFAR-10 results: 2000× faster sampling than predictor-corrector diffusion and approximately 20× faster than FastDDPM is achieved using only 4 denoising steps.The model’s sample quality is reported as competitive among leading diffusion models and GANs.
  • CIFAR-10 results: GAN recall scores below 0.5 indicate limited sample diversity, whereas denoising diffusion GANs obtain competitive quality while improving diversity relative to GANs.StyleGAN2 with adaptive data augmentation is the only GAN identified as having slightly better sample quality.
  • Ablation studies: With T = 1, the model has low sample diversity, while T = 4 gives the best results and larger T produces a slight performance degradation.The authors hypothesize that larger T requires higher capacity because each denoising step uses a conditional GAN.
  • Ablation studies: Removing latent variables makes the denoising distribution unimodal and significantly worsens sample quality, while latent variables yield diverse samples conditioned on the same noisy observation.The conditioning information is largely preserved across the generated samples.
  • Additional studies: On 25-Gaussians, denoising diffusion GAN covers all modes while maintaining high sample quality, whereas diffusion models require 100 or 500 steps to maintain quality.Vanilla GAN suffers severe mode collapse, and WGAN-GP improves coverage but has limited sample quality.
  • Additional studies: On StackedMNIST, denoising diffusion GAN covers all 1000 modes faithfully and achieves the lowest KL divergence among the compared models.The comparison includes mode-coverage-oriented GANs and StyleGAN2.
  • High-resolution images: On 256 × 256 CelebA-HQ and LSUN Church, the model obtains competitive FID, outperforming DDPM and ImageBART on LSUN Church.Some GANs perform better on LSUN Church, but FID does not reflect their mode coverage.

6 CONCLUSIONS

The paper concludes that denoising diffusion GANs address the generative learning trilemma by replacing small-step Gaussian denoising with multimodal denoising distributions. Experiments report competitive quality and diversity with orders-of-magnitude faster sampling and better mode coverage than traditional GANs.

  • Conclusion: Denoising diffusion GANs use complex multimodal distributions to take large denoising steps instead of relying on Gaussian distributions justified only for very small steps.The reverse process is modeled with conditional GANs.
  • Conclusion: Denoising diffusion GANs achieve sample quality and diversity competitive with original diffusion models while sampling orders of magnitude faster.The conclusion presents this as enabling lower-cost application of diffusion models to real-world problems.
  • Conclusion: Compared with traditional GANs, the proposed model has better mode coverage and sample diversity.

7 ETHICS AND REPRODUCIBILITY STATEMENT

The paper discusses the importance of diversity for reducing bias and improving minority representation, and reports that its evaluation uses public datasets without human-subject evaluation or new data collection. Reproducibility materials include experimental details, with source-code release planned.

  • Ethics: Mode coverage and high diversity are presented as important for reducing biases in generative models and improving representation of minorities in a population.
  • Ethics: The evaluation uses public datasets and involves neither human-subject evaluation nor new data collection.
  • Reproducibility: Experimental details, hyperparameters, and training settings are provided in appendices, while source code is planned for future public release.

A DERIVATION FOR THE GAUSSIAN POSTERIOR

This section derives the Gaussian posterior of the forward diffusion process and explains the equivalent DDPM denoising parametrization. The derivation connects noise prediction to clean-data prediction and posterior sampling.

  • Posterior derivation: The Markov property gives the marginal distribution of x_t conditioned on the initial clean data x_0.
  • Posterior derivation: Applying Bayes’ rule to the forward process yields the posterior q(x_t−1|x_t, x_0).
  • Posterior derivation: Because the three factors are Gaussian, the posterior q(x_t−1|x_t, x_0) is Gaussian with mean ˜µ_t(x_t, x_0) and variance ˜β_t.
  • DDPM parametrization: DDPM trains a noise predictor ϵ_θ(x_t, t), which can be converted into a prediction of x_0.
  • DDPM parametrization: The DDPM denoising step is equivalent to predicting x_0 and then sampling x_t−1 from the Gaussian posterior conditioned on x_t and that prediction.
  • Network implementation: The experimental implementation uses a U-net-like generator with residual and attention blocks, sinusoidal time embeddings, and an extra latent variable z.
  • Network implementation: The discriminator distinguishes real and fake x_t−1 conditioned on x_t and t, using concatenated inputs and time conditioning.

C.2 TRAINING

The training setup uses four diffusion steps, adversarial denoising objectives, regularization, and dataset-specific optimization and evaluation procedures.

  • Diffusion Process: The diffusion process uses T = 4 steps with a variance-preserving SDE schedule and equidistant time steps.The schedule uses βmax = 20 and βmin = 0.1; alternative schedules were not explored.
  • Objective: The denoising diffusion GAN is trained with an adversarial objective whose implicit denoising distribution is pθ(xt−1|xt).The outer expectation represents ancestral sampling from q(x0, xt−1, xt).
  • Objective: R1 regularization is added to the discriminator objective, using γ = 0.05 for CIFAR-10 and γ = 1 for CelebA-HQ and LSUN Church.The regularizer is a gradient penalty intended to improve GAN convergence.
  • Optimization: Optimization uses Adam, cosine learning-rate decay, and exponential moving averages of the generator.The EMA is described as crucial for high performance.
  • Compute and Evaluation: Training uses 4 V100 GPUs for CIFAR-10 and 8 for CelebA-HQ and LSUN Church, taking approximately 48 and 180 hours, respectively.Sampling-time benchmarks generate 100 CIFAR-10 samples on a single V100 GPU.
  • Ablations: Ablation models are trained until FID stops increasing, requiring 200k, 400k, and 600k iterations for T = 1 or 2, T = 4, and T = 8.The main experiments use the same network structures and optimization settings across these ablations.

D TRAINING STABILITY

Training remains stable across denoising time steps when T = 4, with higher discriminator loss on noisy intermediate images than at the final denoising step.

  • Training Stability: Training is stable for T = 4, without the loss explosions sometimes reported for other GAN methods.The authors attribute this partly to conditioning both generator and discriminator on xt and to the smoothing effect of diffusion.
  • Training Stability: Discriminator loss is higher for t > 0 than for t = 0 because noisy real and generated distributions are closer and harder to distinguish.Here, t > 0 denotes training on noisy images, while t = 0 is the last denoising step.

E ADDITIONAL QUALITATIVE RESULTS

Additional visualizations show generated samples across CIFAR-10, CelebA-HQ, and LSUN Church, plus intermediate conditional samples at different diffusion times.

  • Additional Samples: Additional qualitative samples are shown for CIFAR-10, CelebA-HQ, and LSUN Church Outdoor.These visualizations supplement the quantitative evaluations with examples from all three image datasets.
  • Intermediate Visualizations: Samples from pθ(x0|xt) become less conditioned on clean data as perturbation increases, and x4 contains almost no information about x0.Intermediate samples need not be sharp except for pθ(x0|x1), because they occur within the sampling process.

G NEAREST NEIGHBOR RESULTS

Nearest-neighbor analyses compare generated images with training-set neighbors in pretrained VGG feature space and report substantial differences between them.

  • Nearest-Neighbor Comparisons: Generated samples appear significantly different from their training-set nearest neighbors in VGG feature space, suggesting that the models generalize well.Generated samples are placed in the leftmost column, with corresponding training-set nearest neighbors in the remaining columns.
  • Nearest-Neighbor Comparisons: The nearest neighbors are computed using feature distance from a pretrained VGG network.The comparison is performed for CIFAR-10 and CelebA-HQ generated samples.
Loading 2112.07804v2…