PaperlayerView exact revision on arXiv

Source-linked AI summary

Denoising Diffusion Probabilistic Models

Jonathan Ho, Ajay Jain, Pieter Abbeel

arXiv:2006.11239v2cs.LGstat.ML

TL;DR

Diffusion models had not yet been shown capable of generating high-quality samples. This paper connects diffusion models with denoising score matching and annealed Langevin dynamics, achieving a CIFAR10 FID score of 3.17 and high-quality image synthesis.

  • Problem

    Diffusion models were straightforward to define and train, but lacked demonstrated evidence of generating high-quality samples.

  • Method

    The paper trains diffusion models with a reweighted variational objective linked to denoising score matching and annealed Langevin dynamics.

  • Results

    3.17 FID: the unconditional CIFAR10 model achieves better sample quality than most models in the literature.

  • Takeaways & Limitations

    Diffusion models emerge as a generally useful deep generative modeling tool with connections to variational inference, score matching, Langevin dynamics, autoregressive models, and progressive compression.

  • Takeaways & Limitations

    The models do not have competitive log likelihoods compared with other likelihood-based models, and most lossless codelength describes imperceptible image details.

Abstract

from arXiv · show

We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN. Our implementation is available at https://github.com/hojonathanho/diffusion

1 Introduction

The paper develops diffusion probabilistic models as efficiently trained Markov chains that reverse gradual noise addition, demonstrating high-quality image generation and connections to denoising score matching. Despite strong sample quality, the models have noncompetitive log likelihoods, with much lossless codelength spent on imperceptible image details.

  • Diffusion probabilistic models: Diffusion models are parameterized Markov chains trained with variational inference to reverse a process that gradually adds noise until signal is destroyed.Their learned reverse transitions produce samples matching the data after finite time.
  • Contributions: The paper shows that diffusion models can generate high-quality samples, sometimes exceeding published results for other generative model types.This addresses the prior lack of demonstrations that diffusion models could generate high-quality samples.
  • Connections: A certain diffusion-model parameterization reveals an equivalence with denoising score matching.
  • Limitation: Despite their sample quality, the models do not have competitive log likelihoods compared with other likelihood-based models.The models’ lossless codelengths are mostly consumed by imperceptible image details.

2 Background

Diffusion models define a learned Gaussian reverse-time Markov chain from noise and a fixed forward Markov chain that gradually adds Gaussian noise to data. Training optimizes a variational bound, with efficient stochastic-gradient estimation and closed-form Gaussian KL computations.

  • Reverse process: The reverse process is a learned Gaussian Markov chain initialized from p(x_T) = N(x_T; 0, I).Its transitions are p_θ(x_{t−1}|x_t) := N(x_{t−1}; μ_θ(x_t, t), Σ_θ(x_t, t)).
  • Forward process: The forward diffusion process is a fixed Markov chain that gradually adds Gaussian noise according to a variance schedule β_1, . . . , β_T.This fixed approximate posterior distinguishes diffusion models from other latent variable models.
  • Training objective: The usual variational bound on negative log likelihood is the training objective.The forward-process variances β_t may be learned by reparameterization or held constant as hyperparameters.
  • Training objective: Random terms of the variational objective can be optimized efficiently with stochastic gradient descent.Variance reduction is obtained by rewriting L as described in Equation (5).
  • KL computation: All KL divergences in Equation (5) compare Gaussian distributions and can therefore be computed in closed form using Rao-Blackwellization.The comparisons are between reverse-process transitions and tractable forward-process posteriors conditioned on x_0.

3 Diffusion models and denoising autoencoders

This section connects diffusion models to denoising score matching, motivating an ε-prediction parameterization and a simplified weighted variational objective. It also specifies fixed forward variances and reverse-process design choices that make training practical and sampling Langevin-like.

  • Model design: Diffusion models offer implementation flexibility through choices of forward variances, reverse-process architecture, and Gaussian parameterization.The authors use a connection to denoising score matching to guide these choices.
  • Model design: Fixing forward variances βt makes the approximate posterior q parameter-free, so LT is constant during training and can be ignored.The variances are fixed to constants by reparameterization.
  • Reverse process: The reverse process uses untrained time-dependent isotropic variances, with σt^2 = βt or σt^2 = (1−ᾱt−1)/(1−ᾱt)βt yielding similar results.The first choice is optimal for Gaussian data, while the second is optimal when data are deterministically set to one point.
  • Denoising score matching: The ε-prediction parameterization makes reverse sampling resemble Langevin dynamics and reduces the variational-bound term to a denoising score-matching objective across noise scales.The model predicts noise ε from xt, while the learned function acts like a gradient of data density.
  • Training objective: The authors train with a simplified weighted variational bound that samples t uniformly and removes the weighting from the denoising-score-matching terms.This objective omits LT because βt is fixed and down-weights small-t terms that correspond to denoising with very little noise.

4 Experiments

Experiments show that diffusion models achieve strong image sample quality while exhibiting a tradeoff between likelihood-based codelength and perceptual quality. They also support progressive lossy compression, generation, and smooth latent interpolation, with large-scale features emerging before fine details.

  • Image quality: 3.17 FID on unconditional CIFAR10 demonstrates sample quality better than most reported models, including class-conditional models.Measured against the training set; the corresponding test-set FID is 5.24.
  • Training objectives: Training on the true variational bound improves codelengths, whereas the simplified objective produces the best sample quality.This tradeoff is reported for CIFAR10 and CelebA-HQ 256 × 256 samples, as well as LSUN 256 × 256 samples.
  • Training objectives: The baseline ˜µ parameterization works well only with the true variational bound, while learning reverse-process variances causes unstable training and poorer sample quality.The comparison concerns reverse-process parameterizations and training objectives.
  • Lossy compression: 1.78 bits/dim rate and 1.97 bits/dim distortion characterize the highest-quality CIFAR10 model, corresponding to 0.95 root mean squared error on a 0–255 scale.More than half of the lossless codelength describes imperceptible distortion.
  • Progressive lossy compression: The progressive rate-distortion curve drops steeply at low rates, indicating that most transmitted bits are allocated to imperceptible distortions.Distortion is measured as root mean squared error, while rate is the cumulative number of bits received.
  • Progressive generation and interpolation: Progressive generation produces large-scale image features first and details later, while latent interpolation yields smooth, plausible changes in pose, skin tone, hairstyle, expression, and background.At t = 1000, interpolation produces novel samples.

5 Related Work

Diffusion models resemble flows and VAEs but use a parameter-free forward process with a nearly information-free top-level latent. Their reverse parameterization connects them to denoising score matching, annealed Langevin dynamics, energy-based modeling, rate-distortion analysis, and progressive decoding.

  • 5 Related Work: Diffusion models resemble flows and VAEs, but q has no parameters and xT has nearly zero mutual information with x0.These design choices distinguish diffusion models from the related model families discussed.
  • 5 Related Work: The ϵ-prediction reverse-process parameterization connects diffusion models to denoising score matching across multiple noise levels and annealed Langevin dynamics for sampling.The passage identifies this connection as a consequence of the reverse-process parameterization.
  • 5 Related Work: The connection between score matching and energy-based modeling suggests implications for recent energy-based models.The passage presents this as a possible implication rather than a demonstrated result.
  • 5 Related Work: Rate-distortion curves are computed over time in one variational-bound evaluation, resembling curves computed over distortion penalties in one annealed-importance-sampling run.The comparison concerns how the curves are computed, not their numerical values.
  • 5 Related Work: The progressive decoding argument can be viewed as related to convolutional DRAW and related models.The supplied passage introduces this relationship but ends before providing further detail.

6 Conclusion

The paper presents high-quality image samples from diffusion models and identifies connections to several established modeling and inference frameworks. It proposes investigating diffusion models in other data modalities and as components of broader systems.

  • Diffusion models produce high-quality image samples.
  • The paper connects diffusion models with variational inference for training Markov chains, denoising score matching, and annealed Langevin dynamics.
  • It also relates diffusion models to energy-based models, autoregressive models, and progressive lossy compression.
  • The authors plan to investigate diffusion models in other data modalities and as components.

Broader Impact

The work advances diffusion models as a generally useful deep generative modeling tool, potentially amplifying both beneficial applications and known societal risks. It may support compression, representation learning, and creative uses, while also making malicious synthetic media easier to produce.

  • Scope and impact: Diffusion models advance a generally useful family of deep generative models and may amplify their broader societal impacts.The work is positioned alongside efforts on GANs, flows, and autoregressive models.
  • Risks: Generative models can facilitate malicious creation of fake images and videos of high-profile figures for political purposes.The passage notes that these models make synthetic-media production easier, although current CNN-generated images retain detectable flaws.
  • Potential benefits: Diffusion models may support data compression as rising resolution and global internet traffic increase the need for internet accessibility.The passage presents compression as a potentially crucial application for serving wide audiences.
  • Potential benefits: Diffusion models might contribute to representation learning on unlabeled raw data for tasks ranging from image classification to reinforcement learning.The passage identifies a broad range of downstream applications.
  • Potential benefits: Diffusion models might become viable for creative uses in art, photography, and music.

Extra information

This section identifies the reported LSUN FID-score tables and clarifies that the progressive compression scheme is currently a proof of concept rather than a practical system. It also references CIFAR10 rate-distortion results accompanying Figure 5.

  • LSUN FID scores: Scores marked with ∗ are StyleGAN2 baselines, while other scores come from their respective authors.The passage specifies the reporting provenance for the LSUN comparisons.
  • LSUN FID scores: Table 3 reports FID scores for 256 × 256 LSUN Bedroom, Church, and Cat datasets.The listed model columns are Progressive compression, LSUN Bedroom, LSUN Church, and LSUN Cat.
  • Progressive compression: The progressive compression argument is only a proof of concept because Algorithms 3 and 4 rely on intractable minimal random coding for high-dimensional data.The algorithms interpret the variational bound (5) as compression, not as a practical compression system.
  • CIFAR10 rate-distortion: Table 4 presents unconditional CIFAR10 test-set rate-distortion values accompanying Figure 5.No numerical values are provided in the supplied passage.

A Extended derivations

This section derives Eq. (5), the reduced-variance variational bound for diffusion models, and presents an alternate form of L that is intractable to estimate but useful for Section 4.3.

  • Derivation of Eq. (5): Eq. (5) is derived as the reduced-variance variational bound for diffusion models.The derivation is included from Sohl-Dickstein et al. [53] for completeness.
  • Alternate objective: An alternate version of L is not tractable to estimate but supports the discussion in Section 4.3.This alternate formulation is presented for analytical discussion rather than direct estimation.

B Experimental details · C Discussion on related work

The experiments use a PixelCNN++-inspired U-Net architecture with dataset-specific training and evaluation procedures. The related-work discussion distinguishes the model from NCSN in architecture, time conditioning, and forward-process scaling.

  • B Experimental details: The architecture follows PixelCNN++’s U-Net-based Wide ResNet backbone, replaces weight normalization with group normalization, and adds self-attention at 16 × 16 resolution.32 × 32 models use four resolutions, while 256 × 256 models use six; each level has two convolutional residual blocks.
  • B Experimental details: All experiments used TPU v3-8; CIFAR10 training ran at 21 steps per second, while larger-image models ran at 2.2 steps per second.CIFAR10 used batch size 128; CelebA-HQ and LSUN used batch size 64.
  • B Experimental details: Most hyperparameter search optimized CIFAR10 sample quality before transferring the resulting settings to the other datasets.An initial hyperparameter choice was made to fit network size within memory constraints.
  • B Experimental details: The selected noise schedule used T = 1000 and a linear β schedule from β1 = 10−4 to βT = 0.02, while CIFAR10 dropout was set to 0.1 by sweeping four values.Without dropout on CIFAR10, samples showed poorer quality resembling overfitting artifacts in unregularized PixelCNN++.
  • B Experimental details: Batch sizes were 128 for CIFAR10 and 64 for larger images, and model parameters used EMA with decay factor 0.9999.Neither batch size nor EMA decay was swept.
  • B Experimental details: Final experiments were trained once, evaluated throughout training, and reported sample-quality scores and log likelihood at the minimum FID value.CIFAR10 Inception and FID scores used 50000 samples, and LSUN FID scores also used 50000 samples.
  • C Discussion on related work: Compared with NCSN, the model changes its architecture, forward process, prior, and training approach in ways reported to improve sample quality.The sampler is trained directly as a latent variable model rather than added post-hoc.
  • C Discussion on related work: The model uses a self-attention U-Net and conditions all layers on t through Transformer sinusoidal position embeddings, while NCSN uses a RefineNet with dilated convolutions.Diffusion models also scale down data at each forward step so variance does not grow when noise is added.

D Samples

The samples section examines stochasticity in the reverse process and shows diffusion-based interpolations spanning fine source-preserving mixtures to novel samples. It also presents uncurated generated samples across CelebA-HQ, CIFAR10, and LSUN, including reported LSUN FID values.

  • Additional samples: Uncurated samples are shown for diffusion models trained on CelebA-HQ, CIFAR10, and LSUN datasets.The figures include generated samples and progressive generation or nearest-neighbor comparisons for several datasets.
  • Latent structure and reverse process stochasticity: Reverse-process stochasticity produces multiple CelebA 256 × 256 images from the same intermediate latent.The prior xT ∼ N(0, I) and Langevin dynamics are both stochastic, enabling multiple draws from pθ(x0|xt).
  • Coarse-to-fine interpolation: Increasing diffusion steps before latent interpolation destroys more source structure, which the reverse process completes.This supports interpolation at both fine and coarse granularities.
  • Coarse-to-fine interpolation: At 0 diffusion steps, interpolation mixes source images in pixel space; at 1000 steps, source information is lost and interpolations are novel samples.These are the limiting cases of the coarse-to-fine interpolation procedure.
  • Additional samples: FID=7.89 for LSUN Church, FID=4.90 for LSUN Bedroom generated samples from the large model, FID=6.36 for the small model, and FID=19.75 for LSUN Cat.These values are reported in the corresponding sample figure captions.
Loading 2006.11239v2…