PaperlayerView exact revision on arXiv

Source-linked AI summary

Denoising Diffusion Implicit Models

Jiaming Song, Chenlin Meng, Stefano Ermon

arXiv:2010.02502v4cs.LGcs.CV

TL;DR

DDPMs generate high-quality images but require many sequential diffusion steps, making sampling slow. DDIMs use non-Markovian implicit processes with the same training objective and achieve comparable sample quality with 10× to 50× faster sampling, while supporting meaningful latent-space interpolation.

  • Problem

    DDPMs require many sequential sampling iterations, creating a speed and latency gap that limits their practicality when computation is constrained.

  • Method

    DDIMs construct non-Markovian implicit diffusion processes trained with the same objective as DDPMs and sampled along shorter trajectories.

  • Results

    10× to 50× speedup compared to the original DDPM is achieved while producing samples with comparable quality to 1000-step models within 20 to 100 steps.

  • Takeaways & Limitations

    DDIMs enable trading computation for sample quality and performing semantically meaningful image interpolation by manipulating the initial latent variable.

  • Takeaways & Limitations

    Further methods for reducing discretization error to improve sample quality with fewer steps remain an open research direction.

Abstract

from arXiv · show

Denoising diffusion probabilistic models (DDPMs) have achieved high quality image generation without adversarial training, yet they require simulating a Markov chain for many steps to produce a sample. To accelerate sampling, we present denoising diffusion implicit models (DDIMs), a more efficient class of iterative implicit probabilistic models with the same training procedure as DDPMs. In DDPMs, the generative process is defined as the reverse of a Markovian diffusion process. We construct a class of non-Markovian diffusion processes that lead to the same training objective, but whose reverse process can be much faster to sample from. We empirically demonstrate that DDIMs can produce high quality samples $10 \times$ to $50 \times$ faster in terms of wall-clock time compared to DDPMs, allow us to trade off computation for sample quality, and can perform semantically meaningful image interpolation directly in the latent space.

1 INTRODUCTION

DDIMs address the slow sampling of DDPMs by providing implicit probabilistic models trained with the same objective. They generalize diffusion to non-Markovian processes and offer accelerated sampling with improved quality and latent consistency.

  • Motivation: DDPMs and related iterative models can match GAN-like sample quality without adversarial training, but require many denoising iterations.DDPM generation approximates the reverse of a forward diffusion process that may contain thousands of steps.
  • Contribution: DDIMs are implicit probabilistic models closely related to DDPMs and trained with the same objective function.This enables DDIMs to target the efficiency gap between DDPMs and GANs without changing the training objective.
  • Method: The method generalizes DDPM’s Markovian forward diffusion to non-Markovian processes while retaining a shared surrogate objective equal to the DDPM training objective.Different generative models can therefore use the same neural network by selecting different non-Markovian diffusion processes.
  • Empirical benefits: 10× to 100× acceleration improves DDIM sample generation quality relative to DDPMs under the proposed sampling method.The comparison is made while accelerating sampling by 10× to 100×.
  • Empirical benefits: DDIMs provide consistency across Markov chains of different lengths when generation begins from the same initial latent variable.The resulting samples share similar high-level features, a property the passage states does not hold for DDPMs.

2 BACKGROUND

DDPMs learn latent-variable models using a fixed forward diffusion process and a reverse generative process that approximates it. Although large T improves the Gaussian approximation of reverse transitions, sequential sampling makes T a key computational bottleneck.

  • DDPM formulation: DDPMs learn a model distribution pθ(x0) that approximates the data distribution q(x0) and is easy to sample from.They are latent-variable models with latent variables in the same sample space as the observations.
  • DDPM formulation: Unlike typical latent-variable models, DDPMs use a fixed inference procedure q(x1:T | x0), with relatively high-dimensional latent variables.Their parameters are learned by maximizing a variational lower bound.
  • Forward and generative processes: The forward process progressively adds noise from x0 to xT, while the generative process samples from xT to x0 to approximate the reverse process.The forward process is autoregressive, and the generative process is modeled as a Markov chain with Gaussian conditional distributions.
  • Forward and generative processes: When αT is sufficiently close to 0, q(xT | x0) converges to a standard Gaussian, motivating the prior pθ(xT) := N(0, I).The forward-process variables can be expressed as linear combinations of x0 and a noise variable ϵ.
  • Sampling cost: T = 1000 motivates DDPMs because larger T makes reverse transitions closer to Gaussian, but all T iterations must be performed sequentially for sampling.Thus, the forward-process length T is an important hyperparameter and creates a sampling-time bottleneck.

3 VARIATIONAL INFERENCE FOR NON-MARKOVIAN FORWARD PROCESSES

The section constructs non-Markovian inference processes that preserve the DDPM marginals while changing the joint distribution, enabling alternative reverse processes. Their variational objectives remain equivalent to weighted DDPM objectives, with the same optimal solution as L1 under the stated parameterization.

  • Motivation: The DDPM objective depends on q(x_t|x_0) marginals rather than the joint q(x_1:T|x_0), motivating non-Markovian alternatives with identical marginals.Many inference distributions can share the same marginals while differing in their joint structure.
  • Inference family: The family Q of inference distributions is indexed by a real vector σ and is constructed so q_σ(x_t|x_0) = N(√α_t x_0, (1 − α_t)I) for every t.The mean function is chosen to preserve the desired marginal distributions.
  • Inference family: Unlike the DDPM diffusion, the resulting forward process is non-Markovian because x_t may depend on both x_t−1 and x_0.The process is derived from Bayes’ rule and remains Gaussian, although that fact is not used later.
  • Inference family: The magnitude of σ controls forward-process stochasticity, with σ → 0 making x_t−1 known and fixed given x_0 and x_t.This is the extreme deterministic limit described for the constructed process.
  • Generative process: Given x_t, the generative process predicts x_0 and uses q_σ(x_t−1|x_t, x_0) to obtain x_t−1, while ε_θ(t)(x_t) predicts the noise without observing x_0.Rewriting the forward equation converts the noise prediction into a denoised-observation prediction.
  • Variational objective: Although J_σ appears to require a separate model for each σ, it is equivalent to an L_γ objective, whose optimum matches L1 when parameters are not shared across t.This equivalence means the optimal solution of J_σ is also the optimal solution of L1.

4 SAMPLING FROM GENERALIZED GENERATIVE PROCESSES

The section generalizes DDPM sampling to non-Markovian processes controlled by σ, including deterministic DDIMs that reuse the same trained model. Sampling can be accelerated by traversing shorter latent trajectories, while the continuous-time formulation connects DDIMs to ODE-based generation and observation encoding.

  • Generalized generative processes: Changing σ yields different generative processes while reusing the same model εθ, so pretrained DDPM models require no retraining.The Markovian choice recovers DDPM sampling, whereas other σ values define alternative processes.
  • DDIM: Setting σt = 0 for all t makes the forward process deterministic apart from t = 1 and defines the denoising diffusion implicit model.The resulting implicit probabilistic model generates samples through a fixed latent-variable procedure from xT to x0.
  • Accelerated sampling: Shorter trajectories over a subsequence of latent variables accelerate generation without training a different model.The generative process samples the selected latent variables in reverse trajectory order while preserving the required marginals.
  • Accelerated sampling: A trained model can in principle support arbitrary forward-step counts or even continuous time, although empirical investigation is left for future work.The construction permits sampling only some of the steps used during training.
  • ODE interpretation: DDIM updates resemble Euler integration of an ODE, and reversing the generation process can encode observations as terminal latent variables xT.This suggests downstream applications requiring latent representations, unlike standard DDPM sampling.

5 EXPERIMENTS

Experiments show that DDIMs can generate high-quality images with substantially fewer sampling steps than DDPMs while preserving high-level features tied to the initial latent state. They also support semantic latent interpolation and improved reconstruction at larger encoding and decoding step counts.

  • Image generation: DDIMs achieve 10× to 100× generation speedups over the original DDPM process when fewer iterations are used.The experiments attribute the speedup to producing samples with shorter trajectories.
  • Image generation: Using the same model and training objective, sampling behavior is controlled only through τ, which sets trajectory speed, and σ, which interpolates between deterministic DDIM and stochastic DDPM.The models use T = 1000 and Lγ with γ = 1.
  • Latent structure: For a fixed initial xT, DDIM samples retain similar high-level features across trajectories, with 20-step samples often resembling 1000-step samples.Longer trajectories improve sample quality without significantly changing high-level features.
  • Latent structure: Interpolating xT yields semantically meaningful transitions between samples, and DDIM reconstruction error decreases as the shared encoding and decoding step count S increases.The interpolation directly controls generated images at a high level, while reconstruction is evaluated on the CIFAR-10 test set.

6 RELATED WORK

DDPMs and NCSNs share denoising objectives and Langevin-like sampling, which contributes to their need for many steps. DDIM instead is an implicit model with latent-determined samples, giving it properties resembling GANs and invertible flows, including semantic interpolation.

  • DDPMs and NCSNs: DDPMs and NCSNs use denoising autoencoder objectives across noise levels and Langevin-like sampling procedures.Because Langevin dynamics discretizes gradient flow, both methods require many steps for good sample quality.
  • DDIM: DDIM is an implicit generative model whose samples are uniquely determined by latent variables.This gives DDIM properties resembling GANs and invertible flows.
  • DDIM: DDIM can produce semantically meaningful interpolations, unlike the sampling constraints associated with Langevin dynamics.The method is derived from a purely variational perspective, where those Langevin restrictions are not relevant.

7 DISCUSSION · A NON-MARKOVIAN FORWARD PROCESSES FOR A DISCRETE CASE

DDIMs provide an efficient implicit generative model with meaningful latent interpolations and motivate broader non-Markovian forward processes. The discrete categorical construction yields tractable variational objectives, while empirical evaluation and improved sampling methods remain future directions.

  • 7 DISCUSSION: DDIMs generate high-quality samples more efficiently than DDPMs and NCSNs while supporting meaningful latent-space interpolations.The model is trained with denoising auto-encoding or score-matching objectives from a variational perspective.
  • 7 DISCUSSION: Non-Markovian forward processes suggest continuous alternatives beyond Gaussian diffusion and can extend to a multinomial discrete process.The original diffusion framework is restricted to Gaussian noise as the only stable finite-variance distribution.
  • 7 DISCUSSION: DDIM sampling resembles a neural ODE, motivating multistep ODE solvers to reduce discretization error and improve quality with fewer steps.Adams-Bashforth methods are identified as one possible direction for further improvement.
  • A NON-MARKOVIAN FORWARD PROCESSES FOR A DISCRETE CASE: The discrete case describes a non-Markovian forward process and corresponding variational objectives, with empirical evaluation left for future work.The appendix focuses on discrete data while the paper’s main acceleration results concern Gaussian diffusion.
  • A NON-MARKOVIAN FORWARD PROCESSES FOR A DISCRETE CASE: For one-hot categorical observations with K possible values, q(x_t|x_0) uses a categorical distribution governed by α_t decreasing from α_0 = 1 to α_T = 0.The uniform categorical vector 1_K has all entries equal to 1/K.
  • A NON-MARKOVIAN FORWARD PROCESSES FOR A DISCRETE CASE: q(x_{t−1}|x_t, x_0) is a three-component categorical mixture over x_t, x_0, and the uniform vector 1_K.The mixture probabilities are σ_t, (α_{t−1} − σ_tα_t), and (1 − α_{t−1}) − (1 − α_t)σ_t.
  • A NON-MARKOVIAN FORWARD PROCESSES FOR A DISCRETE CASE: As the uniform-mixture weight approaches zero, sampling becomes less stochastic and increasingly selects either x_t or the predicted x_0.The resulting KL divergence is between categoricals, and convexity provides an upper bound that becomes tight as the right-hand side approaches zero.
  • A NON-MARKOVIAN FORWARD PROCESSES FOR A DISCRETE CASE: The upper-bound objective reduces to a multiclass classification loss up to constants, so changing σ_t affects weighting rather than the objective’s underlying form.The parameterized mapping θ(x_t) produces a K-dimensional vector.

B PROOFS

The proofs establish the Gaussian marginal structure required for the induction argument and show that the DDIM objective equals the corresponding loss up to a constant. They also prove that DDIM’s ODE with the optimal model is equivalent to the probability-flow ODE of the variance-exploding SDE.

  • Lemma 1: The Gaussian form qσ(xt|x0) = N(√αtx0, (1 −αt)I) is preserved by induction from t = T to 1.The proof derives qσ(xt−1|x0) as Gaussian and recovers the corresponding marginal form for t−1.
  • Objective equivalence: The DDIM training objective satisfies Jσ = Lγ + C, where C does not depend on ϵθ.The equivalence follows from the definition of ≡, which ignores terms independent of the model ϵθ.
  • Proposition 1: The DDIM ODE with the optimal model ϵθ is equivalent to the probability-flow ODE for the variance-exploding SDE.The proof reparameterizes DDIM variables into the VE-SDE variables and derives matching ODE forms.
  • Proposition 1: The DDIM and VE-SDE ODEs are identical because both use initial conditions ¯x(T) ∼N(0, σ2(T)I).The equivalence is obtained after relating the perturbed score function to the DDIM parameterization.

C ADDITIONAL DERIVATIONS … D EXPERIMENTAL DETAILS

The additional derivations show how accelerated sampling uses a shortened non-Markovian inference process while preserving the relevant marginals and variational objective. They also reconcile the paper’s notation with Ho et al. (2020) and derive the corresponding DDPM denoising objectives.

  • C.1 ACCELERATED SAMPLING PROCESSES: C.1 Accelerated Sampling Processes: Accelerated inference selects a subsequence τ of [1, . . . , T] with length S and τS = T.The complement is denoted ¯τ := {1, . . . , T} \ τ.
  • C.1 ACCELERATED SAMPLING PROCESSES: C.1 Accelerated Sampling Processes: The subsequence variables form a chain with x0, while omitted variables form a star graph with x0.This separates the accelerated sampling structure from the full graphical model.
  • C.1 ACCELERATED SAMPLING PROCESSES: C.1 Accelerated Sampling Processes: The accelerated process chooses coefficients so that its marginals match those of the original construction.The derivation explicitly characterizes this condition as the marginals matching.
  • C.1 ACCELERATED SAMPLING PROCESSES: C.1 Accelerated Sampling Processes: Sampling uses only part of the models, with qσ,τ(xτi−1|xτi, x0) incorporated into the inference process.The resulting variational objective is expressed using the accelerated subsequence and defines xτL+1 = ∅ for conciseness.
  • C.1 ACCELERATED SAMPLING PROCESSES: C.1 Accelerated Sampling Processes: Each KL divergence compares Gaussians with θ-independent variance, allowing the variational objective J to be converted to an objective of the form Lγ.The conversion follows an argument similar to the proof of Theorem 1.
  • C.2 DERIVATION OF DENOISING OBJECTIVES FOR DDPMS: C.2 Derivation of Denoising Objectives for DDPMs: The paper uses αt for Ho et al.’s ¯αt, reducing hyperparameter cross-references and simplifying the generalization and acceleration derivations.The notation change is also justified by an isomorphism between α1:T and the corresponding sequence in Ho et al. (2020).
  • C.2 DERIVATION OF DENOISING OBJECTIVES FOR DDPMS: C.2 Derivation of Denoising Objectives for DDPMs: Starting from the diffusion forward process and Ho et al.’s chosen p(t) and parametrization, the variational objective is simplified.The derivation presents the forward-process relation, the resulting variational objective, and its simplified form.

D.1 DATASETS AND ARCHITECTURES · D.2 REVERSE PROCESS SUB-SEQUENCE SELECTION · D.3 CLOSED FORM EQUATIONS FOR EACH SAMPLING STEP

The experiments compare generative processes across four image datasets using matched model settings, while DDIM sampling varies the reverse-process timestep subsequence and update stochasticity. The appendix specifies dataset architectures, linear or quadratic timestep selection, and closed-form sampling updates, including why more stochastic updates perform worse with few steps.

  • D.1 DATASETS AND ARCHITECTURES: Four datasets span resolutions from 32 × 32 to 256 × 256: CIFAR10, CelebA, LSUN Bedroom, and LSUN Church.CIFAR10 is unconditional; the same model is used for each dataset to compare generative processes directly.
  • D.1 DATASETS AND ARCHITECTURES: The diffusion hyperparameters α follow Ho et al. (2020), enabling direct comparison across generative processes.For CIFAR10, Bedroom, and Church, pretrained DDPM checkpoints are used.
  • D.1 DATASETS AND ARCHITECTURES: The model θ(x_t) is a U-Net based on a Wide ResNet; CelebA uses a separately trained five-resolution model from 64 × 64 to 4 × 4.The CelebA model uses the original CelebA dataset rather than CelebA-HQ.
  • D.2 REVERSE PROCESS SUB-SEQUENCE SELECTION: Quadratic selection is used for CIFAR10 and linear selection for the remaining datasets, with each chosen because it achieves slightly better FID than its alternative.These choices are made for the respective datasets.
  • D.3 CLOSED FORM EQUATIONS FOR EACH SAMPLING STEP: The appendix derives each sampling update from the general sampling equation in Eq. (12).The derivation provides a closed-form update for each selected reverse-process timestep.
  • D.3 CLOSED FORM EQUATIONS FOR EACH SAMPLING STEP: The DDPM larger-variance case uses a distinct update equation with ˆσ, changing the coefficient of ϵ while retaining the non-stochastic coefficients.This contrasts with the update for η = 1.
  • D.3 CLOSED FORM EQUATIONS FOR EACH SAMPLING STEP: The larger-variance update is more stochastic than η = 1, explaining its worse performance when dim(τ) is small.Figure 7 compares CIFAR10 samples from 1000-step DDPM, 1000-step DDIM, and 100-step DDIM.

D.4 SAMPLES AND CONSISTENCY · D.5 INTERPOLATION

The paper reports additional DDIM samples and consistency results across CIFAR10, CelebA, and Church. It also constructs image interpolations by applying spherical linear interpolation to Gaussian latents, including line and grid layouts across CelebA, Bedroom, and Church.

  • D.4 SAMPLES AND CONSISTENCY: Additional samples are shown for CIFAR10, CelebA, and Church, alongside CelebA consistency results for DDIM.These results are presented in Figures 7–10.
  • D.5 INTERPOLATION: Line interpolations sample two Gaussian xT values, combine them with spherical linear interpolation, and decode the interpolated latents with DDIM.The resulting values produce DDIM x0 samples.
  • D.5 INTERPOLATION: Grid interpolations sample four latent variables, pair them, apply slerp within pairs under a shared α, and interpolate across the resulting pair samples independently.This procedure generates grid interpolation results.
  • D.5 INTERPOLATION: Additional grid interpolations are shown for CelebA, Bedroom, and Church DDIM models with dim(τ) = 50.These results appear in Figures 11–13.
  • D.4 SAMPLES AND CONSISTENCY: CelebA comparisons include 1000 step DDPM, 1000 step DDIM, and 100 step DDIM samples.Figure 8 compares these three sampling configurations.
  • D.4 SAMPLES AND CONSISTENCY: CelebA consistency is evaluated with the same random xT while varying the number of DDIM steps.Figure 9 visualizes samples under this controlled comparison.
  • D.4 SAMPLES AND CONSISTENCY: Church samples are compared between 100 step DDPM and 100 step DDIM.The comparison appears in Figure 10.
Loading 2010.02502v4…