PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Sampling of Diffusion Models with Exponential Integrator

Qinsheng Zhang, Yongxin Chen

arXiv:2204.13902v4cs.LG

TL;DR

Diffusion models generate high-quality samples but typically require hundreds to thousands of discretization steps, motivating faster sampling. The paper analyzes sampling errors and proposes DEIS, an exponential-integrator method that exploits the learned process's semilinear structure. DEIS achieves high-fidelity sampling with as few as 10 NFEs, including 4.17 FID on CIFAR10 at 10 NFEs.

  • Problem

    Diffusion models have high sample quality but notoriously slow sampling because accurate generation normally requires hundreds to thousands of discretization steps.

  • Method

    DEIS discretizes the learned backward diffusion with an exponential integrator that leverages its semilinear structure and uses higher-order approximations to reduce discretization error.

  • Results

    4.17 FID is achieved with 10 NFEs on CIFAR10 using a pretrained model, while DEIS generates high-fidelity samples with fewer than 10 NFEs.

  • Takeaways & Limitations

    DEIS provides a fast sampler applicable to diffusion models and can also accelerate data log-likelihood evaluation.

  • Takeaways & Limitations

    The exponential-integrator update is exact over an interval only when the nonlinear score term remains constant there.

Abstract

from arXiv · show

The past few years have witnessed the great success of Diffusion models~(DMs) in generating high-fidelity samples in generative modeling tasks. A major limitation of the DM is its notoriously slow sampling procedure which normally requires hundreds to thousands of time discretization steps of the learned diffusion process to reach the desired accuracy. Our goal is to develop a fast sampling method for DMs with a much less number of steps while retaining high sample quality. To this end, we systematically analyze the sampling procedure in DMs and identify key factors that affect the sample quality, among which the method of discretization is most crucial. By carefully examining the learned diffusion process, we propose Diffusion Exponential Integrator Sampler~(DEIS). It is based on the Exponential Integrator designed for discretizing ordinary differential equations (ODEs) and leverages a semilinear structure of the learned diffusion process to reduce the discretization error. The proposed method can be applied to any DMs and can generate high-fidelity samples in as few as 10 steps. In our experiments, it takes about 3 minutes on one A6000 GPU to generate $50k$ images from CIFAR10. Moreover, by directly using pre-trained DMs, we achieve the state-of-art sampling performance when the number of score function evaluation~(NFE) is limited, e.g., 4.17 FID with 10 NFEs, 3.37 FID, and 9.74 IS with only 15 NFEs on CIFAR10. Code is available at https://github.com/qsh-zh/deis

1 INTRODUCTION

Diffusion models offer strong, stable generative performance but are slow to sample. This work attributes fast-sampling quality chiefly to discretization and proposes DEIS, an exponential-integrator sampler that achieves strong quality with few NFEs.

  • Diffusion models provide scalable generative modeling with stable training and relatively low hyperparameter sensitivity.
  • 1000 steps are required for DDPM to generate one sample, with one neural-network evaluation per step.
  • Different discretization schemes can produce dramatically different errors even with the same trained model.
  • DEIS uses exponential integration of the semilinear backward diffusion and supports higher-order approximations of its nonlinear term.
  • 4.17 FID is achieved with 10 NFEs, and 2.86 FID with 20 NFEs on CIFAR10 using a pretrained model.

2 BACKGROUND ON DIFFUSION MODELS

Diffusion models reverse a fixed noising process with a learned denoising process. Their score network approximates the time-dependent score needed to simulate this reverse process.

  • A diffusion model combines a fixed forward noising process with a learned backward denoising process.
  • The forward diffusion is a linear SDE whose coefficients are chosen so conditional marginals are simple Gaussian distributions.
  • VPSDE and VESDE are two popular SDEs, with α_t decreasing and σ_t increasing over time.
  • The reverse-time diffusion matches the forward process in probability law when initialized from its terminal distribution.
  • A time-dependent score network sθ(x, t) is trained to approximate the unavailable score ∇log p_t(x).
  • Denoising score matching uses the closed-form conditional Gaussian score and Monte Carlo samples for stochastic optimization.

3 FAST SAMPLING WITH LEARNED SCORE MODELS

The paper analyzes learned-score diffusion sampling through fitting and discretization errors, then develops DEIS by exploiting the semilinear structure of the probability-flow ODE. The resulting discretization strategy uses exponential integration, score parameterization, and polynomial extrapolation to improve sampling with few NFEs.

  • Learned score models: The learned backward diffusion can be represented by a family of marginal-equivalent SDEs or ODEs, including the probability-flow ODE at λ = 0.With a perfect score model, the induced marginal distribution matches that of the forward diffusion.
  • Error sources: Sampling error comprises fitting error from score mismatch and discretization error from numerically solving the learned backward process.Reducing discretization error normally requires smaller steps and therefore more sampling steps.
  • Fitting error: Score models can have large fitting errors in low-probability regions because training data sampled from the marginal distribution are sparse there.This makes aggressive time steps risky because trajectories may enter regions where the learned score is inaccurate.
  • Discretization error: The probability-flow ODE has a semilinear form with a linear drift and nonlinear score term, motivating an exponential integrator for large-step discretization.The method targets a small number of neural-network evaluations, since those evaluations dominate sampling cost.
  • Discretization error: The basic exponential integrator can underperform Euler when the nonlinear score term changes rapidly along the solution, especially as t approaches 0.This violates the integrator’s assumption that the nonlinear term remains nearly constant over each interval.
  • DEIS design: Using the ϵθ parameterization reduces score approximation error, allowing the exponential integrator to outperform Euler and motivating the DEIS algorithms.Polynomial extrapolation of prior ϵθ evaluations further reduces approximation error, with higher-order polynomials producing better samples when N is small.
  • Relation to DDIM: The resulting discretization provides an alternative numerical explanation for DDIM’s efficacy and extends the approach beyond VPSDE to arbitrary diffusion SDEs.The paper identifies deterministic DDIM as a special case of DEIS.

4 EXPONENTIAL INTEGRATOR: SIMPLIFY PROBABILITY FLOW ODE

DEIS transforms the probability-flow ODE to remove stiffness or discretizes its semilinear structure directly, enabling efficient use of numerical solvers. Its variants inherit convergence guarantees from the underlying RK or Adams–Bashforth methods.

  • Scope: For VPSDE, the transformed treatment can be extended to other scalar diffusion models such as VESDE.The analytical forms of Ψ, G_t, and L_t are used in the VPSDE treatment.
  • ODE transformation: DEIS transforms the probability ODE into a non-stiff black-box ODE that generic solvers can solve efficiently.The transformation removes stiffness caused by the semilinear structure.
  • DEIS variants: The transformed formulation supports ρRK-DEIS and ρAB-DEIS using classical Runge–Kutta and Adams–Bashforth methods.tAB-DEIS instead fits polynomials in t, whereas ρAB-DEIS fits them in ρ.
  • Convergence: DEIS variants retain the convergence-order guarantees of their underlying RK or Adams–Bashforth solvers.This guarantee follows from solving the simplified ODE formulations.

5 EXPERIMENTS

Experiments evaluate DEIS variants and compare them with established samplers across image-generation settings. DEIS generally achieves high-fidelity sampling at small NFE, while higher-order ρRK variants become more competitive as NFE increases.

  • Ablation study: DEIS ingredients significantly improve sampling efficiency on CIFAR10 and outperform standard samplers by a large margin.The ablation study is reported in Figure 5.
  • DEIS variants: Almost all DEIS algorithms generate high-fidelity CIFAR10 images with small NFE, and high-order polynomial approximations can significantly outperform DDIM.DDIM corresponds to the zero-order polynomial approximation.
  • DEIS variants: High-order ρRK-DEIS performs less satisfactorily at small NFE but becomes competitive as NFE increases.The reported explanation is that high-order methods benefit more from small step sizes.
  • Comparisons: Comparisons cover DDPM, DDIM, PNDM, A-DDIM, FastDPM, Ito-Taylor, and an Improved PNDM variant.The experiments use CelebA, class-conditioned ImageNet, and LSUN Bedroom datasets with pretrained models.
  • Qualitative and quantitative evaluation: Figure 6 shows generated DDIM and DEIS samples from an unconditional 256×256 ImageNet pretrained model.Figure 7 reports FID for different sampling algorithms using pretrained diffusion models.
  • Evaluation protocol: Table 2 reports VPSDE results on CIFAR10 under limited NFE, using each algorithm’s best time discretization for fair comparison.For ρRK-DEIS, upper-right numbers indicate extra NFEs, and bold numbers mark the best performance at similar NFE budgets.

6 CONCLUSION

DEIS uses exponential-integrator discretization to accelerate diffusion-model sampling and can also cover likelihood evaluation. The paper relates DEIS to DDIM, Karras et al.’s method, and DPM-Solver while reporting practical solver and scheduling trade-offs.

  • Conclusion: DEIS is a fast sampler based on a novel discretization scheme for backward diffusion and can generate high-fidelity samples with less than 10 NFEs.The authors note that better extrapolation may further improve sampling quality.
  • Likelihood evaluation: DEIS can accelerate likelihood evaluation; RK-based NLL evaluation reaches 3.16 bits/dim with 36 NFE versus 3.15 bits/dim for RK45, approximately 4 times faster.The authors state that this follows because DEIS changes only numerical discretization.
  • Practical limitations: Adaptive solvers can be combined with ρRK-DEIS, but rejected steps may waste the NFE budget; one RK45 rejection costs 5 NFE.The authors report that tuned fixed step sizes work efficiently because trajectories have similar curvature patterns.
  • Relation to Karras et al. (2022): Karras et al. (2022, Algorithm 1) is a special case of ρRK-DEIS obtained with a second-order Heun method.The equivalence follows from the rescaled diffusion formulation and the corresponding simplified ODE.
  • Relation to DPM-Solver: DPM-Solver and DEIS both start from the exact ODE solution but use different discretizations for the nonlinear score component.The comparison describes their midpoint updates as differing in the chosen midpoint.
  • Empirical comparisons: At low NFE, tAB-DEIS generally performs better than other approaches under their best time schedules.With relatively large NFE, third-order Kutta is reported as better than second-order Heun.

C DISCRETIZATION ERROR OF SDE SAMPLING

For SDE sampling, the exponential-integrator formulation computes linear and noise terms exactly, leaving the nonlinear score approximation as the main discretization-error source. The paper therefore finds ODE sampling preferable and does not pursue general SDE discretization further.

  • SDE formulation: The exact SDE solution separates linear, nonlinear, and noise contributions, and the discretization goal is to approximate this solution numerically.The stochastic DDIM is identified as a discretization scheme for the VPSDE case.
  • Open limitation: Polynomial approximation of the score term for general SDE discretization did not work well in practice, so the paper leaves this direction for future work.The authors cite larger nonlinear weights and additional backward-simulation noise as possible reasons.
  • Error sources: Only the nonlinear score term induces discretization error in the exponential-integrator method because the linear and noise terms are computed exactly.For λ > 0, the nonlinear term receives a larger weight than in the probability-flow ODE.
  • Comparison with ODE sampling: The probability-flow ODE with λ = 0 minimizes the nonlinear-term weight, consistent with deterministic DDIM outperforming stochastic DDIM.The analysis also states that VPSDE has a smaller nonlinear weight than VESDE and performs better when N is small.

D PROOF OF PROP 1

The paper proves that the family of SDEs parameterized by λ has λ-independent marginal distributions when the score model is perfect. This establishes equivalence with the reference diffusion’s marginals.

  • Proposition proof: The marginal distribution induced by the λ-parameterized SDE does not depend on λ when the score model is perfect.It equals the marginal distribution induced by the reference SDE.
  • Proof mechanism: The Fokker–Planck–Kolmogorov equation for the induced distribution contains no λ, which establishes marginal-distribution independence from λ.The SDE is considered through its induced partial differential equation.

E PROOF OF PROP 2

The supplied passage contains only a reference to the VPSDE transition function Ψ(t, s), without enough surrounding material to summarize the proof.

  • Proof setup: For the VPSDE, the transition function Ψ(t, s) is obtained by a straightforward calculation from Eq. (6).The passage does not provide the resulting expression.

F PROOF OF PROP 3

The section analyzes numerical errors and discretization choices in diffusion-model sampling, then evaluates DEIS and related solvers across datasets and NFE budgets. It finds that time scheduling, parameterization, and solver structure materially affect quality, especially when NFEs are limited.

  • Time discretization: Quadratic timesteps allocate smaller steps near t=0, where uniform discretization produces larger approximation error.This scheduling follows linspace(t0, √tN, N + 1)^2.
  • Error analysis: The numerical-solution difference is a weighted sum of score-error terms, including accumulation from past errors.A useful discretization must balance local approximation error against accumulated error.
  • Solver design: DEIS uses fixed timestamps because rejected adaptive steps are costly at 10 or 5 NFE, while different samplers prefer different schedules.With exponential integration, the additional ingredients consistently improve FID; tAB-DEIS is advantageous at small NFE, whereas ρRK-DEIS is competitive at larger NFE.

H.8 MORE RESULTS ON VPSDE

Additional VPSDE experiments examine timestep choices, implementation details, and broader ablations. They report that DEIS improves sampling quality across settings, while timestep and endpoint choices substantially influence FID.

  • VPSDE experiments: The experiments compare DEIS with t0 = 10^-4 and with timestep scheduling suggested by Karras et al. (2022).The provided materials identify these as separate VPSDE CIFAR10 evaluations.
  • VESDE results: DEIS significantly accelerates VESDE sampling relative to previous VESDE methods, although it does not match the acceleration achieved for VPSDE.The VESDE evidence concerns CIFAR10 FID results and sampled images.

I MORE RESULTS FOR IMAGE GENERATION

The section reports solver comparisons on CIFAR10, ImageNet32, and CelebA, alongside qualitative samples and VESDE results. DEIS is consistently competitive or superior at limited NFE, while black-box solvers become more viable at larger budgets.

  • Solver comparisons: On the Bao et al. checkpoint, plugin-and-play iPNDM is comparable or slightly better than A-DDIM at small NFE, while DEIS is better than both.This comparison uses the provided checkpoint and time scheduling without ad-hoc tricks.
Loading 2204.13902v4…