PaperlayerView exact revision on arXiv

Source-linked AI summary

DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, Jun Zhu

arXiv:2211.01095v3cs.LGcs.CV

TL;DR

Guided sampling improves DPM condition-sample alignment but commonly requires 100 to 250 DDIM steps, while prior high-order solvers are unstable and insufficiently tested in this setting. The paper proposes DPM-Solver++, which uses data prediction, thresholding, and a multistep variant to stabilize diffusion ODE solving. It reports high-fidelity guided samples that nearly converge within 15 to 20 steps across pixel-space and latent-space DPMs.

  • Problem

    100 to 250 DDIM steps are generally needed for guided sampling, and the effectiveness of high-order solvers had not been carefully examined under guidance.

  • Method

    DPM-Solver++ solves the guided diffusion ODE with a data-prediction model, uses thresholding, and adds a multistep solver to address instability.

  • Results

    DPM-Solver++ generates high-fidelity guided samples and almost converges within only 15 to 20 steps for pixel-space and latent-space DPMs.

  • Takeaways & Limitations

    DPM-Solver++ provides a training-free fast solver for guided sampling that is applicable across pixel-space and latent-space DPMs.

  • Takeaways & Limitations

    The work mainly considers second-order solvers and leaves higher-order solvers for future study.

Abstract

from arXiv · show

Diffusion probabilistic models (DPMs) have achieved impressive success in high-resolution image synthesis, especially in recent large-scale text-to-image generation applications. An essential technique for improving the sample quality of DPMs is guided sampling, which usually needs a large guidance scale to obtain the best sample quality. The commonly-used fast sampler for guided sampling is DDIM, a first-order diffusion ODE solver that generally needs 100 to 250 steps for high-quality samples. Although recent works propose dedicated high-order solvers and achieve a further speedup for sampling without guidance, their effectiveness for guided sampling has not been well-tested before. In this work, we demonstrate that previous high-order fast samplers suffer from instability issues, and they even become slower than DDIM when the guidance scale grows large. To further speed up guided sampling, we propose DPM-Solver++, a high-order solver for the guided sampling of DPMs. DPM-Solver++ solves the diffusion ODE with the data prediction model and adopts thresholding methods to keep the solution matches training data distribution. We further propose a multistep variant of DPM-Solver++ to address the instability issue by reducing the effective step size. Experiments show that DPM-Solver++ can generate high-quality samples within only 15 to 20 steps for guided sampling by pixel-space and latent-space DPMs.

1 INTRODUCTION

DPMs use guided sampling to improve conditional sample quality, but the commonly used DDIM solver typically requires 100 to 250 neural-network evaluations. DPM-Solver++ addresses instability in guided sampling and achieves high-quality results in substantially fewer steps.

  • DPMs generate data by progressively removing Gaussian noise, with sampling implemented by discretizing diffusion SDEs or ODEs.
  • With 15 function evaluations on ImageNet 256×256 at guidance scale 8.0, DPM-Solver++ outperforms first-order DDIM while other high-order samplers perform worse.
  • 100 to 250 steps are generally needed for DDIM, a first-order diffusion ODE solver used for guided sampling.Guided sampling combines unconditional and guidance models, with a guidance scale controlling the latter.
  • Prior high-order solvers achieved 10 to 20-step sampling without guidance, but their effectiveness for guided sampling had not been carefully examined.The paper reports that these solvers can produce samples worse than DDIM under guidance.
  • DPM-Solver++ solves the diffusion ODE with data prediction, applies dynamic thresholding, and introduces a multistep variant to reduce instability.The method is training-free and uses smaller effective step sizes in the multistep formulation.
  • 15 steps suffice for DPM-Solver++ to generate high-quality guided samples, while broader experiments show near-convergence within 15 to 20 steps across pixel-space and latent-space DPMs.

2 DIFFUSION PROBABILISTIC MODELS

The paper frames DPM sampling as solving diffusion dynamics from noise to data, with noise- and data-prediction parameterizations. Guided sampling modifies the prediction using conditional information and a guidance scale, while fast ODE solvers offer efficiency but do not reliably transfer to guided settings.

  • DPMs progressively add Gaussian noise to data and reverse this process to obtain samples from a simple normal initialization.
  • Noise-prediction models estimate the noise in a noisy sample, whereas data-prediction models estimate the original clean data.The parameterizations are related by xθ(xt, t) := (xt − σtϵθ(xt, t))/αt.
  • Diffusion ODE sampling discretizes an ODE while time changes from T to 0 and is generally faster than other sampling methods.
  • Guided sampling combines conditional information with DPM predictions for applications including text-to-image, image-to-image, and class-to-image generation.
  • A guidance scale s controls conditional guidance, and large s is usually preferred to improve condition-sample alignment.Classifier guidance uses a pretrained classifier, while classifier-free guidance combines conditional and unconditional model outputs.
  • DDIM is a typical guided-sampling solver that generates samples in a few hundred steps, while prior high-order ODE solvers were developed primarily for unconditional sampling.Their efficiency does not transfer reliably to guided sampling.

3 CHALLENGES OF HIGH-ORDER SOLVERS FOR GUIDED SAMPLING

Large guidance scales create two challenges for high-order diffusion ODE solvers: instability and train-test mismatch. These effects can make high-order methods produce lower-quality guided samples than DDIM and generate bounded-data outputs that are saturated or unnatural.

  • At guidance scale s = 8.0 with 15 function evaluations, previous high-order solvers produce lower-quality images than first-order DDIM.Sample quality worsens further as solver order increases.
  • Large guidance scales amplify model outputs and derivatives, narrowing high-order solvers’ convergence range and forcing smaller step sizes.Higher-order derivatives are especially sensitive to this amplification.
  • Large guidance scales push conditional noise predictions away from true noise, causing converged samples to leave the bounded data interval.For image data, the resulting images can become saturated and unnatural.

4 DESIGNING TRAINING-FREE FAST SAMPLERS FOR GUIDED SAMPLING

DPM-Solver++ designs training-free high-order diffusion ODE solvers for guided sampling by using data prediction, thresholding, and multistep updates. The multistep formulation reduces effective step sizes to improve stability and efficiency under limited function evaluations.

  • Data-prediction parameterization: DPM-Solver++ solves the diffusion ODE with the data prediction model and uses thresholding to keep guided samples bounded.The data-prediction parameterization is presented as better suited to maintaining bounded solutions under large guidance scales.
  • Data-prediction parameterization: The exact data-prediction ODE solution computes its linear term analytically and leaves an exponentially weighted integral of the data prediction model.The solver construction approximates this remaining integral using Taylor expansions in log-SNR.
  • Singlestep solver: DPM-Solver++(2S) is a second-order singlestep solver that combines function evaluations at an intermediate time and the previous endpoint.It uses two sequential neural-network evaluations per step, with intermediate values discarded afterward.
  • Scope: The work mainly considers second-order solvers because higher-order solvers may be unsuitable for large guidance scales.Solvers for orders k ≥ 3 are left for future study.
  • Multistep solver: DPM-Solver++(2M) is a second-order multistep solver that reuses previous values to compute the next solution without additional intermediate values.Multistep methods can use more steps under a fixed function-evaluation budget, producing smaller effective step sizes than singlestep methods.
  • Multistep solver: Multistep methods have smaller high-order error terms than singlestep methods in the fixed-budget comparison and are slightly better empirically.The paper attributes this to multistep step sizes being around 1/k of singlestep step sizes for a k-th order method.

5 FAST SOLVERS FOR DIFFUSION SDES

This section extends DPM-Solver++ to diffusion SDEs in the log-SNR domain. It derives exact SDE solutions and discretizes the resulting integrals using noise- or data-prediction parameterizations.

  • SDE formulation: The SDE formulation is rewritten with respect to log-SNR, and corresponding second-order solvers are derived.The notation reparameterizes time-dependent states, noise scales, and model predictions in the log-SNR variable.
  • Exact solution: The exact diffusion-SDE solution is obtained with the variation-of-constants formula from an initial state at an earlier time.The resulting Itô integral can be computed using a Gaussian random variable.
  • Solver construction: SDE solvers are formed by discretizing the integral with either the noise prediction model or the data prediction model.The section defines first-order SDE-DPM-Solver and SDE-DPM-Solver++ variants, plus a second-order multistep form.
  • Solver construction: SDE-DPM-Solver++(2M) uses a previous solution and model output to construct a second-order multistep update.Its update assumes access to a prior state at an earlier time and uses the data-prediction parameterization.

6 RELATIONSHIP WITH OTHER FAST SAMPLING METHODS

DPM-Solver++ is positioned as a high-order generalization of DDIM using the data prediction model, alongside comparisons with other diffusion samplers and exponential-integrator solvers.

  • Related samplers: Training-free DPM samplers can be understood as discretizations of diffusion SDEs or diffusion ODEs.The paper places DPM-Solver++ in the diffusion-ODE family.
  • Exponential-integrator comparison: Table 1 compares DEIS, DPM-Solver, and DPM-Solver++ as high-order diffusion ODE solvers based on exponential integrators.The table is used to distinguish the proposed solver from previous exponential-integrator methods.
  • Exponential-integrator comparison: DPM-Solver++ is the high-order generalization of DDIM with η = 0 for the data prediction model xθ.The paper emphasizes that the high-order versions differ despite equivalent first-order versions.
  • Stochastic DDIM comparison: SDE-DPM-Solver++(2M) is a second-order generalized version of first-order stochastic DDIM.The paper states that stochastic DDIM is equivalent to SDE-DPM-Solver++1.
  • Related samplers: SDE-based samplers generally need more steps to converge than ODE-based samplers because SDE randomness makes denoising more difficult.The comparison distinguishes solver families by their treatment of stochasticity during sampling.

7 EXPERIMENTS

Experiments compare DPM-Solver++ with tuned baseline samplers across pixel-space and latent-space guided sampling. The method provides stable, faster convergence, with performance depending on guidance scale, parameterization, multistep design, and thresholding.

  • Experimental setup: DPM-Solver++ is evaluated against DPM-Solver, DEIS, PNDM, and DDIM using varying numbers of function evaluations for pixel-space and latent-space guided sampling.NFE counts model calls, and the experiments use continuous-time versions of the discrete-time DPMs.
  • Experimental setup: Uniform t is selected as the best time-step schedule across guidance scales and is used for all solver evaluations.The study compares uniform t, uniform λ, and power-function schedules.
  • Baseline tuning: For large guidance scales, second-order configurations are the best choices among the previous solvers, so each baseline is reported at its best result per NFE.The compared configurations include DPM-Solver-2 and -3, and DEIS-1, -2, and -3.
  • Pixel-space DPMs with guidance: Changing from noise prediction to data prediction makes DPM-Solver++(2S) stably faster than first-order DDIM under guidance.The ablation supports using high-order solvers with the data prediction model xθ rather than the noise prediction model ϵθ.
  • Pixel-space DPMs with guidance: Thresholding improves sample quality at fixed NFE, and thresholded DPM-Solver++ remains faster than first-order DDIM.Thresholding changes the data prediction model and therefore the converged diffusion-ODE solution.
  • Latent-space DPMs with guidance: For latent-space DPMs, all solvers reach FID around 15.0 to 16.0 within 10 steps, while DPM-Solver++ matches the default 50-step PNDM convergence error in 15 to 20 steps.The latent-space evaluation uses stable-diffusion with guidance scale s = 7.5 and no thresholding because latent codes are unbounded.

8 CONCLUSIONS

The paper addresses accelerated guided sampling by introducing DPM-Solver++, which uses data prediction, thresholding, and singlestep or multistep variants. Experiments show high-fidelity samples and near convergence within 15 to 20 steps for pixel-space and latent-space DPMs.

  • DPM-Solver++ is a training-free diffusion ODE solver based on data prediction models, with thresholding and singlestep and multistep variants for guided sampling.

ETHICS STATEMENT

The ethics statement warns that accelerating guided sampling may amplify harmful uses of diffusion probabilistic models.

  • The proposed solver may accelerate image editing and photorealistic fake-image generation, potentially amplifying malicious applications of DPMs.

REPRODUCIBILITY STATEMENT

The paper describes released-or-planned code, public datasets, detailed implementation settings, and convergence proofs, while documenting assumptions and comparisons underlying the solver analysis.

  • Reproducibility resources: The implementation uses official DPM-Solver code, pretrained checkpoints, and publicly available datasets, with detailed settings in Appendix C and convergence proofs in Appendix A.
  • Convergence assumptions: The convergence analysis assumes continuity and boundedness of the model derivatives, an L-Lipschitz data-prediction map, and hmax = O(1/M).
  • Convergence guarantee: Under these assumptions and sufficiently small hmax, both proposed algorithms are second-order with error O(hmax^2).
  • Solver comparison: DPM-Solver++(2S) differs from DPM-Solver-2 through an additional coefficient e^-rihi < 1 on the second term, reducing the high-order error constant.
  • Experimental implementation: The experiments convert discrete-time DPMs to continuous time, use uniform time steps after schedule ablations, and solve from t = 1 to t = 10^-3.

C.3 EXPERIMENT SETTINGS

The experiments use uniform time-step schedules and evaluate solvers across specified function-evaluation budgets. The latent-space sample illustration uses Stable Diffusion with a fixed prompt.

  • Uniform time-step schedules are used across all experiments.Intermediate steps are converted to uniform t for DPM-Solver and DPM-Solver++ experiments.
  • High-order solvers are evaluated at 10, 15, 20, and 25 function evaluations, while DDIM is additionally evaluated at 50, 100, and 250.Diffusion ODEs are solved from t = 1 to t = 10^-3.
  • The latent-space sample in Fig. 6 uses a fixed castle-and-waterfall prompt attributed to Josef Thoma.The prompt is used for the sampled image comparison in Fig. 6.

D EXPERIMENT DETAILS

The experiment details report sample-quality evaluations and visual comparisons across pixel-space and latent-space diffusion models. The listed evaluations vary function evaluations, with the latent-space setting using guidance scale 7.5.

  • Table 3 measures ImageNet 256×256 sample quality with FID while varying the number of function evaluations.
  • Table 4 measures COCO2014 validation-set sample quality with MSE on a discrete-time latent model while varying function evaluations.
  • The latent-space evaluation uses guidance scale 7.5, the recommended Stable Diffusion setting.
  • Figures 5–7 provide visual sampler comparisons for ImageNet, Stable Diffusion, and DeepFloyd-IF guided sampling.Figure 7 states that SDE-DPM-Solver++(2M) can generate better samples than other samplers.
Loading 2211.01095v3…