PaperlayerView exact revision on arXiv

Source-linked AI summary

DPM-Solver-v3: Improved Diffusion ODE Solver with Empirical Model Statistics

Kaiwen Zheng, Cheng Lu, Jianfei Chen, Jun Zhu

arXiv:2310.13268v3cs.CVcs.LG

TL;DR

DPM sampling is expensive, and existing fast ODE solvers manually choose among limited inference parameterizations. The paper introduces DPM-Solver-v3, which learns empirical model statistics for a lower-error ODE formulation and combines them with multistep predictor-corrector sampling. Experiments report consistent quality improvements across pixel- and latent-space unconditional and conditional settings, especially at 5–10 steps, while noting remaining speed and design limitations.

  • Problem

    Fast DPM samplers rely on manually selected parameterizations, mainly noise or data prediction, whose suitability for sampling is insufficiently studied.

  • Method

    DPM-Solver-v3 introduces empirical model statistics for a new ODE formulation and combines them with multistep predictor-corrector solvers and few-step techniques.

  • Results

    DPM-Solver-v3 consistently improves sample quality from 5 to 20 steps across unconditional and conditional pixel- and latent-space DPMs, with significant gains at 5–10 steps.

  • Takeaways & Limitations

    The solver provides a training-free approach for improving DPM sample quality in few-step regimes and under large guidance scales.

  • Takeaways & Limitations

    DPM-Solver-v3 still lags behind training-based methods, is not fast enough for real-time applications, and does not explore global timestep-schedule design.

Abstract

from arXiv · show

Diffusion probabilistic models (DPMs) have exhibited excellent performance for high-fidelity image generation while suffering from inefficient sampling. Recent works accelerate the sampling procedure by proposing fast ODE solvers that leverage the specific ODE form of DPMs. However, they highly rely on specific parameterization during inference (such as noise/data prediction), which might not be the optimal choice. In this work, we propose a novel formulation towards the optimal parameterization during sampling that minimizes the first-order discretization error of the ODE solution. Based on such formulation, we propose DPM-Solver-v3, a new fast ODE solver for DPMs by introducing several coefficients efficiently computed on the pretrained model, which we call empirical model statistics. We further incorporate multistep methods and a predictor-corrector framework, and propose some techniques for improving sample quality at small numbers of function evaluations (NFE) or large guidance scales. Experiments show that DPM-Solver-v3 achieves consistently better or comparable performance in both unconditional and conditional sampling with both pixel-space and latent-space DPMs, especially in 5$\sim$10 NFEs. We achieve FIDs of 12.21 (5 NFE), 2.51 (10 NFE) on unconditional CIFAR10, and MSE of 0.55 (5 NFE, 7.5 guidance scale) on Stable Diffusion, bringing a speed-up of 15%$\sim$30% compared to previous state-of-the-art training-free methods. Code is available at https://github.com/thu-ml/DPM-Solver-v3.

1 Introduction

DPMs generate high-quality images but require costly repeated model evaluations. DPM-Solver-v3 targets faster training-free sampling through a new ODE formulation and achieves improved quality, especially at very few steps.

  • DPMs typically require hundreds of pretrained-model evaluations to gradually remove noise, creating a major sampling bottleneck.
  • Training-free samplers use diffusion ODE solvers because their deterministic dynamics support high-quality samples in around 20 steps.
  • DPM-Solver-v3 reformulates the diffusion ODE with empirical model statistics and adds arbitrary-order multistep predictor-corrector methods.
  • DPM-Solver-v3 consistently improves sample quality over previous fast samplers from 5 to 20 steps, with particularly strong gains within 10 steps.

2 Background

DPMs diffuse data into Gaussian noise and generate samples by solving a reverse-time diffusion ODE. Existing exponential-integrator samplers reduce errors, but sampling parameterizations remain manually chosen and insufficiently studied.

  • DPMs define a forward process that gradually transforms data into approximately pure Gaussian noise using a smooth noise schedule.
  • Sampling can be performed by solving the diffusion ODE from time T to time 0.
  • Classifier-free guidance combines unconditional and conditional noise predictors using a guidance scale that trades off condition alignment and diversity.
  • Noise, score, data, and v predictors are theoretically equivalent parameterizations, yet they can affect empirical performance.
  • Exponential-integrator solvers exactly compute the diffusion ODE’s linear term, reducing discretization errors at small numbers of evaluations.
  • Existing sampling parameterizations are manually selected and largely limited to noise or data prediction.

3 Method

DPM-Solver-v3 introduces a flexible ODE parameterization whose coefficients are chosen to reduce first-order discretization error, then builds high-order local and global solvers around it. The resulting multistep predictor-corrector scheme has formal convergence guarantees, while pseudo-order corrections target very small NFE regimes.

  • 3.1 Improved Formulation of Exact Solutions of Diffusion ODEs: The method rewrites the diffusion ODE using a generalized parameterization gθ and three coefficient types while preserving an equivalent exact solution.
  • 3.1 Improved Formulation of Exact Solutions of Diffusion ODEs: The coefficients are selected by analyzing first-order discretization error and restricting the nonlinear component’s gradient norm.
  • 3.1 Improved Formulation of Exact Solutions of Diffusion ODEs: Empirical model statistics are coefficients with analytic pretrained-model-based solutions that can be estimated unbiasedly using Monte Carlo samples.
  • 3.2 Method: An (n + 1)-th-order local approximation uses an n-th-order Taylor expansion of gθ and n additional previous function values for derivative estimation.
  • 3.2 Method: Under regularity assumptions, the local truncation error is O(h^(n+2)), giving the local approximation (n + 1)-th-order accuracy.
  • 3.2.2 Global Solver: The global solver repeatedly applies local transitions, caches previous values for multistep updates, and uses predictor-corrector refinement without extra NFE.
  • 3.2.2 Global Solver: Under local and global assumptions, an (n + 1)-th-order predictor has final error O(h^(n+1)).
  • 3.3 Practical Techniques: Pseudo-order correctors no longer guarantee n-th-order accuracy when n > 2, but are used because that guarantee is less important at small NFE.

4 Experiments

Experiments across pixel-space and latent-space diffusion models evaluate unconditional and conditional sampling from 5 to 20 NFEs. DPM-Solver-v3 generally improves sample quality, with especially strong gains at low NFE and under large guidance scales.

  • Experimental setup: Experiments cover unconditional and conditional sampling across pixel-space and latent-space DPMs, using 5–20 NFEs and diverse image resolutions.The evaluation includes unconditional CIFAR10 and LSUN-Bedroom, conditional ImageNet-256 and Stable Diffusion, and resolutions from 32 to 256.
  • Unconditional sampling: DPM-Solver-v3 consistently achieves better FID on unconditional sampling, including FID 12 at 5 NFE on CIFAR10 ScoreSDE and FID 2.51 at 10 NFE with EDM.On LSUN-Bedroom, it reaches FID 3.06 with 12 NFE, approximately 60% of UniPC’s 20-NFE sampling cost.
  • Conditional sampling: DPM-Solver-v3 achieves better sample quality or convergence at most conditional NFEs, including Stable Diffusion below 10 NFE.The EMS and half-corrector together outperform DPM-Solver++ in the Stable Diffusion setting at guidance scale 7.5.
  • Empirical model statistics: The empirical model statistics show l_λ transitioning from data prediction toward noise prediction as sampling proceeds, while s_λ and b_λ are more model-specific and fluctuate.The reported fluctuations of s_λ and b_λ do not cause sampler instability.
  • Visual quality: At small NFE, prior methods show reduced color contrast, whereas DPM-Solver-v3 produces less-biased samples with more visual details.At large guidance scales, correctors in prior methods can distort images, while DPM-Solver-v3 avoids the reported castle-to-hill and translucency artifacts.

5 Conclusion

The paper formulates diffusion ODE sampling around empirical model statistics and introduces DPM-Solver-v3 with high-order multistep predictor-corrector methods. Experiments show improved sample quality for pixel-space and latent-space models, especially at 5–10 steps, while important speed and design limitations remain.

  • Contribution: DPM-Solver-v3 reformulates the diffusion ODE with empirical model statistics to minimize first-order discretization error.The formulation introduces a new parameterization and coefficients computed from the pretrained model.
  • Contribution: The solver combines a multistep predictor-corrector framework with techniques for few-step sampling and large guidance scales.The framework is designed to support high-order solvers for diffusion ODEs.
  • Conclusion: Experiments demonstrate effectiveness for unconditional and conditional sampling with pixel-space and latent-space pretrained DPMs, especially in 5–10 steps.The paper reports consistently better or comparable performance across the evaluated settings.
  • Limitations and broader impact: DPM-Solver-v3 still lags behind training-based methods, is not fast enough for real-time applications, and leaves global design choices such as timestep schedules unexplored.The authors also note potential misuse for accelerating fake and malicious content generation.
  • Background: Diffusion probabilistic models support high-quality image generation and applications including controllable generation, translation, editing, likelihood estimation, compression, and inverse problems.The paper situates fast sampling within a broader diffusion-model application landscape.

A.2 Comparison with Existing Solvers Based on Exponential Integrators

DPM-Solver-v3 generalizes exponential-integrator ODE formulations by introducing coefficients that select a new model parameterization. This framework contains prior formulations as special cases while using EMS to improve first-order discretization accuracy.

  • DPM-Solver-v3 introduces coefficients l_λ, s_λ, and b_λ that define a new predictor g_θ for the diffusion ODE.The formulation is designed to support a broader parameterization family than fixed noise or data prediction.
  • Previous exponential-integrator ODE formulations are special cases obtained by assigning specific values to l_λ, s_λ, and b_λ.
  • Unlike UniPC’s equivalent local substitutions, DPM-Solver-v3 avoids tuning choices such as B_1(h)=h or B_2(h)=e^h−1.The authors state that this avoids the associated tuning effort.
  • Under a Gaussian assumption, DPM-Solver++ is a model-agnostic approximation of DPM-Solver-v3 and approximately computes the linear terms at the initial stage.This provides an explanation for why data-prediction DPM-Solver++ can outperform noise-prediction DPM-Solver there.
  • DPM-Solver-v3’s first-order solver differs from DDIM because it uses EMS-based parameterization, and empirically performs better than DDIM.

B Proofs

The proofs establish local accuracy and global convergence guarantees for the proposed solver under smoothness, boundedness, step-size, Lipschitz, and initialization assumptions. They use Taylor expansions, multistep error analysis, and bounds on errors from inexact previous values.

  • Under the stated assumptions, the proof framework establishes local and global order guarantees for the solver and predictor.
  • The analysis assumes continuous derivatives, bounded and sufficiently smooth coefficients, and regular step sizes to support Taylor expansion and local truncation-error bounds.Assumption B.3 requires δ_k to have the same order as λ_t−λ_s.
  • Global convergence analysis additionally assumes a noise-prediction model that is Lipschitz in x, nearly uniform step sizes, and sufficiently accurate starting values.
  • O(h^(n+2)) is the concluded local approximation error after substituting the derived coefficient and derivative bounds.
  • O(h^(n+1)) bounds the propagated global error term |Δ_M| under the multistep assumptions and initialization accuracy.

D Experiment Details

The experiments evaluate pixel-space and latent-space diffusion models across unconditional, conditional, and text-to-image settings using pretrained checkpoints and EMS estimated from sampled data. Sampling uses high-order predictor-corrector configurations, while runtime remains governed primarily by model evaluations.

  • EMS is estimated from forward-diffusion or model-generated samples, with timestep and sample counts adjusted across datasets and guidance settings.Stable Diffusion uses latents generated at the corresponding guidance scale because its training distribution differs from MS-COCO images.
  • DPM-Solver-v3 uses third-order predictor-corrector sampling by default, with pseudo higher-order variants at 5 NFE in several settings.Configurations vary by dataset, including second-order predictors with pseudo third-order correctors.
  • EMS computation for EDM on CIFAR10 takes approximately 3.5 hours on eight NVIDIA A40 GPUs and uses different N,K settings for NFE below versus at least 10.
  • For ImageNet-256, degenerated l_λ=1 performs better for FID, and EMS computation takes approximately 9.5 hours on eight NVIDIA A40 GPUs.
  • Runtime differs by about ±1% across solvers at matched NFE, because serial evaluations of the neural network dominate computation.

F Quantitative Results

Across the reported datasets and settings, DPM-Solver-v3 achieves consistently better or comparable performance than previous fast samplers, especially at 5–10 NFEs. Ablations show that EMS sample count is more important than timestep count for performance.

  • DPM-Solver-v3 achieves consistently better or comparable quantitative performance across the reported datasets, especially at 5–10 NFEs.The results cover LSUN-Bedroom, ImageNet-256, and MS-COCO2014 evaluations.

G.1 Varying the Number of Timesteps and Datapoints for the EMS

The EMS ablations show that datapoint count is more important than timestep count for performance, while pseudo-order components improve very-low-NFE sampling. Multistep DPM-Solver-v3 is the most stable among compared parameterizations.

  • EMS Configuration: EMS datapoint count K is crucial to performance, whereas timestep count N mainly affects results at 5–10 NFEs.For NFE>10, N can be reduced to 50 with even better FIDs, but EMS computation costs scale with NK.
  • First-Order Solvers: DPM-Solver-v3-1 performs better than DDIM as a first-order solver on ScoreSDE CIFAR10.The comparison uses both quantitative results and random samples at 5, 10, and 20 NFE.
  • Pseudo-Order Solver: At 5 NFE, the pseudo-order predictor improves FID from 15.91 to 12.76 for ScoreSDE and from 12.72 to 12.21 for EDM.The predictor is applied in the 5-NFE CIFAR10 case for maximum performance improvement.
  • Pseudo-Order Solver: Switching to a pseudo-order corrector consistently improves results across the tested settings.The authors recommend pairing an n-th order predictor with a pseudo-(n+1)-th order corrector.
  • Singlestep vs. Multistep: For multistep methods, DPM-Solver-v3 outperforms DPM-Solver++ and DPM-Solver across NFEs and is the most stable overall.For singlestep methods, DPM-Solver can outperform DPM-Solver-v3 when NFE≥10, while DPM-Solver performs worst when NFE<10.

H FID/CLIP Score on Stable-Diffusion

On Stable Diffusion, DPM-Solver-v3 achieves better FID with similar CLIP scores across tested settings. The paper cautions that FID does not directly measure latent-space convergence, for which MSE is more appropriate.

  • Sample Quality and Alignment: DPM-Solver-v3 achieves consistently better FID and similar CLIP scores on Stable Diffusion.The evaluation uses MS-COCO2014 prompts, guidance scale 7.5, and 10,000 samples.
  • Sample Quality and Alignment: 15.4 FID is achieved in 8 NFE, close to the reported FID of Stable-Diffusion v1.4.This result is reported for the Stable-Diffusion evaluation.
  • Parameterization: The generalized parameterization gθ covers a wide range of parameterization families while enabling coefficients optimized for first-order discretization error.This equivalence holds when α(λ)≠0 and α(λ), γ(λ) have first-order derivatives.
  • Solver Design: Higher-order solvers are commonly used because higher-order Taylor expansions reduce sampling errors.Third-order solvers are used for unconditional sampling and second-order solvers for conditional sampling.

I.3 The Extra Error of EMS Estimation and Integral Estimation

EMS estimation introduces a finite-data error that mainly affects first-order discretization error at few steps, while integral estimation contributes high-order, practically negligible error. Precomputed EMS add modest cost and support flexible schedules and common guidance scales.

  • EMS Estimation Error: EMS estimation error affects first-order discretization error and therefore performance at few sampling steps.The ODE solution and convergence order remain valid for bounded estimated coefficients, and 1024 datapoints are robust in most cases.
  • Integral Estimation Error: Integral estimation by the trapezoidal rule contributes extra error of high order and is considered ignorable.The bound follows from bounded EMS and their first-order derivatives under the stated assumption.
  • Practical Computation: Precomputed EMS add O(ND) memory and negligible sampling overhead after one-time preprocessing.On Stable Diffusion, the extra memory is approximately 125M compared with roughly 4G for the pretrained model.
  • Flexibility: EMS computed on unconditional models can be reused across sampling schedules and common guidance scales.They are unsuitable for extremely large guidance scales such as CFG 15, where conditional effects strongly influence denoising.
  • Practical Value: At around 20 NFE, sample-quality gains are small because diffusion-ODE samplers nearly converge, but LSUN-Bedroom reaches up to 40% faster convergence.The speed-up is more evident in less diverse domains.

K Additional Samples

Additional samples compare DPM-Solver-v3 with prior solvers across CIFAR10, EDM, ImageNet-256, and Stable Diffusion settings. The reported visual advantages include less shallow bias, higher saturation, and more detail.

  • Additional Samples: DPM-Solver-v3 visual quality outperforms previous state-of-the-art solvers in the additional samples.The comparisons include ScoreSDE and EDM on CIFAR10, Guided-Diffusion on ImageNet-256, and Stable Diffusion samples at 5 NFE.
  • Additional Samples: The generated images show reduced shallow bias, higher saturation, and more visual details.These qualitative characteristics are reported for the additional visual comparisons.
Loading 2310.13268v3…