PaperlayerView exact revision on arXiv

Source-linked AI summary

Pseudo Numerical Methods for Diffusion Models on Manifolds

Luping Liu, Yi Ren, Zhijie Lin, Zhou Zhao

arXiv:2202.09778v2cs.CVcs.LGmath.NAstat.ML

TL;DR

DDPMs produce high-quality samples but require hundreds to thousands of denoising iterations, motivating faster inference without sacrificing quality. PNDMs recast diffusion as differential equations on data manifolds and achieve faster, higher-quality sampling across several datasets and schedules.

  • Problem

    DDPMs require hundreds to thousands of iterations for high-quality samples, while existing acceleration methods can degrade quality or introduce noise at high speedup rates.

  • Method

    PNDMs solve diffusion-model differential equations on data manifolds using nonlinear transfer parts, with DDIMs treated as simple pseudo numerical methods.

  • Results

    20x faster sampling without loss of quality is reported on Cifar10 and CelebA, while 250-step sampling improves FID by around 0.4 points and reaches 2.71 FID on CelebA.

  • Takeaways & Limitations

    PNDMs generate high-quality images with fewer steps and work with different variance schedules, supporting further denoising acceleration.

  • Takeaways & Limitations

    Classical numerical methods can introduce noticeable noise at high speedup rates, and explicit manifold expressions are difficult because the target x0 and random noise are unavailable in reverse sampling.

Abstract

from arXiv · show

Denoising Diffusion Probabilistic Models (DDPMs) can generate high-quality samples such as image and audio samples. However, DDPMs require hundreds to thousands of iterations to produce final samples. Several prior works have successfully accelerated DDPMs through adjusting the variance schedule (e.g., Improved Denoising Diffusion Probabilistic Models) or the denoising equation (e.g., Denoising Diffusion Implicit Models (DDIMs)). However, these acceleration methods cannot maintain the quality of samples and even introduce new noise at a high speedup rate, which limit their practicability. To accelerate the inference process while keeping the sample quality, we provide a fresh perspective that DDPMs should be treated as solving differential equations on manifolds. Under such a perspective, we propose pseudo numerical methods for diffusion models (PNDMs). Specifically, we figure out how to solve differential equations on manifolds and show that DDIMs are simple cases of pseudo numerical methods. We change several classical numerical methods to corresponding pseudo numerical methods and find that the pseudo linear multi-step method is the best in most situations. According to our experiments, by directly using pre-trained models on Cifar10, CelebA and LSUN, PNDMs can generate higher quality synthetic images with only 50 steps compared with 1000-step DDIMs (20x speedup), significantly outperform DDIMs with 250 steps (by around 0.4 in FID) and have good generalization on different variance schedules. Our implementation is available at https://github.com/luping-liu/PNDM.

1 INTRODUCTION

DDPMs generate high-quality samples but are slow because they require hundreds to thousands of iterative denoising steps. PNDMs treat diffusion as differential equations on manifolds and accelerate sampling while preserving or improving quality.

  • Motivation: DDPMs generate samples through iterative denoising but require hundreds to thousands of network evaluations for high-quality output.The process starts from white noise and progressively denoises it according to model-predicted noise.
  • Prior work: Prior acceleration methods adjust variance schedules or inference equations, including DDIMs, but can degrade quality or introduce noise at high speedup rates.DDIMs accelerate denoising by taking multiple steps per iteration.
  • Approach: PNDMs solve diffusion-model differential equations along a manifold representing the high-density region of the data.The method establishes a theoretical connection between DDPMs and numerical methods and treats DDIMs as simple pseudo numerical methods.
  • Approach: The pseudo linear multi-step method is reported as the fastest method under similar generated quality in most situations.The paper compares classical numerical methods after analyzing why they can sample away from the data manifold.
  • Results: Around 0.4 FID points of improvement is reported on Cifar10 and CelebA at 250 steps, alongside a CelebA FID score of 2.71 and generalization across variance schedules.The implementation is available from the project repository.

2 BACKGROUND

The background presents DDPMs as iterative denoising processes and as stochastic or ordinary differential equations. It then introduces classical numerical methods used to accelerate the reverse process.

  • DDPMs: DDPMs model a data distribution by reversing a multi-step noising process through iterative denoising.The process maps a Gaussian distribution toward an image distribution.
  • DDPMs: The variance schedule controls the speed of noise addition, while N denotes the total number of denoising steps and µθ and βθ are neural networks.The network parameters are denoted by θ.
  • DDPMs: The denoising objective trains ϵθ to estimate Gaussian noise, with µθ expressed through the current sample, schedule terms, and predicted noise.The passage states that ϵθ estimates ϵ and gives the relationship µθ = 1/√αt (xt − βt/√(1−¯αt) ϵθ).
  • Differential-equation view: DDPMs can also be viewed through variance-preserving stochastic differential equations whose continuous-time limit admits an ODE with the same marginal probability density.Probability Flows use the ODE formulation, while VP-SDEs correspond to the stochastic formulation.
  • Differential-equation view: Removing the random term yields a denoising equation that supports numerical acceleration; DDIMs and Probability Flows use this deterministic perspective.The passage characterizes PFs as an acceleration of DDIMs and VP-SDEs as an acceleration of DDPMs.
  • Classical numerical methods: Forward Euler, Runge-Kutta, and linear multi-step methods are introduced as classical numerical approaches for solving ordinary differential equations.Forward Euler uses the current derivative, while Runge-Kutta uses more information per step to achieve higher accuracy.

3 PSEUDO NUMERICAL METHOD FOR DDPM

The paper recasts diffusion denoising as solving differential equations on data manifolds, then constructs pseudo numerical methods whose transfer steps remain close to those manifolds. PNDMs combine this transfer design with higher-order gradient methods, while treating DDIMs as a simple special case.

  • 3.2 Classical Numerical Method: Classical numerical methods can introduce noise because they leave the data’s high-density region and encounter an unbounded equation near the final steps.Most data lie in a narrow band, while classical methods follow straight-line updates; the diffusion equation often tends to infinity as t approaches zero.
  • 3.3 Pseudo Numerical Method on Manifold: PNDMs solve diffusion differential equations on the high-density data manifold by replacing classical linear transfer steps with nonlinear manifold-compatible transfers.The method separates numerical updates into gradient and transfer parts, retaining the model’s noise prediction as the gradient component.
  • 3.3 Pseudo Numerical Method on Manifold: If the predicted noise is precise, the proposed transfer produces a precise previous sample, keeping denoising results on the target manifold.This property is formally stated as Property 3.1 and motivates using the diffusion model prediction as the gradient part.
  • 3.3 Pseudo Numerical Method on Manifold: DDIMs are a simple case of pseudo numerical methods because their inference equation uses the same proposed transfer with the model’s predicted noise.The paper denotes this interpretation as DDIMs* to emphasize DDIM’s status as a pseudo numerical method.
  • 3.4 Gradient Part: PNDMs combine the nonlinear transfer part with the gradient part of a linear multi-step method, combining DDIM-like transfers with higher-order numerical gradients.The paper also presents corresponding pseudo versions of linear multi-step and Runge-Kutta methods, with the linear multi-step method used as the main approach.
  • 3.5 Algorithm: S/F-PNDMs have third-order local error and are second-order convergent, while the proposed linear multi-step construction reuses four previous model evaluations and computes the model once per step.The paper reports that this design combines the convergence benefits of higher-order methods with the computational efficiency needed for accelerated denoising.

4 EXPERIMENT

Experiments on Cifar10, CelebA, and LSUN evaluate PNDMs against DDIMs and classical numerical methods across computation budgets, step counts, variance schedules, and visual trajectories. PNDMs achieve strong sample efficiency and quality while avoiding the off-manifold behavior observed for classical methods.

  • Experimental setup: PNDMs were evaluated on Cifar10, CelebA, LSUN-church, and LSUN-bedroom using pretrained models and multiple numerical-method baselines.The evaluation measures FID across different step counts, numerical methods, and variance schedules.
  • Efficiency: 20x faster: F-PNDM reaches lower FID than 1000-step DDIM using only 50 steps without losing quality.This comparison uses the Cifar10 and CelebA linear-schedule results.
  • Efficiency: S-PNDM initially achieves the best FID, but F-PNDM becomes best after its higher-cost first three pseudo Runge-Kutta steps.The computation-cost/FID comparison is shown in Figure 3 using one-step DDIM cost as the time unit.
  • Quality: F-PNDM improves the best FID by around 0.4 and reaches a 2.71 FID on CelebA, while its FID converges after more than 250 steps.The reported convergence behavior is more pronounced on LSUN.
  • Generalization: PNDMs retain similar acceleration and sampling-quality conclusions across datasets and image sizes, and they work with both linear and cosine variance schedules.Cosine schedules lower FID at relatively large step counts, with limited effect when the number of steps is small.
  • Sample on manifolds: FON trajectories can leave the data’s high-density region, whereas PNDM trajectories avoid this behavior and fit the target result more appropriately.The visualization uses norm distributions and two-pixel generation curves to compare trajectories.

5 DISCUSSION

The discussion presents PNDMs as numerical methods for DDPM-associated ODEs that reduce sampling steps without sacrificing image quality, while identifying several directions for further improvement.

  • Contribution: PNDMs solve the corresponding DDPM ODEs and generate high-quality images with fewer steps without loss of quality.The paper identifies variance schedules, higher-order pseudo methods, and extensions beyond the current setting as future directions.
  • Future directions: Future work includes finding better variance schedules, developing higher-order convergent pseudo numerical methods, and extending PNDMs.The paper states that S-PNDM and F-PNDM are both second-order convergent, while higher-order transfer parts remain an open direction.

A.1 RELATED WORK

Prior work accelerated DDPMs through inference equations, variance schedules, and differential-equation formulations, while also connecting DDPMs with NCSNs and neural differential equations. PNDMs combine ideas from DDIMs and numerical methods for neural differential equations and apply them to DDPM acceleration.

  • DDPM acceleration and quality: DDPM research has targeted faster inference, better variance schedules, improved quality, and combinations with variational autoencoders.The cited works include DDIMs, variance-schedule methods, and hybrid DDPM–VAE approaches.
  • Connections to NCSNs: DDPMs and NCSNs share a Langevin-dynamics-like sampling process, allowing some NCSN improvements to transfer to DDPMs.This connection motivates applying methods developed for score-based models to diffusion models.
  • Neural differential equations: Probability-flow work places DDPMs and NCSNs within neural differential equations, enabling numerical methods from that field to accelerate DDPMs.The paper positions numerical differential-equation methods as a route to diffusion-model acceleration.
  • Applications: DDPMs have also been applied to text-to-speech, singing voice, 3D point clouds, text generation, and conditional sample generation.These applications extend beyond unconditional image generation.

A.2 CONVERGENT ORDER OF METHOD

The section distinguishes local from global numerical error and relates convergence order to the number of iterations needed for a target accuracy.

  • Local error measures the error after one step, whereas global error measures the error after many steps.The global error is evaluated at x_{t+Mδ} for finite, sufficiently large M.
  • The global error is one order lower than the local error under a Lipschitz condition on the target ODE.
  • The forward Euler method has local error O(δ^2) and global error O(δ), so it is first-order.
  • The linear multi-step method has local error O(δ^5) and global error O(δ^4), so it is fourth-order.
  • Higher-order methods can converge faster as δ approaches zero, allowing larger iteration intervals and fewer iterations for similar global error.

A.3 PSEUDO SECOND-ORDER METHOD

This section introduces second-order numerical methods and constructs corresponding pseudo methods by combining pseudo improved Euler and pseudo linear multi-step schemes.

  • The improved Euler method is presented as a second-order numerical method using two evaluations, k1 and k2.
  • A second-order linear multi-step method is introduced as another second-order scheme.
  • The paper defines a corresponding pseudo improved Euler method.
  • The paper also defines a pseudo second-order linear multi-step method.
  • Combining the pseudo methods yields S-PNDMs.

A.4 THE EXISTENCE OF A DERIVATIVE

The section identifies a boundary condition for a well-defined derivative and argues that directly treating DDPMs as ODEs is theoretically weak for freely chosen variance schedules.

  • At t=0, a well-defined derivative requires b=0; otherwise a term involving ϵθ can diverge.
  • Because DDPMs can choose variance schedules freely, most schedules need not satisfy this derivative condition.
  • The paper therefore concludes that directly treating DDPMs as ODEs is not proper and has theoretical weakness.

A.5 RELATIONSHIP BETWEEN t, ϵθ AND xt

The section studies how denoising behavior changes as the step index approaches zero, then reports toy-example, visualization, and FID evaluation materials across datasets and schedules.

  • Relationship between t, ϵθ and xt: As the step index tends to zero, the denoising process converges in both the ϵθ domain and the sample/image domain.
  • Relationship between t, ϵθ and xt: The paper interprets this convergence as increasingly precise predicted noise at smaller steps.
  • Relationship between t, ϵθ and xt: Under the stated precise-gradient assumption, the representation of x_{t′} remains precise for all t′≤t.
  • Relationship between t, ϵθ and xt: The analysis generalizes the denoising update as φ(x(t),ϵ,t,δ)=f(x(t),t,δ)+g(t,δ)ϵ(x(t),t), with f and g vanishing at δ=0.
  • Relationship between t, ϵθ and xt: The paper computes local errors and convergence orders for S-PNDMs and states that the corresponding F-PNDM computation is similar.
  • Relationship between t, ϵθ and xt: PNDMs work with both linear and cosine variance schedules, while F-PNDM benefits more from schedule smoothness because it uses four consecutive steps.
Loading 2202.09778v2…