Gotta Go Fast When Generating Data with Score-Based Models

Alexia Jolicoeur-Martineau; Ke Li; Rémi Piché-Taillefer; Tal Kachman; Ioannis Mitliagkas

Source-linked AI summary

Gotta Go Fast When Generating Data with Score-Based Models

Alexia Jolicoeur-Martineau, Ke Li, Rémi Piché-Taillefer, Tal Kachman, Ioannis Mitliagkas

arXiv:2105.14080v1May 28, 2021cs.LGcs.CVmath.OCstat.ML

TL;DR

Score-based generative models produce realistic and diverse data but are slow because reverse diffusion requires many score-network evaluations. This paper develops an adaptive-step SDE solver tailored to these models, using two score evaluations and extrapolation. It reports faster generation with better or equal sample quality, while removing step-size and schedule tuning, though evaluation is limited to continuous-time image-generation models.

Problem
Score-based models generate data through a long iterative reverse-diffusion process requiring many expensive score-network evaluations.
Method
The paper designs an adaptive-step SDE solver tailored to score-based generative models, using two score evaluations and extrapolation for efficient error control.
Results
The solver is faster than Euler-Maruyama and produces higher-quality or more diverse samples at equal computational budgets.
Takeaways & Limitations
The approach reduces computational requirements for score-based image generation without step-size or schedule tuning and applies to VE and VP diffusion processes.
Takeaways & Limitations
The approach was tested only on continuous-time image-generation models, so broader data types, architectures, and model variants remain future work.

Abstract

from arXiv · show

Score-based (denoising diffusion) generative models have recently gained a lot of success in generating realistic and diverse data. These approaches define a forward diffusion process for transforming data to noise and generate data by reversing it (thereby going from noise to data). Unfortunately, current score-based models generate data very slowly due to the sheer number of score network evaluations required by numerical SDE solvers. In this work, we aim to accelerate this process by devising a more efficient SDE solver. Existing approaches rely on the Euler-Maruyama (EM) solver, which uses a fixed step size. We found that naively replacing it with other SDE solvers fares poorly - they either result in low-quality samples or become slower than EM. To get around this issue, we carefully devise an SDE solver with adaptive step sizes tailored to score-based generative models piece by piece. Our solver requires only two score function evaluations, rarely rejects samples, and leads to high-quality samples. Our approach generates data 2 to 10 times faster than EM while achieving better or equal sample quality. For high-resolution images, our method leads to significantly higher quality samples than all other methods tested. Our SDE solver has the benefit of requiring no step size tuning.

1 Introduction

Score-based models generate high-quality, diverse data but their iterative reverse-diffusion process is slow. The paper proposes an adaptive-step SDE solver tailored to these models to improve speed without sacrificing sample quality or requiring step-size tuning.

Motivation: Score-based models generate data by reversing a diffusion process, but repeated score-network evaluations make generation undesirably slow.The reverse process starts from Gaussian noise and iteratively reconstructs data.
Limitations of existing methods: Existing acceleration methods often require considerable step-size or schedule tuning and may not work for both VE and VP diffusion processes.Some faster VP schedules are arbitrary, model-dependent, and require careful tuning.
Proposed approach: The paper introduces an SDE solver with adaptive step sizes, designed specifically for score-based generative models.Its design uses an ℓ2 error norm, two score-function evaluations, extrapolation, and an RGB-range-based absolute tolerance.
Limitations of existing methods: Off-the-shelf SDE solvers can diverge, run slower than baseline methods, or produce substantially worse samples in generative modeling.Score-based models involve high-dimensional states and expensive score-function evaluations, making standard solver trade-offs unsuitable.
Contributions: The solver is faster than reverse-diffusion with Langevin dynamics and Euler-Maruyama while producing higher-quality or more diverse samples than EM at equal computational budgets.The contribution claims no step-size or schedule tuning and applicability to VE and VP diffusion processes.

2 Background

Score-based modeling corrupts data into Gaussian noise and learns to reverse that process using a neural-network score estimate. VE and VP are two primary forward-diffusion choices, while reverse diffusion can be solved with methods such as Euler-Maruyama or Predictor-Corrector sampling.

A forward diffusion process gradually corrupts data into approximately Gaussian noise independent of the original sample.The reverse process starts from Gaussian noise and solves a reverse diffusion process to generate data.
The forward SDE is characterized by drift f(x, t), diffusion g(t), and a Wiener process w(t) over t ∈[0, 1].These functions are chosen so the terminal state is approximately Gaussian and independent of the initial data.
The reverse diffusion process requires the score ∇x log pt(x), which is approximated by a neural-network score estimator.The score network is trained with an objective whose minimizer equals the distribution score.
The literature primarily uses Variance Exploding (VE) and Variance Preserving (VP) forward-diffusion processes.For VP, β(t) is linearly scheduled with βmin = 0.1 and βmax = 20, yielding an approximately standard Gaussian terminal distribution.
Euler-Maruyama is a basic reverse-diffusion solver, while Predictor-Corrector sampling adds Langevin dynamics to improve Reverse-Diffusion results.Without the additional Langevin step, Reverse-Diffusion produced poor results in the cited work.

3 Efficient Method for Solving Reverse Diffusion Processes

The method uses adaptive step sizes tailored to reverse diffusion, because generic higher-order or adaptive SDE solvers can be slower or produce lower-quality samples. It combines two low-order integration estimates, error control, and extrapolation to improve efficiency while preserving solver stability.

3.1 General algorithm: Higher-order off-the-shelf SDE solvers were 6 to 8 times slower, while Lamba’s adaptive method was faster but produced lower-quality images.These results motivated restricting the search to low-order methods tailored to score-based generation.
3.1 General algorithm: Adaptive step sizes are introduced to gain speed over fixed-step algorithms while controlling local integration error.Two integration methods provide the error estimate used to adjust step sizes and accept or reject proposals.
3.1.1 Integration method: The stochastic Improved Euler method requires two score function evaluations and reuses the evaluation needed by Euler-Maruyama.This keeps the evaluation cost low while supplying a higher-order estimate for adaptive control.
3.1.3 Tolerance: For images, ϵabs is tied to one 8-bit color level: 0.0078 for VP ranges [−1, 1] and 0.0039 for VE ranges [0, 1].The relative tolerance ϵrel controls the speed–precision trade-off: larger values increase speed but reduce precision.
3.1.2 Error estimation: The solver measures error with the ℓ2 norm rather than the ℓ∞ norm to prevent one high-error pixel or channel from shrinking the global step size.The ℓ∞ approach can cause rejection or slowdown in high-dimensional image-space SDEs.
3.2 Algorithm: The resulting algorithm accepts proposals when the scaled error satisfies E2 ≤1, extrapolates the accepted estimate, and updates the next step size dynamically.It applies independently chosen step sizes across batch samples and is straightforward to parallelize across the batch dimension.
3.1.4 Hyperparameters: The default dynamic-step parameters are r = 0.9, θ = 0.9, and hmax equal to the remaining time t.Empirical testing found r ∈[0.5, 1] effective for both VE and VP, with r ∈[0.8, 0.9] slightly faster.

4 Experiments

Experiments compare the adaptive SDE solver with EM, Reverse-Diffusion with Langevin dynamics, Probability Flow, and DDIM across low- and high-resolution image generation. The method provides substantial speedups without apparent quality loss on CIFAR-10 and stronger quality–speed performance on high-resolution images.

4.1 Performance: Experiments evaluate VP, VE, VP-deep, and VE-deep models on 32x32 CIFAR-10, plus VE models on 256x256 LSUN-Church and FFHQ.Performance is measured using FID and IS, with low FID and high IS indicating higher quality and diversity.
4.1 Performance: Compared with EM, the method achieves 2 to 3× speedups and better quality/diversity for high-resolution images at ϵrel = 0.02.On CIFAR-10, it provides > 5× computational speedups at ϵrel ∈{0.02, 0.05} without apparent disadvantage in FID.
4.1 Performance: Reverse-Diffusion with Langevin has the lowest CIFAR-10 FID for VE models but incurs a 4× computational overhead over the proposed method.Its advantage disappears for VP models and high-resolution image generation.
4.1 Performance: At equal computational budgets, the proposed solver is always immensely preferable to EM for high-resolution images and VP models.For VE CIFAR-10, it achieves better FID when NFE is sufficiently large, with the cited threshold at 270 NFE.
4.1 Performance: The VE process is slower to solve than VP because its larger Gaussian noise causes larger local errors.The method uses smaller initial steps for VE and later increases the step size as the noise becomes smaller.
4.2 Solving an ODE instead of an SDE: The proposed SDE solver generally outperforms Probability Flow, especially for high-resolution images where it obtains greatly lower FIDs at similar budgets.Probability Flow produces very poor high-resolution FIDs, suggesting no convergence in that setting.
4.2 Solving an ODE instead of an SDE: DDIM retains lower FID than the proposed solver below 50 NFE, but its performance is poor and its FID worsens substantially as NFE decreases.The increase is nevertheless less dramatic than for EM.

5 Limitations

The evaluation covers continuous-time image-generation models, while the method still requires selecting a relative tolerance that affects solver steps.

Experiments only cover continuous-time image-generation models, leaving other data types, architectures, and learned-variance models for future work.The authors attribute this scope to the availability of suitable pretrained models and suggest extending evaluation to additional settings.
The solver still requires choosing a relative tolerance, which indirectly controls the number of steps and could be tuned for a target metric.This residual hyperparameter choice partially limits the claim that step-size and schedule tuning are unnecessary.
ϵrel = 0.01 is suggested for precise results, while ϵrel = 0.05 is suggested for fast results.The authors report that FID remains stable with respect to ϵrel.

6 Conclusion

The proposed SDE solver generates images at comparable or better quality than Euler-Maruyama while reducing computational requirements and offering a quality–speed trade-off.

2 to 5× lower computational budgets are required while maintaining comparable or better image quality than Euler-Maruyama.The method also provides a way to trade quality for additional speed.
Image generation remains slow, taking a few minutes compared with generative models that use a single forward pass.

7 Broader Impact

Faster score-based generation brings the technology closer to real-time applications, while broader generative-model risks remain relevant.

Faster generation takes score-based generative models closer to real-time applications.
Generative models can be used to maliciously deceive humans, including through deepfakes.
Generative models risk reproducing biases present in existing datasets.

Appendices

Preliminary solver experiments on the VP model found that most off-the-shelf methods were unstable or slower than Euler-Maruyama, while the fastest promising adaptive method produced lower-quality images.

A DifferentialEquations.jl: Table 3 reports short experiments with DifferentialEquations.jl SDE solvers on a VP model using a small mini-batch.
A DifferentialEquations.jl: Most tested solvers either failed to converge with instability warnings or converged substantially slower than Euler-Maruyama.

B Effects of modifying Algorithm 1

This section evaluates modifications to Algorithm 1 across VP- and VE-CIFAR10 settings, focusing on configuration effects and the adaptive solver procedure.

Experimental settings: The VP-CIFAR10 experiments vary settings using Inception Score, FID, and Number of score Function Evaluations (NFE).Table 4 reports results from 10k samples with mini-batches of 1k samples.
Setting effects: Most tested settings lead to better results, while r has little impact on FID.Using r ∈[0.8, 0.9] produces slightly fewer score function evaluations and sometimes lower FID.
Experimental settings: The VE-CIFAR10 experiments likewise evaluate settings using Inception Score, FID, and Number of score Function Evaluations (NFE).Table 5 uses the same 10k-sample and mini-batch structure.
Algorithm: The implementation retains the full trajectory and preserves the noise after rejection to avoid bias in rejected steps.The formulation also accounts for forward-time integration and diffusion dependence on x.
Algorithm: Algorithm 1 solves a forward-time diffusion process with dynamic step-size extrapolation.The algorithm specifies score network and time-bound inputs, an initial step size, and adaptive update parameters.
Algorithm: The procedure proposes an Euler-Maruyama step, computes an improved stochastic Euler step, and accepts or rejects the sample using a normalized error estimate.The error uses an ℓ2 norm with absolute and relative tolerances, followed by extrapolation after acceptance.

D Implementation Details

The implementation details document baseline discrepancies, time discretization, endpoint handling, and the corrected denoising procedure used in the experiments.

Baseline implementation: The reimplemented baselines differ slightly from the original reports: VP and VP-deep EM give 2.55 and 2.49, while VE and VE-deep Reverse-Diffusion with Langevin gives 2.40 and 2.21.The differences arise from changed SDE-solving settings.
Time discretization: For CIFAR-10, the solver uses N = 1000 time steps, while LSUN uses N = 2000.The time sequence starts at t0 = 1 and uses ϵ = 1e-3 for VP models and ϵ = 1e-5 for VE models.
Time discretization: The EM step size is 1/N, differing negligibly from the 1−ϵ/N spacing implied by the time sequence.The text states that this discrepancy has little to no impact.
Endpoint handling: The final predictor step previously moved from t = ϵ to a negative time, and denoising was then performed while assuming t = ϵ.The authors choose instead to stop at t = ϵ and denoise because denoising is helpful.
Denoising: Correct denoising uses the Tweedie formula with transition-kernel variance 0.01 for VE and 1 for VP.The variance is Var[x(t)|x(0)].
Denoising: Correct denoising has little impact on VE but significantly lowers FID for VP.The incorrect and correct procedures differ only slightly for VE and produce a larger change for VP.
Evaluation: Table 6 reports Inception Score on CIFAR-10 at 32x32 using 50K samples.

F Stability and Bias of the Numerical Scheme

The paper analyzes stability and bias for its adaptive SDE scheme under Wiener-process assumptions. It concludes that the added adaptive steps preserve EM stability and unbiasedness in mean and mean square.

Assumptions: The analysis assumes stochastic dynamics driven by a Wiener process and time-symmetric Brownian motion.These assumptions are described as consistent with commonly used literature formulations.
Scheme analysis: Algorithm 1 combines Euler-Maruyama with an adaptive step-size predictor-corrector to obtain speedup while conserving EM stability and convergence.The section introduces stability and bias as the criteria for analyzing the numerical SDE solution.
Definitions: For the linear test SDE, asymptotic unbiasedness is assessed through convergence of the numerical solution toward a zero-mean Gaussian distribution.The text notes that Gaussian solutions are controlled by their first two moments.
Definitions: Mean stability and mean-square stability are defined through conditions on the numerical solution of the linear SDE.These provide the two stability criteria used in the analysis.
Bias analysis: The first Euler-Maruyama step remains unbiased, while the reverse-diffusion analysis uses the time-reversed Wiener process.The algorithm performs consecutive forward and backward steps with t = 2h.
Mean analysis: Under |1 + λh| < 1, the scheme is numerically stable and unbiased with respect to the mean.
Mean-square analysis: Under the same consecutive-step assumption, the numerical scheme is stable and unbiased in the mean square.
Overall conclusion: Because the adaptive step decreases and then remains unchanged, the entire algorithm is stable and unbiased in both the mean and square mean.