PaperlayerView exact revision on arXiv

Source-linked AI summary

Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

Fan Bao, Chongxuan Li, Jun Zhu, Bo Zhang

arXiv:2201.06503v3cs.LG

TL;DR

DPM inference is expensive because it typically requires thousands of timesteps, including estimating reverse-process variance at each step. Analytic-DPM derives score-function-based analytic estimates and, empirically, improves likelihood and sample quality while achieving 20× to 80× speedups.

  • Problem

    DPM inference often requires thousands of timesteps, while estimating the reverse-process variance remains a key challenge for efficient sampling and density evaluation.

  • Method

    Analytic-DPM uses score-function-based analytic forms, Monte Carlo estimation, a pretrained score model, and bounded variance clipping for training-free inference.

  • Results

    Analytic-DPM improves likelihood and sample quality across DPMs, achieving 20× to 80× speedups with shorter inference trajectories.

  • Takeaways & Limitations

    The framework provides plug-and-play, training-free inference that can make pretrained DPMs faster while maintaining or improving reported likelihood and sample-quality results.

  • Takeaways & Limitations

    Analytic-DDPM does not always outperform the baseline under FID, so likelihood improvements do not uniformly translate to better sample-quality scores.

Abstract

from arXiv · show

Diffusion probabilistic models (DPMs) represent a class of powerful generative models. Despite their success, the inference of DPMs is expensive since it generally needs to iterate over thousands of timesteps. A key problem in the inference is to estimate the variance in each timestep of the reverse process. In this work, we present a surprising result that both the optimal reverse variance and the corresponding optimal KL divergence of a DPM have analytic forms w.r.t. its score function. Building upon it, we propose Analytic-DPM, a training-free inference framework that estimates the analytic forms of the variance and KL divergence using the Monte Carlo method and a pretrained score-based model. Further, to correct the potential bias caused by the score-based model, we derive both lower and upper bounds of the optimal variance and clip the estimate for a better result. Empirically, our analytic-DPM improves the log-likelihood of various DPMs, produces high-quality samples, and meanwhile enjoys a 20x to 80x speed up.

1 INTRODUCTION

DPM inference is costly because it often traverses thousands of timesteps, with reverse-process variance estimation remaining a key challenge. Analytic-DPM derives analytic variance and KL forms from the score function and uses them for training-free, faster inference with improved likelihood and sampling quality.

  • Motivation: DPM inference often requires thousands of timesteps, making it two or three orders of magnitude slower than other generative models such as GANs.Estimating the reverse-process variance at each timestep is identified as a key inference problem.
  • Method: Analytic-DPM shows that optimal reverse variance and corresponding optimal KL divergence have analytic forms with respect to the score function.The score function is the gradient of a log density.
  • Method: Analytic-DPM is a training-free, plug-and-play framework that estimates analytic variance and KL forms using Monte Carlo and a pretrained score-based model.It is applicable to a variety of diffusion probabilistic models.
  • Bias correction: Lower and upper bounds on optimal variance are derived to correct score-model bias, and the variance estimate is clipped for improved results.The paper also reports a relationship between the score function and the data covariance matrix.
  • Empirical results: 20× to 40× speed up is achieved while consistently improving the log-likelihood of several diffusion probabilistic models.Analytic-DPM also improves DDIM sample quality and can use up to 50 timesteps for comparable FID to the corresponding full-timestep process.

2 BACKGROUND

DPMs define a noisy forward process and a denoising Markov reverse process, with reverse means learned from score-based models. However, standard training provides no signal for reverse variances, which are typically handcrafted and generally suboptimal.

  • Forward and reverse processes: DPMs first inject noise into q(x_0) through a forward process, then reverse it to recover the data distribution.The forward process is specified by a noise schedule β_n ∈ (0, 1).
  • Forward and reverse processes: The reverse process gradually denoises from p(x_N) = N(x_N|0, I) using a Markov process that approximates q(x_0).The paper focuses on a general non-Markov forward process whose marginals are independent of λ.
  • Score-based parameterization: The reverse mean μ_n(x_n) is generally parameterized by a time-dependent score-based model s_n(x_n).The optimal score-matching solution is s*_n(x_n) = ∇_x_n log q_n(x_n).
  • Training objective: The reverse process is learned by optimizing a variational bound on negative log-likelihood, equivalently a KL divergence between forward and reverse processes.A reweighted variational objective is also used in practice to improve sample quality.
  • Reverse variance: The training objective provides no learning signal for σ_n^2, so prior work generally handcrafts reverse variances, which are not generally optimal and can cause suboptimal performance.This limitation motivates estimating the optimal reverse variance instead of relying on fixed settings.

3 ANALYTIC ESTIMATE OF THE OPTIMAL REVERSE VARIANCE

The section shows that the optimal reverse mean and variance, along with the corresponding KL objective, admit analytic score-function representations. Analytic-DPM estimates the variance from a pretrained score model using Monte Carlo samples and reduces bias by clipping estimates with distribution-independent bounds.

  • Analytic optimal solution: Theorem 1 gives analytic score-function forms for the optimal reverse mean and variance and the corresponding KL objective.The result applies to discrete-time diffusion probabilistic models and can be extended to continuous timesteps.
  • Analytic optimal solution: The analytic optimal mean coincides with the prior parameterization, while replacing the true score with a score-based model enables training-free variance estimation.The estimate requires a pretrained score-based model and does not require an extra training process.
  • Monte Carlo estimation: M=10 or 100 Monte Carlo samples suffice to obtain sufficiently small sampling variance and performance similar to that with a large M.The vector Γ is computed once for a pretrained model and reused in downstream computations.
  • Bias correction: The estimate can be biased because score approximation error is irreducible and its coefficient may grow when sampling with a shorter trajectory.Theorem 2 provides lower and upper bounds, and Analytic-DPM clips the estimate using the minimum applicable bound; prior handcrafted variances can underestimate the optimum.

4 ANALYTIC ESTIMATION OF THE OPTIMAL TRAJECTORY

The section accelerates inference by restricting the diffusion process to a shorter K-timestep trajectory and analytically estimating its optimal reverse-process quantities. Trajectory selection becomes a least-cost-path problem solvable by dynamic programming.

  • Shorter Trajectories: K can be much smaller than N, enabling a shorter forward process along selected timesteps to speed up inference.The trajectory satisfies 1 = τ1 < ··· < τK = N.
  • Optimal Reverse Process: The optimal reverse conditional mean and variance for each selected timestep are derived under KL minimization.The section also derives bounds and an estimate for the optimal reverse variance.
  • Analytic Trajectory Cost: The trajectory objective decomposes into K −1 analytic score-function costs, each evaluated for an adjacent timestep pair.The cost is estimated using the logarithm of the estimated optimal variance, without neural-network computation once Γ is given.
  • Trajectory Optimization: The trajectory optimization reduces to a canonical least-cost-path problem on a directed graph with N nodes and a fixed path length of K nodes.The path starts at 1 and terminates at N.
  • Trajectory Optimization: Dynamic programming solves the least-cost path, and the formulation can also be extended to diffusion models with continuous timesteps.The dynamic-programming algorithm is attributed to Watson et al. (2021).

5 RELATIONSHIP BETWEEN THE SCORE FUNCTION AND THE DATA COVARIANCE MATRIX

The section establishes that the data covariance matrix decomposes into two conditional-covariance terms, with the first represented by the score function. When n is sufficiently large, the second term is negligible because x0 and xn are almost independent, making covariance nearly score-determined.

  • The data covariance matrix decomposes into Eq(xn)Covq(x0|xn)[x0] and Covq(xn)Eq(x0|xn)[x0].
  • The first covariance term can be represented by the score function.
  • When n is sufficiently large, the second term is negligible because x0 and xn are almost independent.
  • Under these conditions, the data covariance matrix is almost determined by the score function.

6 EXPERIMENTS

Experiments evaluate Analytic-DPM against DDPM and DDIM baselines using even and optimal trajectories across multiple datasets and pretrained score-based models. The method improves likelihood and sample quality with substantially fewer timesteps, while FID does not always improve monotonically.

  • Experimental setup: Analytic-DPM is compared with original DDPM and DDIM using even trajectories based on fixed strides and optimal trajectories computed by dynamic programming.For baselines, optimal trajectories use variational bounds computed with their handcrafted variances.
  • Experimental setup: Experiments apply Analytic-DPM to pretrained score-based models on CelebA 64x64, ImageNet 64x64, and LSUN Bedroom, plus models trained on CIFAR10.The CIFAR10 models use two different forward noise schedules.
  • 6.1 Likelihood: Across all three datasets, Analytic-DPM consistently improves DDPM likelihood under both even and optimal trajectories, and shorter optimal trajectories can outperform full-timestep baselines.Likelihood results are reported only under the DDPM forward process because the DDIM variational bound is infinite for σ_n = 0.
  • 6.1 Likelihood: 3.61 bits/dim: Analytic-DPM’s full-timestep NLL on ImageNet 64x64 is close to the 3.57 reported for a neural-network variance baseline.The comparison also addresses a strong variance-prediction baseline from Nichol and Dhariwal (2021).
  • 6.2 Sample quality: 20× to 80× speed up: Analytic-DDIM consistently improves DDIM sample quality and generates high-quality samples with fewer than 50 timesteps.FID is the sample-quality metric, with lower values indicating better quality.
  • 6.2 Sample quality: Analytic-DDPM does not always outperform its baseline on FID, and increasing the number of timesteps does not necessarily improve FID.The paper attributes this behavior to the differing natures of likelihood and FID metrics.

7 RELATED WORK

This section situates diffusion probabilistic models among foundational formulations, acceleration methods for shorter trajectories, and approaches to learning diffusion variances. It also distinguishes this work’s focus on estimating optimal reverse variance from prior forward-schedule learning.

  • DPMs and their applications: DPM research spans variational-bound training, reparameterized objectives, and continuous-time stochastic differential equation formulations.Sohl-Dickstein et al. introduced DPMs; subsequent work developed parameterizations, reweighted objectives, and continuous-timestep SDE models.
  • Faster DPMs: Faster-DPM methods seek short trajectories, ranging from six-step grid-search trajectories to dynamic programming for least-cost path optimization.Grid search becomes impractical for longer trajectories because its time complexity grows exponentially; this work uses the dynamic programming algorithm.
  • Variance Learning in DPMs: Variance-learning research also targets the forward noise schedule, while this work primarily applies optimal reverse-variance estimation to DDPMs and DDIMs.Variational diffusion models parameterize forward variance with a signal-to-noise ratio function and optimize the variational bound for better log-likelihood.

8 CONCLUSION

The paper derives analytic forms for a DPM’s optimal reverse variance and corresponding optimal KL divergence from its score function, then proposes Analytic-DPM to estimate them during training-free inference. It also derives variance bounds to correct potential score-model bias.

  • The optimal reverse variance and corresponding optimal KL divergence of a DPM have analytic forms with respect to its score function.
  • Analytic-DPM is a training-free inference framework that estimates these analytic quantities using Monte Carlo sampling and a pretrained score-based model.
  • Bounds on the optimal variance are derived to correct potential bias introduced by the score-based model.

ETHICS STATEMENT … A.3 PROOF OF THEOREM 2

The appendices establish foundational entropy, KL, Markov, and score-function lemmas, then derive analytic optimal reverse-process solutions and bounds. The paper also states that its machine-learning technique presents no obvious negative consequences and is unlikely to be misused at its current stage.

  • ETHICS STATEMENT: The ethics statement characterizes the work as fundamental machine-learning research whose negative consequences are not obvious and whose misuse is currently unlikely.
  • A.1 LEMMAS: The appendix develops lemmas relating cross-entropy and KL divergence to Gaussian distributions, Markov-chain structure, and entropy calculations for DDPM forward processes.It also treats conditional and generalized DDPM forward processes and KL divergence to Markov chains.
  • A PROOFS AND DERIVATIONS: The proof appendix derives the theorem results by decomposing joint KL optimization into timestep-level problems and identifying the minimum through Gaussian moment calculations.
  • A.1 LEMMAS: Lemma 9 shows that optimizing a Markov reverse process with Gaussian transitions is equivalent to matching the first two moments of q(x_n−1|x_n).The lemma applies without assuming a particular form for q(x_0:N), including more general Gaussian models such as multilayer VAEs with Gaussian decoders.
  • A.1 LEMMAS: Score-function lemmas express conditional expectations and covariances through marginal scores, and convert moments of q(x_n−1|x_n) into moments of q(x_0|x_n).The marginal q_n(x_n) is identified as the forward-process distribution at timestep n, with d denoting data dimension.
  • A.2 PROOF OF THEOREM 1: Theorem 1 gives a score representation of the optimal solution to Eq. (4), with its proof based on the conditional-moment and moment-conversion lemmas.
  • A.3 PROOF OF THEOREM 2: Theorem 2 establishes lower and upper bounds for the optimal reverse variance, with an additional upper bound when q(x_0) is bounded in [a, b]^d.The bounded-distribution argument uses that q(x_0|x_n) remains bounded and applies the bounded-covariance lemma.

A.4 PROOF OF THE DECOMPOSED OPTIMAL KL … C THE BOUNDS OF THE OPTIMAL REVERSE VARIANCE CONSTRAINED ON A TRAJECTORY

The appendices formalize the decomposed optimal KL, connect score functions to conditional data covariance, describe dynamic programming for least-cost trajectories, and establish bounds for trajectory-constrained optimal reverse variance.

  • A.4 PROOF OF THE DECOMPOSED OPTIMAL KL: Theorem 3 states the decomposed optimal KL for the shorter forward process and its optimal reverse process.The proof uses the KL decomposition into conditional and marginal terms, with a trajectory-independent constant c.
  • A.5 THE FORMAL RESULT FOR SECTION 5 AND ITS PROOF: The expected conditional covariance matrix of the data distribution is determined by the score function ∇x_n log q_n(x_n).This covariance contributes to the data covariance matrix through the law of total variance.
  • A.5 THE FORMAL RESULT FOR SECTION 5 AND ITS PROOF: The law of total variance expresses data covariance as expected conditional covariance plus covariance of the conditional mean.The appendix proves this classical statistical identity for completeness.
  • B THE DP ALGORITHM FOR THE LEAST-COST-PATH PROBLEM: Dynamic programming stores minimized costs C[k, n] and predecessor indices D[k, n], then backtracks from τ_K = N to recover the optimal trajectory.Algorithm 1 initializes C and D, computes stagewise minima, and recursively sets τ_k−1 = D[k, τ_k].
  • B THE DP ALGORITHM FOR THE LEAST-COST-PATH PROBLEM: The least-cost-path problem seeks a k-node trajectory 1 = τ_1 < ··· < τ_k = n minimizing the sum of transition costs J(τ_i, τ_i+1).The cost function is defined for s < t, with J(s, t) = ∞ when s ≥ t ≥ 1.
  • C THE BOUNDS OF THE OPTIMAL REVERSE VARIANCE CONSTRAINED ON A TRAJECTORY: Corollary 1 establishes bounds for the optimal reverse variance constrained on a trajectory.The result extends the trajectory-constrained variance analysis with bounds analogous to Theorem 2.
  • C THE BOUNDS OF THE OPTIMAL REVERSE VARIANCE CONSTRAINED ON A TRAJECTORY: If q(x_0) is bounded in [a, b]^d, the trajectory-constrained optimal reverse variance σ_n*^2 admits a further upper bound.The bounded-support assumption applies to a d-dimensional data distribution.

D SIMPLIFIED RESULTS FOR THE DDPM FORWARD PROCESS … G.1 VISUALIZATION OF REVERSE VARIANCES AND VARIATIONAL BOUND TERMS

The paper simplifies its DDPM results, extends the analytic variance and trajectory framework to continuous timesteps, and documents implementation choices and additional visualizations. These sections establish analytic score-based formulas, practical estimation procedures, and experimental settings for evaluating reverse variances and variational-bound terms.

  • D SIMPLIFIED RESULTS FOR THE DDPM FORWARD PROCESS: For DDPM forward processes, the paper states simplified corollaries for the optimal solution, optimal reverse-variance bounds, and decomposed optimal KL.These results specialize the general analytic framework to DDPMs.
  • E EXTENSION TO DIFFUSION PROCESS WITH CONTINUOUS TIMESTEPS: The continuous-time extension uses an SDE parameterization with scalar functions αt and βt over t ∈ [0, 1], inducing Gaussian transition distributions.The transitions are specified for both q(zt|x0) and q(zt|zs).
  • E.1 ANALYTIC ESTIMATE OF THE OPTIMAL REVERSE VARIANCE: The continuous-time reverse process has an analytic KL-optimal solution, with both optimal mean and variance expressed in terms of the score function.The framework estimates the expected squared score norm by Γt.
  • E.1 ANALYTIC ESTIMATE OF THE OPTIMAL REVERSE VARIANCE: Because continuous time has infinitely many timesteps, the method evaluates Γ at finitely many selected timesteps and linearly interpolates between them.This provides a practical estimate of the optimal reverse variance between sampled timepoints.
  • E.2 ANALYTIC ESTIMATION OF THE OPTIMAL REVERSE TRAJECTORY: The optimal continuous-time reverse trajectory minimizes KL, and dynamic programming can optimize it after restricting the continuous objective to finitely many timesteps.The continuous objective is defined on 0 ≤ s < t ≤ 1, so the unrestricted problem is not directly handled by the DP algorithm.
  • F EXPERIMENTAL DETAILS; F.1 DETAILS OF SCORE-BASED MODELS: Experiments use pretrained score-based models for CelebA 64x64, LSUN Bedroom, and ImageNet 64x64, while CIFAR10 models are trained with linear and cosine noise schedules.The reported model configurations include N = 1000 timesteps for CelebA, LSUN, and CIFAR10, and N = 4000 for ImageNet 64x64.
  • F.2 LOG-LIKELIHOOD AND SAMPLING; F.3 CHOICE OF THE NUMBER OF MONTE CARLO SAMPLES AND CALCULATION OF Γ; F.4 IMPLEMENTATION OF THE EVEN TRAJECTORY; F.5 EXPERIMENTAL DETAILS OF TABLE 3: Implementation details define discrete log-likelihood evaluation, noise-scale clipping during sampling, Monte Carlo sample counts, Γ reuse, even-trajectory construction, and matched model comparisons.The analytic estimate uses dataset-specific default M values, Γ is calculated once per pretrained model, and Table 3 compares methods using the same score-based models.
  • G ADDITIONAL RESULTS; G.1 VISUALIZATION OF REVERSE VARIANCES AND VARIATIONAL BOUND TERMS: Additional figures compare analytic reverse-variance estimates with handcrafted βn and ˜βn across datasets and compare per-timestep Lvb terms whose area gives Lvb.Figure 1 covers CIFAR10 with the linear schedule, while Figure 2 adds cosine-schedule CIFAR10, CelebA 64x64, and ImageNet 64x64 results.

G.2 ABLATION STUDY ON THE NUMBER OF MONTE CARLO SAMPLES … H.4 FUTURE WORKS

The experiments show that Analytic-DPM remains effective with few Monte Carlo samples, tight variance bounds, sampling-oriented clipping, and short trajectories, while additional discussion addresses cost, stochasticity, Gaussian-model connections, and future extensions.

  • G.2 ABLATION STUDY ON THE NUMBER OF MONTE CARLO SAMPLES: M = 10 or 100 Monte Carlo samples usually suffice: M = 10 keeps Γn’s RSD below 0.1 at n = 1, while M = 100 yields about 0.025.For n > 100, even M = 1 keeps Γn’s RSD below 0.05, and M = 10 matches M = 50000 under NLL and FID.
  • G.3 TIGHTNESS OF THE BOUNDS: The combined upper and lower bounds nearly overlap for full-timestep trajectories, while Eq. (11) is tighter at small timesteps and Eq. (12) at large timesteps.The clipping frequency rises with n because the two bounds converge there, with similar curves across Monte Carlo sample counts.
  • G.4 ABLATION STUDY ON THE CLIPPING OF σ2 DESIGNED FOR SAMPLING: Sampling-oriented clipping greatly improves Analytic-DPM sample quality, using a threshold 1 to 3 orders of magnitude below the combined upper bound when K is small.Clipping also slightly improves the handcrafted DDPM reverse variance, whereas DDIM’s zero variance generally remains below the threshold.
  • G.5 SAMPLE QUALITY COMPARISON BETWEEN DIFFERENT TRAJECTORIES: The even trajectory provides better FID than the optimal trajectory, although the optimal trajectory significantly improves likelihood.On ImageNet 64x64, Analytic-DPM reaches NLL 3.61 versus 3.57 for Improved DDPM with full timesteps, while Analytic-DPM improves original DDIM consistently on CelebA 64x64.
  • G.8 COMPARISON TO OTHER CLASSES OF GENERATIVE MODELS: Analytic-DPM achieves competitive sample quality against other generative-model classes, and Analytic-DDIM with K = 50 produces samples comparable to the best-FID setting.The comparison reports FID, model function evaluations, and time for generating 10 samples on CIFAR10.
  • H.1 THE EXTRA COST OF THE MONTE CARLO ESTIMATE: The Monte Carlo estimate adds MN model evaluations, but its Γ values can be computed once and reused across inference trajectories because shorter-process marginals match full-process marginals.This discussion explains why the additional estimation cost is small and reusable after pretraining.
  • H.2 THE STOCHASTICITY OF THE VARIATIONAL BOUND AFTER PLUGGING THE ANALYTIC ESTIMATE: Plugging the estimated variance makes the variational bound stochastic; Monte Carlo variance can be reduced with more samples, whereas score-model error is irreducible for a fixed pretrained model.This motivates clipping the estimate.
  • H.3 COMPARISON TO OTHER GAUSSIAN MODELS AND THEIR RESULTS: Analytic-DPM relates to expectation propagation through moment matching but must handle nontrivial conditional moments involving unknown data distributions, and future work includes other modalities, continuous DPMs, and training integration.The method converts the difficult second moment of q(xn−1|xn) into one involving q(x0|xn), expressible through the score.
Loading 2201.06503v3…