PaperlayerView exact revision on arXiv

Source-linked AI summary

Denoising Diffusion Samplers

Francisco Vargas, Will Grathwohl, Arnaud Doucet

arXiv:2302.13834v2cs.LGstat.ML

TL;DR

The paper addresses sampling from unnormalized densities and estimating their intractable normalizing constants when score matching cannot be used. It introduces Denoising Diffusion Samplers, which approximate a target-to-Gaussian diffusion’s time reversal, and finds them competitive with state-of-the-art SMC on challenging multimodal targets while being more stable than PIS.

  • Problem

    Sampling approximately from an unnormalized density and estimating its intractable normalizing constant is difficult because the target cannot be sampled for score matching.

  • Method

    Denoising Diffusion Samplers diffuse the target toward a Gaussian, approximate the reverse process with qθ, and optimize KL(qθ||p) through an ELBO and auxiliary reference process.

  • Results

    DDS is empirically competitive with state-of-the-art SMC and numerically more stable than PIS, while outperforming carefully tuned SMC on the challenging multimodal NICE example.

  • Takeaways & Limitations

    DDS extends denoising-diffusion ideas to sampling unnormalized probability distributions and estimating their normalizing constants.

  • Takeaways & Limitations

    Probability-flow normalizing flows and underdamped DDS were disappointing in high-dimensional scenarios, motivating better integrators and neural-network parameterizations.

Abstract

from arXiv · show

Denoising diffusion models are a popular class of generative models providing state-of-the-art results in many domains. One adds gradually noise to data using a diffusion to transform the data distribution into a Gaussian distribution. Samples from the generative model are then obtained by simulating an approximation of the time-reversal of this diffusion initialized by Gaussian samples. Practically, the intractable score terms appearing in the time-reversed process are approximated using score matching techniques. We explore here a similar idea to sample approximately from unnormalized probability density functions and estimate their normalizing constants. We consider a process where the target density diffuses towards a Gaussian. Denoising Diffusion Samplers (DDS) are obtained by approximating the corresponding time-reversal. While score matching is not applicable in this context, we can leverage many of the ideas introduced in generative modeling for Monte Carlo sampling. Existing theoretical results from denoising diffusion models also provide theoretical guarantees for DDS. We discuss the connections between DDS, optimal control and Schrödinger bridges and finally demonstrate DDS experimentally on a variety of challenging sampling tasks.

1 INTRODUCTION

The paper adapts denoising-diffusion ideas to sampling from unnormalized densities and estimating their normalizing constants, where score matching is unavailable because target samples cannot be generated. DDS instead approximates a target diffusion's time reversal using variational objectives and related generative-modeling tools.

  • DDS targets approximate sampling from π and estimation of its intractable normalizing constant Z when γ is pointwise evaluable.
  • The method follows a forward diffusion that transforms π toward a Gaussian, then approximates the diffusion's time reversal with a learned process qθ.
  • Because the forward diffusion cannot be sampled from without drawing x0 from π, DDS cannot use DDPM's score-matching approach.
  • DDS minimizes KL(qθ||p), equivalently maximizes an ELBO, using an auxiliary reference process to obtain lower-variance objective and gradient estimates.
  • The framework imports probability-flow ODEs and underdamped Langevin diffusions from generative modeling to construct additional samplers for Monte Carlo applications.

2 DENOISING DIFFUSION SAMPLERS: CONTINUOUS TIME

DDS formulates sampling as approximating the time reversal of a diffusion that carries the target density to a Gaussian. A Gaussian-initialized reference process enables a variational KL formulation, while a learned reverse process produces approximate target samples without score matching.

  • An Ornstein–Uhlenbeck forward diffusion approximately transports π to a Gaussian marginal by time T.
  • The time reversal starts from an approximately Gaussian state and would yield samples from π if it could be simulated, but its score terms are intractable.
  • 2.2 REFERENCE DIFFUSION AND VALUE FUNCTION: DDS introduces a Gaussian-initialized reference process whose marginals remain Gaussian, making the target path measure a KL minimizer under the initial-marginal constraint q0=π.
  • 2.3 PARAMETERIZING THE TIME-REVERSAL: A parameterized reverse path measure Qθ uses a neural network score approximation, but unlike DDPM its KL objective remains dependent on intractable target scores.
  • 2.3 PARAMETERIZING THE TIME-REVERSAL: Reparameterizing the reverse process with sθ(t,x)+x/σ^2 yields a compact KL expression, and optimizing it allows approximate samples from π to be generated by simulating the learned reverse dynamics.

0. We can obtain an unbiased estimate of Z via the following importance sampling identity

The paper extends DDS with probability-flow ODE normalizing flows and underdamped Langevin dynamics. These constructions support approximate sampling and, for the ODE flow, unbiased normalizing-constant estimation, while the underdamped extension augments the state with momentum.

  • The probability-flow ODE shares the diffusion's marginals and can generate approximate samples from π using a learned approximation to ∇ln ϕt(x).
  • The ODE flow supports an unbiased importance-sampling estimate of Z because its endpoint density can be computed with the instantaneous change-of-variables formula.
  • The underdamped extension diffuses the augmented target π(x)N(m;0,M) using Langevin dynamics and defines a corresponding path measure with stationary Gaussian reference marginals.
  • The underdamped reverse process is parameterized analogously to the basic DDS construction, with its KL objective admitting a compact formulation and a normalizing-flow variant.
  • Algorithm 1 trains DDS by repeatedly simulating reverse integration steps from Gaussian initial states with a neural network and scheduled diffusion coefficients.

3 DENOISING DIFFUSION SAMPLERS: DISCRETE-TIME

The discrete-time method requires integrators for both the learned reverse process and the reference process so that normalizing-constant estimates preserve an ELBO. Standard Euler–Maruyama discretization is usable but fails the condition needed for this guarantee.

  • Discrete DDS requires integrators for both Qθ and Pref, with careful design needed to preserve the ELBO used for normalizing-constant estimation.
  • 3.1 INTEGRATOR FOR Pref: Euler–Maruyama discretization is applicable to the SDEs but is problematic for the reference process.
  • 3.1 INTEGRATOR FOR Pref: The EM discretization of Pref does not satisfy the condition guaranteeing that EQθ[ln ˆZ] is an ELBO for the estimate ˆZ.
  • 3.1 INTEGRATOR FOR Pref: The EM-based estimator can therefore satisfy EQθ[ln ˆZ] ≥ ln Z, motivating a dedicated Pref integrator to resolve the issue.
  • 3.1 INTEGRATOR FOR Pref: The reference diffusion can be integrated exactly from a Gaussian initial state, providing the basis for the required reference-process construction.

2. This defines the discrete-time reference process. We propose

DDS defines a diffusion-based sampler by approximating the time-reversal of a process that transforms an unnormalized target toward a Gaussian. Its objective connects reverse-KL optimization with control and Schrödinger-bridge formulations, while importance sampling provides an unbiased normalizing-constant estimate.

  • DDS approximates the time-reversal of a target-to-Gaussian diffusion, producing samples from a learned process qθ initialized from a Gaussian.The final sample distribution approximates π by design.
  • Importance sampling yields an unbiased estimate of the intractable normalizing constant from trajectories sampled under qθ.The estimator uses the target density, reference-path density, Gaussian initialization density, and proposal-path density.
  • Existing diffusion-model guarantees imply DDS can achieve ε-close total variation error in O(L^2d/ε^2) steps under Lipschitz-score and approximation-error assumptions.The bound does not require a log-concavity assumption, although the score-approximation assumption is less realistic for DDS.
  • DDS has an optimal-control interpretation and approximately solves a Schrödinger-bridge problem whose endpoint measures are π and a Gaussian.For sufficiently large K, the Schrödinger-bridge solution is approximately represented by the forward noising process and its time-reversal.
  • Compared with PIS, DDS uses overdamped or underdamped Ornstein–Uhlenbeck reference dynamics rather than pinned Brownian motion.The paper presents DDS as an alternative whose reference dynamics differ from those used by PIS.

4 EXPERIMENTS

The experiments evaluate DDS against SMC, PIS, and Gaussian MF-VI across standard targets, Bayesian models, and a high-dimensional NICE target. The study also examines optimization stability, mode collapse, and the effect of implementation choices.

  • Detaching the target score stabilized optimization for both approaches without affecting the final result.The authors apply this choice across experiments and report an ablation in Appendix C.8.1.
  • DDS is compared with SMC, PIS, and Gaussian MF-VI across challenging sampling and normalizing-constant estimation tasks.The benchmark includes standard targets, Bayesian datasets, Brownian motion, and a pretrained normalizing-flow target.
  • PIS used a more favorable σf = 1 setting in prior work than the σf = 3 setting used for other methods, explaining differences from the reported results.The paper identifies this hyperparameter mismatch as a source of discrepancies with Zhang and Chen (2022).
  • The NICE experiment evaluates high-dimensional mode collapse using Sinkhorn distance between target and sampler-generated samples.The target is a trained normalizing flow, allowing direct comparison with true samples.

5 DISCUSSION

The discussion positions DDS as a promising but still developing approach for sampling unnormalized distributions and estimating normalizing constants. DDS is competitive with SMC on challenging multimodal targets, but incurs training costs and inherits reverse-KL mode-dropping risks.

  • 5 DISCUSSION: DDS is empirically competitive with state-of-the-art SMC and more numerically stable than PIS, although training time can make SMC preferable on simple targets.On the challenging multimodal NICE example, even carefully tuned SMC with Hamiltonian Monte Carlo transitions was not competitive with DDS.
  • 5 DISCUSSION: DDS and PIS remain prone to mode dropping because both rely on reverse KL.
  • 5 DISCUSSION: Probability-flow ODE normalizing flows and underdamped DDS performed disappointingly in high-dimensional experiments.
  • 5 DISCUSSION: The authors identify improved numerical integrators and neural-network parameterizations as needed for stronger normalizing-flow and underdamped-DDS performance.
  • 5 DISCUSSION: The paper presents DDS as a class of algorithms worth further investigation, including through improved diffusion, integration, KL-control, and path-integral-control techniques.
  • 5 DISCUSSION: The DDS construction uses Girsanov-based path-measure comparisons and value-function derivatives to specify proposal drifts.

A.3 PROOF OF PROPOSITION 2

This proof develops the discrete-time DDS construction by relating reference and proposal transitions, establishing the KL decomposition, and showing how the learned reverse process yields target-distributed samples. It also explains why invariant-marginal preservation matters for the estimator's lower-bound interpretation.

  • A.3 PROOF OF PROPOSITION 2: If an integrator fails to preserve the reference marginals, the resulting expression need not define a probability density, so the estimator of ln Z is not guaranteed to remain a lower bound.
  • A.3 PROOF OF PROPOSITION 2: The underdamped construction augments the target with momentum and uses a Hamiltonian flow that preserves the Gaussian invariant distribution before composing stochastic transitions.
  • A.3 PROOF OF PROPOSITION 2: The KL proof relies on Radon–Nikodym derivatives between proposal and reference path measures, with Girsanov's theorem supplying the continuous-time density-ratio relation.
  • A.3 PROOF OF PROPOSITION 2: The learned reverse process can produce a sample with distribution π(yT)N(nT; 0, M) when integrated from the terminal marginal.
  • A.3 PROOF OF PROPOSITION 2: The proposal uses a learned approximation fθ∗(t, x, m) ≈ ∇m ln ϕt(x, m) to construct a process whose samples are close to π × N(0, M), followed by importance-sampling correction.
  • A.3 PROOF OF PROPOSITION 2: The discrete-time reference and proposal transitions are constructed with matching deterministic components and Gaussian stochastic components, enabling tractable density-ratio calculations.

B.6 DERIVATION OF UNDERDAMPENED KL IN DISCRETE TIME

This derivation obtains the discrete-time KL density ratio for the underdamped DDS construction. The integrator is designed so deterministic components cancel between proposal and reference transitions, simplifying the ratio calculation.

  • B.6 DERIVATION OF UNDERDAMPENED KL IN DISCRETE TIME: The underdamped discrete-time log density ratio is defined as the logarithm of the proposal-to-reference path-density ratio.
  • B.6 DERIVATION OF UNDERDAMPENED KL IN DISCRETE TIME: For small step sizes, the discrete coefficients satisfy approximations consistent with the continuous-time dynamics.
  • B.6 DERIVATION OF UNDERDAMPENED KL IN DISCRETE TIME: Matching deterministic transition components makes their contributions cancel, leaving a well-defined ratio between the proposal and reference stochastic transitions.
  • B.6 DERIVATION OF UNDERDAMPENED KL IN DISCRETE TIME: The remaining calculations follow the same Radon–Nikodym strategy used earlier for the non-underdamped construction.

C EXPERIMENTS

The experiments evaluate DDS across standard sampling, normalizing-flow, numerical-integration, and ablation settings, showing strong mode coverage and improved stability with suitable integrators and schedules. They also identify limitations in image quality, probability-flow ODE estimation, and underdamped dynamics.

  • C.3 BROWNIAN MOTION MODEL WITH GAUSSIAN OBSERVATION NOISE: PIS drifts had larger, more variable magnitudes and greater numerical instability than the lower-variance OU drift, which helps explain failed convergence at many steps.This observation came from scalar Gaussian-target experiments comparing learned drift magnitudes.
  • C.8 EULER–MAYURAMA VS EXPONENTIAL INTEGRATOR: EM discretization significantly overestimated ln Z at lower step counts, whereas the proposed exponential integrator was introduced to preserve the ELBO.The comparison was performed on Funnel and Ionosphere, with reliable or known normalizing constants available.
  • C.9 COSINE BASED DECAY FOR PIS: Cosine-squared scheduling significantly improved DDS across every task, while cosine discretization for PIS generally decreased or preserved performance except on Sonar.The schedule comparison was conducted as an ablation against uniform discretization.
  • C.10 PROBABILITY FLOW ODE: Probability-flow ODE samples roughly matched DDS SDE marginals after Heun integration and additional training changes, but its ln Z estimator still drastically overestimated.More complex mixture distributions caused stiffness and incorrect sample shapes under the initial discretization.
  • C.11 UNDERDAMPED OU RESULTS: The underdamped approach performed worse than DDS with a standard OU reference process and PIS, though it remained better than VI and overestimated ln Z on Funnel at 64 steps.The authors attribute the Funnel overestimate to possible numerical error during sample generation.

D DENOISING DIFFUSION SAMPLERS IN DISCRETE-TIME: FURTHER

The discrete-time DDS construction approximates the forward process’s time-reversed ancestral sampler, while replacing unavailable backward kernels with tractable learned approximations. It also develops reference-process, optimization, and normalizing-flow components for sampling and evidence estimation.

  • D.1 FORWARD PROCESS AND ITS TIME REVERSAL: The method approximates ancestral sampling from the backward decomposition that would recover π if the terminal and backward distributions were tractable.The target starts as p0(x0) = π(x0), but the required backward kernels cannot be implemented exactly.
  • D.1 FORWARD PROCESS AND ITS TIME REVERSAL: DDS initializes the reverse procedure with a Gaussian sample because the forward process is designed to end near N(0, σ2I).Exact ancestral sampling remains unavailable because the intermediate marginal densities are not available in closed form.
  • D.4 EXPERIMENTS: The experiments evaluate ln Z estimates across Funnel, LGCP, Logistic Ionosphere, Logistic Sonar, Brownian motion, and NICE as the number of steps K changes.The figures compare estimates with MF-VI and a gold standard; the supplied passages do not state the resulting numerical winners.
  • D.2 A REFERENCE PROCESS: A simple Gaussian reference process makes the intractable backward kernels expressible as twisted versions of tractable reference-process kernels.The reference process uses Gaussian marginals, enabling a path-space formulation of the approximation problem.
  • D.3 APPROXIMATING THE BACKWARD KERNELS: The backward-kernel approximation targets ∇ln ϕk through learned criteria because direct Monte Carlo estimates of the value functions are high variance and conditional sampling would defeat the methodology.The resulting qθ uses qK(xK) = N(xK; 0, σ2I) and parameterized transitions intended to approximate the backward kernels.
  • D.4 EXPERIMENTS: The implementation reports fitted hyperparameters for the funnel, LGCP, ion, sonar, Brownian, VAE, and NICE experiments to support reproducibility.The reported optimization setup includes Adam with learning rate 0.0001 and 11000 training iterations for experiments other than Funnel.
Loading 2302.13834v2…