PaperlayerView exact revision on arXiv

Source-linked AI summary

When, why, and how do diffusion posterior samplers fail? A finite-sample lens

Benjamin A. Burns, Sara Fridovich-Keil

arXiv:2605.30330v1cs.LG

TL;DR

Posterior sampling with diffusion models uses inexact intermediate-time likelihood approximations whose downstream effects are poorly understood. The paper introduces a finite-sample lens to examine these effects and finds that approximation errors in intermediate posterior spread can lead to posterior errors, even with linear measurements and a unimodal posterior.

  • Problem

    The downstream effects of tractable likelihood approximations in diffusion posterior sampling are poorly understood, despite their use in imaging inverse problems.

    Source: §1 · p. 1

  • Method

    The paper uses a finite-sample perspective to approximate posteriors at intermediate timesteps and study sampler errors.

    Source: §4 · p. 7, §6 · p. 13

  • Results

    Gaussian and Dirac approximations can misrepresent intermediate posterior spread and cause mode-weighting errors or hallucinations, even with linear measurements and a unimodal posterior when the prior is multimodal.

    Source: §1 · p. 2, §6 · p. 13

  • Takeaways & Limitations

    Intermediate-time posterior accuracy is important for understanding how terminal errors arise and for evaluating diffusion posterior samplers.

    Source: §6 · p. 13

    Finite-sample estimates near time zero require substantially more prior samples, increasing memory and computational cost.

    Source: §4.1 · p. 9

Abstract

from arXiv · show

Diffusion models have excellent capacity to model complex distributions of natural data, which has made them a popular and effective choice for posterior sampling in imaging inverse problems. Existing methods can incorporate any measurement model at inference time but must use an inexact approximation for the likelihood at intermediate timesteps for computational tractability. Although these approximations can often work well empirically, their downstream effect on the sampled posterior is poorly understood and can result in unexplained failures. To understand when, why, and how these likelihood approximations propagate to erroneous posterior distributions, we introduce a finite-sample perspective on posterior sampling that approximates the posterior to arbitrary precision as training set size tends towards infinity, for any forward model and prior distribution. Using this finite-sample lens, we observe that popular posterior sampling approximations tend to under- or over-estimate the spread of the posterior at intermediate timesteps, causing downstream consequences including sensitivity to early stopping time, inaccurate relative weighting of posterior modes, and hallucination, both of prior modes that are not in the posterior and likelihood modes that are not supported by the prior. Moreover, we find that the cause of these posterior errors requires neither a nonlinear measurement model nor a multimodal posterior, but can arise solely due to a multimodal prior and inaccurate posterior spread at intermediate sampling times. Our finite-sample posterior sampling approach is agnostic to the type of likelihood approximation and the type of (linear or nonlinear) forward model, and can thus serve as a drop-in diagnostic to evaluate the accuracy and failure modes of existing and future posterior samplers.

1 Introduction

Posterior sampling matters when measurements and priors do not uniquely determine an image, including cases where low-probability modes can have high impact. The paper uses a finite-sample lens to analyze how likelihood approximations in diffusion posterior samplers produce errors.

1 Introduction

  • Popular likelihood approximations can fail in imaging settings, but their sporadic failures and strong performance are poorly understood.Quantized sensing is one cited failure context.Source: §1 · p. 2
  • The paper introduces a finite-sample perspective that approximates the posterior to arbitrary precision as training-set size grows, for any forward model and prior.It uses this perspective to analyze approximation errors and their downstream effects.Source: §1 · p. 2
  • Gaussian approximations can commit to prior modes too early, misweight posterior modes, and hallucinate prior modes absent from the posterior.The paper also finds that DPS's parameter ζ can over- or under-weight the likelihood at intermediate timesteps, causing erroneous spread and hallucinations.Source: §1 · p. 2
  • These errors can arise with linear measurements, Gaussian noise, and a unimodal posterior when the prior is multimodal.Source: §1 · p. 2

2 Related work

Prior work develops moment-matching samplers with practical advantages, while comparisons have focused more on terminal samples than intermediate behavior. This paper brings a finite-sample perspective to the study of posterior sampling.

Generative posterior samplers

  • Moment-matching methods [16] [18] can reuse pre-trained prior scores and let forward model A change at inference without additional training; changing an inpainting mask necessitates retraining methods learning likelihood or posterior scores.Source: §2 · p. 2

Benchmarking vs. understanding

  • Existing benchmarks compare terminal-time sample quality, while intermediate-time behavior remains less understood despite its relevance to explaining terminal errors and anticipating failures [13] [20] [21] [22].Source: §2 · p. 3

Exact diffusion

  • The paper presents what it describes as the first finite-sample perspective for understanding posterior sampling [26].Source: §2 · p. 3

3 Preliminaries

The preliminaries frame posterior sampling as Bayesian inversion with diffusion priors, then review moment-matching likelihood approximations and practical variants. They motivate a finite-sample method for examining exact and intermediate-time posteriors.

3.1 Bayesian inverse problems

  • Bayesian inverse problems use measurements from a known, possibly nonlinear operator and a prior to recover a signal when inversion is challenging or ambiguous.The posterior represents reconstructions consistent with both prior and measurements.Source: §3.1 · p. 3

3.2 Diffusion-based generative models

  • Score-based diffusion models represent complex priors, and their reverse process uses the prior score to generate samples from that prior.The process starts from Gaussian noise at large time and runs toward the prior distribution.Source: §3.2 · p. 4

3.3 Diffusion posterior sampling via moment matching

  • Moment-matching methods replace the denoiser with a tractable distribution matching its first and/or second moments, computable from the marginal score and noise schedule via Tweedie’s formula.Source: §3.3 · p. 5
  • Dirac approximations simplify across measurement models but omit denoiser spread, whereas Gaussian approximations encode covariance but require affine measurements and additive Gaussian noise [16–18].The analysis further limits Gaussian approximations to isotropic measurement-noise covariance.Source: §3.3 · p. 6

3.4 Practical diffusion posterior sampling

  • For DPS, a data-dependent step size ρ(xt), commonly ζ divided by the measurement residual, replaces the constant noise prefactor and is equivalent to a Laplace likelihood approximation [16].ζ is problem-dependent and must be tuned.Source: §3.4 · p. 7
  • Song et al.'s noisy-problem demonstrations are restricted to operators with orthonormal rows, while Boys et al.'s full covariance approach can require costly assembly and inversion [18].Heuristic covariance approximations can reduce computational cost.Source: §3.4 · p. 7
  • Likelihood approximations are difficult to assess because errors in the likelihood score can propagate through iterative sampling alongside the prior score.The paper introduces finite-sample access to exact and intermediate-time posteriors to study those effects.Source: §3.4 · p. 7

4 Finite-sample diffusion posterior sampling

The finite-sample regime replaces a continuous prior with a discrete measure of i.i.d. samples, making intermediate-time diffusion and posterior quantities analytically accessible. The authors use this representation to examine likelihood approximations, while noting that it does not generalize beyond the training data without modification.

4 Finite-sample diffusion posterior sampling

  • Representing the prior as a discrete measure of N i.i.d. samples makes the denoiser analytically computable at arbitrary timesteps.This finite-sample view provides a closed-form approximation to the prior and permits analysis of likelihood-approximation errors.Source: §4 · pp. 7–8
  • The denoiser assigns zero probability outside the dataset, so generalization requires an additional modification such as smoothing.The discrete structure is useful for analyzing posterior sampling, but the method needs substantially more prior samples near time zero and may require modification to generalize beyond the dataset.Source: §4 · p. 8, §4.1 · p. 9

4.1 Posterior sampling in the finite-sample regime

  • The finite-sample denoiser yields a closed-form observation likelihood even when the forward model is nonlinear.Dirac approximations fail to capture higher moments, while Gaussian approximations apply only when the forward model is affine.Source: §4.1 · p. 8
  • The likelihood weights favor training points that both plausibly noise to the current state and produce measurements consistent with y.The noising term depends on the current state, whereas the measurement likelihood depends on the observed measurement.Source: §4.1 · pp. 8–9
  • Analytically accessible posterior scores and densities enable direct draws at positive timesteps without iterative denoising.This avoids iterative procedures such as Euler–Maruyama denoising and is significantly cheaper numerically.Source: §4.1 · p. 9
  • The finite-sample method serves as an intermediate-timestep proxy because it is relatively inaccurate near time zero, even with large datasets.The required sample count scales on the order of N^-1/2, with a time-dependent constant that increases memory and computation near zero.Source: §4.1 · p. 9

5 Experiments

The experiments compare finite-sample posterior sampling with moment-matching approximations on linear and nonlinear measurement problems. They show that finite-sample accuracy depends on diffusion time, while approximation errors can distort posterior spread, mode coverage, and mode weighting.

Experiments

  • The finite-sample posterior converges at the expected Monte Carlo rate for each fixed time, but requires more samples closer to time zero.The true and finite-sample posteriors are qualitatively identical at intermediate timesteps.Source: §5 · p. 10, Figure 1 · p. 10, Figure 2 · p. 11
  • Second-moment Gaussian approximations overestimate posterior covariance with a multimodal prior, producing measurement-inconsistent samples even with the true unconditional marginal score.The examples use linear measurements and an informative measurement that yields a unimodal posterior.Source: §5 · p. 10
  • The DPS Dirac approximation has incorrect variance at all times and incorrect mean at intermediate times, while Gaussian approximations sample prior modes inconsistent with measurements.Figure 2 compares the methods’ reverse processes with ground-truth and finite-sample posteriors.Source: Figure 2 · p. 11
  • At fixed N = 4096, finite-sample total variation accuracy deteriorates as t approaches zero but remains accurate at moderately large times.Near zero, the discrete finite-sample measure and a density have total variation distance 1.Source: Figure 3 · p. 11, §5 · p. 11
  • For nonlinear operators, σ-DPS often misses true posterior modes, samples unsupported prior modes, or weights captured modes incorrectly.The comparisons use analytically known posteriors for the first two rows and the finite-sample regime as a proxy for later rows.Source: §5 · p. 11, Figure 4 · p. 12
  • Tuning ζ can improve posterior-mode weighting and reduce hallucinated likelihood-consistent modes, but cannot eliminate either kind of hallucinated mode in every setting.The examples include prior-consistent modes unsupported by the measurement and measurement-consistent modes implausible under the prior.Source: §5 · pp. 11–12, Figure 4 · p. 12
  • Increasing ζ raises the measurement likelihood’s weight relative to the prior, but can cause mode collapse onto a hallucinatory mode.Smaller ζ increases the prior’s weight relative to the likelihood.Source: §5 · p. 12

6 Discussion

The finite-sample lens gives approximate access to posterior distributions at intermediate timesteps and exposes failures that can occur even with linear measurements and a unimodal posterior. These errors include covariance inflation, mode errors, and inaccurate intermediate distributions; dataset-size requirements are difficult to assess without explicit constant factors.

6 Discussion

  • The finite-sample lens provides approximate access to the posterior at all intermediate timesteps for any prior and measurement model, allowing analysis of existing samplers’ errors.The discussion presents this as a way to study how sampling errors arise.Source: §6 · p. 13
  • Failures can occur with a linear measurement operator and a unimodal posterior, contrary to the common belief that failures require nonlinear operators or multimodal posteriors.Second-moment Gaussian approximations can overinflate covariance with a multimodal prior, producing hallucinations even when the true prior score is used; first-moment Dirac approximations can miss posterior modes or misweight them under nonlinear measurements.Source: §6 · p. 13
  • Intermediate distributions can be inaccurate even when time-zero samples are high quality, and low-probability marginal regions can leave the learned score poorly controlled.The discussion links this to unpredictable behavior and notes consequences for tasks using intermediate states, including out-of-distribution detection.Source: §6 · p. 13
  • Without explicit computation of the relevant constant factors, it is difficult to determine how large a dataset must be for sufficient pointwise posterior accuracy at time t.The factors depend on the noise schedule, time, prior, and sample count.Source: §6 · p. 13
  • Studying learned-prior effects is a proposed extension, and the authors conjecture that certain errors may amplify when the prior is learned.The conjecture concerns particles moving from measurement-consistent into prior-consistent regions of the reverse process.Source: §6 · p. 13

A.1 Generative posterior samplers

The study analyzes likelihood and likelihood-score approximations for posterior sampling when a pretrained prior score is available but the forward model is unavailable during training. It excludes methods that learn likelihood or posterior scores after the forward model is fixed, variable-splitting methods, and flow-based generative samplers.

A.1 Generative posterior samplers

  • The study analyzes likelihood and likelihood-score approximations assuming access to a pretrained prior score but no forward-model access during training.This defines the study’s setting for posterior sampling.Source: §A.1 · p. 17
  • Methods that learn likelihood or posterior scores after the forward model is fixed are outside the study’s scope.The stated reason is the assumption that the forward model is inaccessible offline.Source: §A.1 · p. 17
  • The study excludes variable-splitting methods and limits its analysis to diffusion models rather than flow-driven generative samplers.Variable-splitting methods often perform exact likelihood sampling; the paper notes moment-matching approximations can initialize more expensive methods.Source: §A.1 · p. 17

A.2 Benchmarking vs. understanding

The section distinguishes benchmarking end distributions from explaining when and how sampling errors arise, then presents lemmas for discrete, Gaussian, and Gaussian-mixture updates and their scores. It also places the paper’s use of the finite-sample perspective in relation to prior work.

A.2 Benchmarking vs. understanding

  • End-distribution benchmarks can identify struggling algorithms but do not explain why their behavior arises; this section instead studies how errors arise in examples.Prior work had noted difficulty with multimodal posteriors, while the timing of divergence and how to correct it remained unclear.Source: §A.2 · p. 18

A.3 Exact diffusion

  • Earlier finite-sample posterior-sampling works developed samplers rather than using the perspective to study existing samplers.The cited works include linear-forward-model analysis and a method whose nonlinear operator is linearized in practice.Source: §A.3 · p. 18

B Useful lemmas

  • The useful-lemmas section establishes notation for vectors, measurement spaces, linear and possibly nonlinear forward maps, and covariance matrices.It denotes the forward map by A and the prior and measurement covariances by Cpr and Σy.Source: B Useful lemmas · p. 18

B.1 Linear updates of discrete measures

  • With a Gaussian likelihood and discrete prior, the marginal is a Gaussian mixture and the posterior is a reweighting of the prior.The posterior weights are nonnegative and sum to one.Source: §B.1 · pp. 18–19

B.2 Linear updates of Gaussians

  • The Gaussian-update lemmas give Gaussian marginals and show that a Gaussian prior with Gaussian likelihood has a Gaussian posterior.The posterior is specified by its mean and covariance.Source: §B.2 · pp. 19–20

B.3 Linear updates of Gaussian Mixtures

  • With a Gaussian-mixture prior and Gaussian likelihood, the posterior remains a Gaussian mixture with updated component distributions and weights.The section gives the mixture marginal and component-level updates used to obtain the posterior.Source: §B.3 · pp. 21–23

B.4 Gradients and scores

  • The score lemmas provide a Gaussian-mixture score and a gradient expression for the component-weight function.The proof passages derive the weight gradient using the quotient rule and simplify its terms.Source: §B.4 · pp. 23–25

C.1 Proof of Proposition 4

The proof uses the diffusion forward transition as a likelihood and applies the discrete-measure marginal and posterior lemmas with prior pN to obtain the stated quantities.

C.1 Proof of Proposition 4

  • The proof begins from the diffusion forward transition probability for state xt given initial condition x0.It then applies Lemma 7 with that transition likelihood and prior pN to obtain the marginal, and Lemma 8 to obtain the posterior.Source: §C.1 · p. 26

C.2 Proof of Proposition 5

The appendix derives unconditional diffusion quantities and conditional likelihood, posterior, and score objects for discrete, Gaussian, and Gaussian-mixture priors. Its derivations show where closed forms are available and where general forward models make conditional quantities intractable.

C.2 Proof of Proposition 5

  • The marginal likelihood in Proposition 5 is a Gaussian mixture with means constant in y and x_t, obtained by combining the measurement model with the discrete diffusion posterior.The measurement likelihood is Gaussian, and the diffusion posterior over clean samples is discrete.Source: §C.2 · p. 26

C.3 Proof of Proposition 6

  • Proposition 6 derives the time-t posterior by Bayes’ rule, substituting the likelihood and diffusion marginal and rearranging.The numerator uses the results of Propositions 5 and 4; the denominator uses the measurement model and finite-sample regime.Source: §C.3 · p. 26

D.1.1 Discrete prior

  • For a discrete prior, the denoiser remains discrete, with its mean a weighted average of atoms and covariance encoding their weighted spread.The marginal is a Gaussian mixture, and the conditional marginal integral reduces to a finite sum.Source: §D.1.1 · pp. 27–28

D.1.2 Gaussian prior

  • For a Gaussian prior, the denoiser covariance is independent of x_t, while the conditional likelihood has no closed form for general A.For affine A, the likelihood has a closed form; the posterior is Gaussian.Source: §D.1.2 · pp. 29–30

D.1.3 Gaussian mixture prior

  • For a Gaussian-mixture prior, the denoiser covariance separates within-component covariance from spread between component means.The conditional likelihood is a mixture of integrals, intractable for general A but tractable componentwise for affine A; conditioning and reweighting yields a Gaussian-mixture posterior.Source: §D.1.3 · p. 31

D.2 Experimental framework

The experiments evaluate six posterior distributions on one-dimensional problems with additive Gaussian measurement noise, using a range of discrete, Gaussian, and Gaussian-mixture priors. The per-prior reconstructions vary prior shape across seven test cases.

D.2 Experimental framework

  • The framework evaluates up to six posterior methods on uniform time and spatial grids; ΠGDM and TMPD are restricted to affine measurement models.Computations use a VP-SDE with a linear β-schedule and additive Gaussian measurement noise.Source: §D.2 · pp. 31–32

D.3.1 Discrete prior tri_equal

  • The tri_equal discrete prior has three symmetric atoms with equal weights.It is evaluated across three forward operators and three measurements alongside the methods in the experimental framework.Source: §D.3 · p. 32, §D.3.1 · p. 33

D.3.2 Discrete prior pent_asym

  • The cited passage describes a five-atom discrete prior with mildly asymmetric tails, assigning probabilities 0.10, 0.25, 0.30, 0.25, and 0.10 to its five atoms.Source: §D.3.2 · p. 35

D.3.3 Discrete prior wild

  • The wild discrete prior concentrates 80% of its mass on a tight negative cluster.The remaining atoms carry smaller weights on either side of that cluster.Source: §D.3.3 · p. 37

D.3.4 Gaussian prior narrow

  • The narrow Gaussian prior has small variance and is displaced from zero, so its forward marginal visibly drifts toward the unit Gaussian as t approaches 1.Source: §D.3.4 · p. 39

D.3.5 Gaussian prior wide

  • The wide Gaussian prior has large variance and is displaced from zero.Source: §D.3.5 · p. 41

D.3.6 Gaussian-mixture prior tri_equal

  • The tri_equal Gaussian-mixture prior has three symmetric components with equal weights and equal isotropic variances.Source: §D.3.6 · p. 43

D.3.7 Gaussian-mixture prior bi_asym

  • The bi_asym Gaussian-mixture prior has two components with asymmetric weights and component variances.Source: §D.3.7 · p. 45

E ζ-DPS tuning

The ζ sweep shows that changing the modified DPS sampler’s constant can improve some reconstructions, but does not consistently recover posterior spread, mode weights, and mode coverage together. Outcomes vary across prior and measurement settings, including nonlinear forward models.

ζ-DPS tuning

  • For ζ = 0.01, the likelihood score guides states toward the measurement at 0.2 < t < 0.5, but the time-zero estimate remains indistinguishable from the prior.At ζ ∈ (0.09, 0.11), the result better resembles the finite-sample posterior but is skewed toward the positive-mean component; above 0.2, the mode is placed correctly but its covariance is underestimated.Source: Figure 6 · p. 47
  • For all ζ < 0.2, modified DPS samples prior atoms inconsistent with the measurement, and no ζ accurately captures covariance at intermediate times.The covariance changes with the noise schedule, whereas ζ is fixed across time.Source: Figure 8 · p. 48
  • For ζ < 0.2, modified DPS samples inconsistent prior atoms, and no ζ both yields accurate intermediate-time posterior mean and samples only the posterior Dirac.The weight on inconsistent atoms decreases as ζ increases.Source: Figure 10 · p. 49
  • For the nonlinear case with y = +1.09, the true posterior has two majority modes and one minority mode, but unmodified DPS captures only one majority mode.The reported cause is bias from an incorrect posterior-mean estimate at intermediate timesteps.Source: E ζ-DPS tuning · p. 50
  • For the nonlinear case with y = +6.25, unmodified DPS hallucinates prior modes and treats the effectively Dirac posterior atom as a rare event.Source: E ζ-DPS tuning · p. 51
  • For a symmetric trimodal Gaussian-mixture prior with a cubic forward model, the asymmetric bimodal posterior’s minority mode is missed by unmodified DPS.Source: Figure 15 · p. 52

Related papers

Loading 2605.30330v1…