PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Diffusion Models for Inverse Problems

Giannis Daras, Hyungjin Chung, Chieh-Hsin Lai, Yuki Mitsufuji, Jong Chul Ye, Peyman Milanfar, Alexandros G. Dimakis, Mauricio Delbracio

arXiv:2410.00083v1cs.LGcs.AIcs.CV

TL;DR

The survey addresses how pre-trained diffusion models can serve as unsupervised priors for inverse problems without retraining, despite posterior and measurement-score intractability. It taxonomizes methods, unifies their common measurement-consistency updates, and discusses latent-space and computational challenges. The surveyed families include explicit approximations, variational inference, CSGM-type frameworks, and asymptotically exact methods, with exact sampling requiring increased computation.

  • Problem

    Using diffusion models as inverse-problem priors is difficult because the posterior and time-dependent measurement-matching term can be intractable, while retrained conditional models are application-specific.

  • Method

    The survey taxonomizes pre-trained unconditional diffusion-prior methods, relates them through a common mathematical language, and examines latent diffusion challenges.

  • Results

    The survey identifies four method families: explicit measurement-score approximations, variational inference, CSGM-type frameworks, and asymptotically exact methods.

  • Takeaways & Limitations

    The methods share measurement-consistency errors that are lifted to image space for gradient updates, while their approaches trade off approximation, optimization, exactness, and computation.

  • Takeaways & Limitations

    CSGM-type frameworks are computationally demanding because probability-flow ODE inversion requires several diffusion-model calls.

Abstract

from arXiv · show

Diffusion models have become increasingly popular for generative modeling due to their ability to generate high-quality samples. This has unlocked exciting new possibilities for solving inverse problems, especially in image restoration and reconstruction, by treating diffusion models as unsupervised priors. This survey provides a comprehensive overview of methods that utilize pre-trained diffusion models to solve inverse problems without requiring further training. We introduce taxonomies to categorize these methods based on both the problems they address and the techniques they employ. We analyze the connections between different approaches, offering insights into their practical implementation and highlighting important considerations. We further discuss specific challenges and potential solutions associated with using latent diffusion models for inverse problems. This work aims to be a valuable resource for those interested in learning about the intersection of diffusion models and inverse problems.

1 Introduction

Inverse problems recover unknown signals from corrupted measurements across diverse scientific domains, but information loss makes perfect recovery impossible. This survey frames diffusion priors as a flexible unsupervised approach for generating plausible posterior reconstructions across inverse problems.

  • Problem setting: Inverse problems recover an unknown signal from measurements produced by a corruption model, spanning imaging, weather, oceanography, audio, and medical applications.
  • Problem setting: Denoising, inpainting, compressed sensing, convolutions, cryo-EM, deconvolution, phase retrieval, and compression removal instantiate linear, blind, or nonlinear inverse problems.
  • Problem setting: Because inverse problems lose information, perfect recovery is impossible; MAP and MMSE objectives therefore encode different notions of a desirable reconstruction.
  • Existing approaches: Denoisers and deep generative models can serve as learned priors within optimization or Bayesian frameworks for reconstructing plausible signals from measurements.
  • Diffusion priors: Diffusion-prior methods target posterior reconstruction without retraining for each application, unlike conditional diffusion models that often require application-specific retraining.
  • Diffusion priors: Unsupervised inversion requires estimating the conditional log-likelihood gradient, combining a score function with a measurement-matching term.

2 Background

Diffusion models transform data into noise through a forward process and recover samples by reversing it with score-based dynamics. The survey extends this framework to inverse problems, emphasizing conditional scores, denoising-based score estimation, latent-space diffusion, and pretrained unconditional models.

  • Forward and reverse processes: Diffusion models gradually add Gaussian noise to data, producing a time-indexed distribution that approaches Gaussian noise.The process can be formulated discretely or as a continuous-time stochastic differential equation with drift and diffusion coefficients.
  • Forward and reverse processes: Reverse-time SDE sampling recovers the data distribution when initialized from the terminal Gaussian distribution and supplied with the score function.A deterministic probability-flow formulation can produce the same marginal distributions and be solved with numerical ODE methods.
  • Score estimation: Tweedie’s formula identifies the score through E[X0|Xt = xt], which is the MMSE denoiser of the clean image from a noisy observation.A neural network can learn this denoiser across noise levels by conditioning on t or σt.
  • Latent diffusion: Latent diffusion improves efficiency by applying diffusion in the latent space of a pretrained autoencoder and decoding the generated clean latent back into an image.The survey notes that inverse problems with latent diffusion require special treatment because the measurement process must be handled in latent space.
  • Diffusion for inverse problems: Conditional diffusion targets p0(·|y) by estimating the conditional score, but direct conditional training requires the degradation operator during training and retraining when it changes.This motivates methods that reuse pretrained unconditional diffusion models without further training, including approaches based on score decomposition and measurement likelihoods.
  • Measurement-only learning: Ambient Diffusion and related methods learn priors from corrupted or noisy measurements, with Consistent Diffusion Meets Tweedie providing guarantees for sampling from the target distribution using noisy data.Ambient DPS reports that corrupted-data priors can outperform clean-data priors in usefulness for inverse problems at high corruption.

3 Reconstruction Algorithms

The survey organizes diffusion-based reconstruction methods by their approximation strategy, solvable inverse-problem type, and optimization technique. It explains representative approaches for measurement matching, denoiser-based sampling, projections, and nonlinear posterior guidance.

  • Method families: Four method families are identified: explicit score approximations, variational inference, CSGM-type methods, and asymptotically exact methods.Explicit methods approximate the measurement-matching score; variational methods use tractable posterior approximations; CSGM-type methods optimize diffusion latents; asymptotically exact methods target posterior sampling with MCMC or SMC.
  • Optimization techniques: Methods enforce measurement consistency through gradients, projections, sampling, or other optimization techniques.Gradient methods update x_t using a consistency gradient, projection methods move estimates toward the measurement subspace, and sampling methods propagate particles using proposal distributions.
  • Explicit approximations: Explicit-approximation methods replace the measurement-matching score ∇x_t log p(y|X_t = x_t) with a tractable expression analyzed under a common framework.The survey often gives formulas for linear inverse problems and uses a template to clarify differences between methods.
  • Denoiser-based sampling: Denoiser-based sampling uses Tweedie’s formula to interpret denoiser outputs as score approximations and then applies stochastic gradient ascent to generate high-probability prior samples.The method uses a denoiser residual, a step size, and newly introduced Gaussian noise.
  • Linear inverse problems: For linear inverse problems, Score ALD guides diffusion estimates using the lifted measurement error y − Ax_t, while Score-SDE adds stochasticity by noising measurements.Score-SDE is categorized as projection because its gradient step yields a noisy projection; RePaint is its inpainting special case.
  • Nonlinear inverse problems: DPS extends explicit measurement guidance to nonlinear inverse problems by approximating the likelihood at x_t with the likelihood evaluated at the posterior mean estimate E[X_0|X_t = x_t].In practice, DPS reweights guidance using a step size inversely proportional to the measurement-error norm.

GibbsDDRM.

GibbsDDRM extends DDRM to blind linear inverse problems by sampling the unknown operator parameters alongside the reconstruction. The section also relates posterior-mean approximations to proximal, projection, and likelihood-based updates.

  • GibbsDDRM: GibbsDDRM handles the blind linear model y = A_ϕx + σ_yz, where the degradation operator is parameterized by unknown ϕ.The operator is represented through a singular-value decomposition involving U_ϕ, Σ_ϕ, and V_ϕ.
  • GibbsDDRM: At each diffusion step, GibbsDDRM replaces DDRM’s posterior mean with a ϕ-dependent estimate and samples ϕ from p(ϕ|x_t:T, y).The sampled parameter is updated for several iterations using Langevin dynamics.
  • Posterior approximations: The survey connects posterior-mean updates with likelihood approximations through the relationship between conditional scores and posterior means.This connection allows posterior-based approximations to be rewritten in terms of posterior scores.
  • DDNM family: DDNM approximates the conditional posterior mean by applying data-consistency projections to E[X_0|X_t = x_t], using a Moore–Penrose pseudo-inverse for SVD-decomposable operators.With noisy measurements, the method uses soft updates rather than the noiseless projection.
  • Proximal methods: DDS and DiffPIR both use proximal updates for the conditional posterior mean, but differ in how they solve the optimization and schedule λ_t.DDS uses a few conjugate-gradient steps with constant λ_t, whereas DiffPIR uses a closed-form solution and an SNR-proportional schedule.

3.2 Variational Inference

Variational-inference methods approximate inverse-problem posteriors with tractable distributions and optimize objectives combining diffusion-prior terms with measurement consistency. The section covers image-only, blind, likelihood-based, and sampling-based formulations, including efficiency trade-offs.

  • RED-diff: RED-diff approximates the true posterior with q := N(µ, σ2I_n) by minimizing their KL divergence.Its variational objective combines reconstruction and score-matching terms involving diffusion denoisers across timesteps.
  • Blind RED-diff: Blind RED-diff jointly estimates the latent image and unknown forward-model parameters using variational inference.The optimization alternates stochastic updates of the image x0 and parameters γ.
  • Blind RED-diff: The blind formulation assumes conditional independence between x0 and γ given y and requires a specific prior form for γ.Its KL objective includes image-prior, parameter-prior, and negative-log-likelihood terms.
  • Score prior: Score-prior methods use normalizing flows to represent variational distributions and compute diffusion-prior terms through the probability-flow ODE.This avoids approximation error but requires costly repeated evaluations and separate optimization for each measurement y.
  • Efficient score prior: A denoising-likelihood surrogate reduces the required evaluations from hundreds to thousands of NFEs to a single NFE, improving efficiency and scalability.The surrogate value is smaller when the diffusion denoiser incurs a smaller denoising loss.
  • Asymptotically exact methods: Asymptotically exact methods trade computation for approximation error and converge to the true posterior as network evaluations increase, under stated assumptions.Their guarantees generally require infinite computation, leaving practical scalability unresolved.

3.4 CSGM-Type methods

CSGM-type methods optimize latent noise variables whose deterministic diffusion sampling produces reconstructions consistent with measurements. Their main practical challenges are repeated diffusion evaluations, backpropagation memory, and latent-to-pixel nonlinearities, with few-step or one-step models offering efficiency improvements.

  • Latent optimization: CSGM-type diffusion methods optimize the latent noise input to a deterministic sampler rather than directly optimizing image pixels.The generated estimate is the sampler solution initialized at the optimized latent z.
  • Latent optimization: Unlike GANs and flows, diffusion samplers require multiple NFEs to solve the generative SDE or ODE.This increases the computational cost of latent optimization.
  • Computational trade-offs: Backpropagation through time can require exploding memory, so DMPlug and SHRED use few-step sampling approximations.The cited examples use 3 steps for DMPlug and 10 for SHRED.
  • Computational trade-offs: Consistency Models can replace ODE sampling with one-step generation to reduce computational requirements.They solve the Probability Flow ODE in one step after distillation.
  • Intermediate latent optimization: Score-ILO regularizes intermediate latent variables with diffusion models to prevent them from leaving the manifold of realistic images.This extends CSGM-style optimization beyond the original input latent space.
  • Latent diffusion challenges: Latent diffusion makes linear inverse problems effectively nonlinear because diffusion occurs in latent space while measurements remain in pixel space.Guidance can project measurements into latent space or project latents into pixel space through the decoder.
  • Latent diffusion challenges: Repeated decoding is computationally burdensome, while the non-one-to-one decoding-encoding map makes meaningful latent guidance difficult.These issues directly constrain inverse-problem use of latent diffusion models.
  • Latent DPS and refinements: Latent DPS performs one-step latent denoising and measures decoded-latent agreement with the observations.Its approximation can move toward any latent matching the measurements, rather than the specific latent favored by the score.

3.6 MPGD He et al. [158]

MPGD addresses costly measurement-alignment gradients in diffusion-based inverse problems by updating a clean estimate derived through Tweedie’s formula. The surrounding methods also explore text conditioning and additional stochastic-gradient terms for latent diffusion guidance.

  • MPGD: MPGD bypasses the direct gradient through the diffusion model by updating with the gradient of the clean estimate x0|t.The passage attributes theoretical guarantees to this alternative update.
  • Text conditioning: Text conditioning in latent diffusion can be optimized during inversion, as in P2L, to select embeddings that improve reconstructive performance.The optimized embedding is used at each timestep during sampling.
  • Text conditioning: P2L can additionally decode, optimize in pixel space, and re-encode to leverage the VAE prior.This supplements online text-embedding optimization with pixel-space optimization.
  • Text conditioning: Text conditioning can also provide an extra guiding signal for steering reconstructions toward a specific mode.The section notes that ordinary conditional diffusion and naive classifier-free guidance may provide insufficient guidance in practice.
  • Covariance-aware guidance: STSL incorporates covariance information from the reverse diffusion distribution by using a fidelity loss and multiple stochastic-gradient updates per timestep.The trace in the covariance-related term can be approximated with Gaussian or Rademacher samples.

4 Thoughts from the authors

The survey does not identify a universally best diffusion-based inverse-problem method. It instead unifies existing approaches, highlights measurement-score and lifting choices, and points to computational trade-offs and benchmarking needs.

  • No conclusive answer is available for which diffusion-based inverse-problem approach works best.
  • A common mathematical language expresses methods such as DPS and DDRM through explicit approximations to the measurement score.
  • Existing methods differ in whether they compute measurement-consistency errors from noisy or clean measurements.
  • Methods increasingly use complex lifting matrices, with some approximations requiring additional computation, such as Moments Matching.
  • The survey argues that standardized benchmarks are needed to compare computational performance trade-offs across methods.
  • Variational methods may fail to capture the complexity of the true posterior because they estimate a simpler distribution.
  • CSGM-type frameworks are computationally demanding because Probability Flow ODE inversion requires multiple diffusion-model calls.
  • Asymptotically exact methods provide guarantees only with infinite computation, leaving practical scalability unresolved.

5 Conclusion

The survey organizes diffusion-prior methods for inverse problems into four families and explains their shared challenge: the posterior distribution is intractable. It also characterizes trade-offs among these frameworks.

  • The survey identifies four families: explicit measurement-score approximations, variational inference, CSGM-type frameworks, and asymptotically exact sampling methods.
  • These frameworks address the fundamental problem that the inverse-problem posterior distribution is intractable.
  • The survey unifies seemingly different approaches and explains their trade-offs, including increased computation for asymptotically exact sampling.
  • The authors position the survey as a reference point for the field of diffusion models for inverse problems.

A Proofs

The proof establishes that the mean-squared-error-minimizing function hθ(xt, t) approximates the conditional expectation E[x0|xt].

  • Lemma A.1 considers random variables X0 and Xt and a parameterized function hθ(xt, t).
  • The function minimizing mean squared error with respect to x0 is the one that best approximates E[x0|xt].
  • The proof proceeds by evaluating a second term and then concludes.

A.1 Tweedie’s Formula

Lemma A.2 introduces Tweedie’s Formula.

  • Lemma A.2 is titled Tweedie’s Formula and begins by specifying its assumptions or definitions.

A.2 Denoising Score Matching

Denoising score matching connects clean-image prediction and score estimation through MMSE and Tweedie’s formula. Theorem A.3 states that the two corresponding objectives share the same minimizer.

  • A.2 Denoising Score Matching: MMSE interpretation and Tweedie’s formula allow the score function to be approximated by predicting the clean image from a corrupted observation.The clean-image predictor is trained with supervised learning and converted into a score-approximating model at inference.
  • A.2 Denoising Score Matching: The clean-image training procedure is known as x0-prediction loss.
  • A.2 Denoising Score Matching: Direct score training provides an alternative equivalent formulation to clean-image prediction training.
  • A.2 Denoising Score Matching: Under the theorem’s stated support and differentiability assumptions, objectives J1 and J2 have the same minimizer.The section includes a completeness proof and begins by reducing the argument to sufficient identities, starting with the second term.

A.3 Jacobian of the score

This section introduces a lemma concerning the Jacobian of the score function and proceeds to analyze that Jacobian.

  • A.3 Jacobian of the score: Lemma A.4 addresses the Jacobian of the score function.
  • A.3 Jacobian of the score: The section proceeds by analyzing the score-function Jacobian.
Loading 2410.00083v1…