PaperlayerView exact revision on arXiv

Source-linked AI summary

FB-GDM: Fully-Bayesian Guided Diffusion Models for High-Dimensional Linear Inverse Problems via Unsupervised Variational Inference

Gatien Séguy, Thomas Rodet

arXiv:2609.29216v1cs.LGcs.AIstat.ML

TL;DR

DPS and ΠGDM require task-specific tuning, often unavailable when ground truth or noise levels are unknown. FB-GDM derives a closed-form conditional score and infers its two guidance precisions from the observation. It stays close to a ΠGDM oracle across changes in operator, noise, and image distribution, without task-specific tuning.

  • Problem

    DPS and ΠGDM rely on task-specific hyperparameters, but calibration is unavailable when ground truth is absent and noise levels may be unknown.

    Source: §I · p. 1

  • Method

    FB-GDM derives a closed-form conditional score and estimates the denoising and observation-likelihood precisions from the data at each reverse step.

    Source: §I · p. 2, A. General conditional score · p. 9

  • Results

    FB-GDM matches ground-truth ΠGDM oracles in distribution and stays close to a per-problem oracle across changes in operator, noise level, and image distribution.

    Source: §V · p. 9

  • Takeaways & Limitations

    Its automatic balance between data and prior limits hallucinations on images outside the training distribution, at a cost comparable to one ΠGDM run.

    Source: §I · p. 2, §V · p. 9

Abstract

from arXiv · show

Diffusion models are powerful priors for linear inverse problems, but the reference guidance methods, Diffusion Posterior Sampling (DPS) and Pseudoinverse-Guided Diffusion Models ($Π$GDM), rely on scalar hyperparameters tuned per task, usually against the ground truth. We introduce FB-GDM, a fully-Bayesian guided diffusion method that removes this calibration step. Starting from the Gaussian approximation of $Π$GDM, we derive a closed-form conditional score that depends on two precision parameters (inverse variances), one associated with the denoising approximation and one with the observation likelihood, and treat them as latent variables inferred by variational inference at each reverse step. A separable factorization makes each update scale linearly with the number of pixels, so the inference stays tractable at full image resolution, at a cost comparable to one $Π$GDM run. FB-GDM requires neither the noise level nor the ground truth: its only inputs are the observation and the forward operator. Experiments on CelebA-HQ inverse problems establish two results. (i) The precision parameters, inferred from the observation alone, allow FB-GDM to outperform $Π$GDM at its nominal setting, even when the latter is given the true noise level, by up to 14 dB depending on the operator, and to match the ground-truth-calibrated $Π$GDM oracle within 0.1 dB. (ii) FB-GDM is robust when the forward operator, the noise level, or the image distribution changes: it stays close to a per-problem $Π$GDM oracle throughout and does not exhibit the hallucinations observed with DPS, whereas DPS substantially degrades at a fixed scale and $Π$GDM stays competitive only if it is re-tuned against the ground truth for each new problem. When the prior is applied to images outside its training set, this re-balancing between data and prior keeps FB-GDM faithful where a fixed face-prior guidance can otherwise hallucinate.

I. INTRODUCTION

Diffusion priors support linear inverse problems, but reference guidance methods require task-specific hyperparameters that are difficult to calibrate without ground truth. FB-GDM infers its guidance precisions from the observation and remains robust across changes in operator, noise, and image distribution.

I. INTRODUCTION

  • Diffusion priors are used for linear inverse problems, where the central challenge is automatically balancing information from the likelihood and learned prior.Conditional methods require a score that has no exact expression, so methods including DPS and ΠGDM approximate it.Source: §I · p. 1
  • DPS and ΠGDM depend on task-specific scalar hyperparameters, with no calibration procedure available when ground truth is absent.The settings vary with task, noise level, and data.Source: §I · p. 1
  • When ground truth and noise level are unavailable, exact prior evaluation through the probability-flow ODE is robust out of distribution but restricted to low-dimensional problems by its computational cost.Source: §I · p. 1
  • FB-GDM estimates two guidance precisions from the observation, removing task-specific tuning and requiring only the observation y and forward operator A.The precisions correspond to the Gaussian denoising approximation and the Gaussian guidance likelihood.Source: §I · p. 2
  • A separable variational update scales linearly with pixel count, making one FB-GDM run comparable in cost to one fixed-parameter ΠGDM run.Source: §I · p. 2
  • FB-GDM's automatic balance between data and prior limits hallucinations on images outside the training distribution..Source: §I · p. 2
  • The paper reports that FB-GDM stays close to a ground-truth-tuned ΠGDM oracle in distribution and remains robust to changes in operator, noise level, and image distribution.Source: §I · p. 2

C. Conditional diffusion models

Conditional diffusion generates from the posterior by replacing the unconditional score with a conditional score. Because its measurement term is intractable, DPS and ΠGDM approximate it using different assumptions and task-dependent settings.

C. Conditional diffusion models

1) DPS:

  • DPS approximates p_t(x0 | x_t) by a point mass at the Tweedie estimate, and its scale factor ζ′ is tuned per task and noise level.Source: 1) DPS · p. 3
  • ΠGDM uses Gaussian approximations for denoising and the observation likelihood, with variances treated as hyperparameters.Its guidance weights the measurement term with an adaptive VP factor and scalar w, which is chosen per task.Source: 1) DPS · p. 3
  • ΠGDM's measurement term has pseudo-inverse form, and its variance and scalar weight are selected per task.Source: 1) DPS · p. 3

III. METHOD

The method computes the conditional score in closed form and estimates its uncertainty parameters by variational Bayesian inference at each reverse step.

III. METHOD

  • At each reverse step, FB-GDM estimates uncertainty parameters by variational Bayesian inference to compute the conditional score.Source: §III · p. 3
  • Inference is performed at fixed x_t, which conditions the calculation but is not itself inferred.Source: §III · p. 3
  • The inferred precisions set the balance between measurements and the Tweedie prediction in the posterior mean, while the score correction retains its deterministic VP-dependent scale.Source: §III · p. 3

A. Closed form conditional score

The conditional score is obtained from a Gaussian denoising approximation and a Gaussian observation likelihood. FB-GDM makes their precisions latent and infers them at each reverse step.

A. Closed form conditional score

B. Hierarchical Bayesian model

C. Variational Bayesian inference

The method approximates the intractable posterior over the target and precisions with a variational distribution that maximizes a lower bound on the log evidence. Its factorization enables coordinate updates that preserve the chosen distribution families.

C. Variational Bayesian inference

  • The variational density q(v) maximizes free energy, a lower bound on log p_t(y | x_t), equivalently minimizing KL divergence to the posterior.The posterior normalization requires an intractable integral over v.Source: C. Variational Bayesian inference · pp. 4–5
  • The approximation factorizes into a Gaussian block for x_0 and Gamma blocks for the two precisions γ_r and γ_b.The precision blocks use shape-rate Gamma distributions.Source: C. Variational Bayesian inference · p. 5
  • Coordinate Ascent Variational Inference updates each block in turn while holding the others fixed, and conjugacy keeps the blocks within their chosen families.The x_0 update follows from a quadratic expected log joint; the precision updates remain Gamma-distributed.Source: C. Variational Bayesian inference · p. 5
  • Alternating the three updates increases the free energy at each step.Source: C. Variational Bayesian inference · p. 5

D. Separable factorization and fast updates

A full covariance update would require an infeasible n×n matrix inversion at image scale. The method instead factorizes the target distribution into independent components and updates them simultaneously.

D. Separable factorization and fast updates

E. Algorithm

Each FB-GDM reverse step reuses the ΠGDM sampler structure and refines its initialization through variational updates. The setup uses a pretrained CelebA-HQ DDPM and supplies only the observation and forward operator to FB-GDM.

E. Algorithm

  • FB-GDM starts from ΠGDM’s initialization heuristic, then refines it; all reverse-step components not explicitly changed remain identical to ΠGDM.Source: E. Algorithm · p. 5

1) Setup:

  • Experiments use an unconditional DDPM pretrained on CelebA-HQ at 256×256 without finetuning, with an ancestral DDPM sampler of T=1000 steps.Except in the noise-robustness study, measurement SNR is fixed at 20 dB; variational inference uses K=100 inner iterations.Source: 1) Setup · p. 5
  • At 20 dB, Fig. 2 orders Gaussian-deblurring examples as ground truth, observation, ΠGDM variants, DPS, and FB-GDM, with PSNR below each reconstruction.Source: Figure 2 · p. 6
  • The method receives only y and A, without the noise level, ground truth, or experiment-specific hyperparameter adjustment.Source: 1) Setup · p. 6
  • Reconstruction quality is measured with PSNR and SSIM.Source: 1) Setup · p. 6

2) Comparison with the state of the art:

On CelebA-HQ Gaussian deblurring, FB-GDM leads the compared methods without noise-level or ground-truth calibration and nearly matches the calibrated ΠGDM oracle. Across changed operators, it stays near per-operator oracles without retuning, while fixed-scale methods can degrade substantially.

B. First experiment: Gaussian deblurring

C. Second experiment: robustness to the operator

  • 26.76–28.86 dB: FB-GDM remains within this PSNR range across three operators without ground truth or operator-specific retuning.The tests change only the forward operator at 20 dB: Gaussian blur, a uniform kernel, and ×4 super-resolution.Source: C. Second experiment: robustness to… · p. 7
  • For ×4 super-resolution, FB-GDM stays within 0.4 dB of both ΠGDM oracles, while tuned DPS reaches 27.82 dB and 0.787 SSIM and nominal ΠGDM reaches 20.53 dB.For Gaussian blur, FB-GDM attains the best PSNR and is within 0.01 dB of ΠGDM⋆; DPS is comparable at its recommended scale.Source: C. Second experiment: robustness to… · p. 7
  • On the uniform kernel, FB-GDM scores 26.76 dB and 0.752 SSIM versus ΠGDM⋆ at 27.48 dB and 0.769 SSIM, while nominal ΠGDM falls to 17.02 dB and DPS to 19.53 dB and hallucinates a face.The oracle's gains over FB-GDM are 0.72 dB and 0.017 SSIM, but required more than 130 ground-truth evaluations to recalibrate both weight and variance.Source: C. Second experiment: robustness to… · pp. 7–8, Table II · p. 7, Table III · p. 7

D. Third experiment: robustness to the noise level

Across changing noise levels and image distributions, FB-GDM stays close to ground-truth-calibrated ΠGDM while avoiding task-specific tuning. Its cost is comparable to one ΠGDM pass, and its inferred precisions adapt to each observation.

D. Third experiment: robustness to the noise level

E. Fourth experiment: out-of-distribution generalization and hallucinations

  • Under an unseen ImageNet distribution, FB-GDM preserves scene structure without face-like artifacts, while DPS hallucinates structures unsupported by measurements.The experiment applies a CelebA-HQ face prior with a 9 × 9 averaging kernel at SNR=20 dB; transferred ΠGDM⋆ is also visually consistent but relies on in-distribution calibration.Source: E. Fourth experiment:… · p. 8, Figure 5 · p. 9

F. Computational cost

  • A single FB-GDM pass replaces ΠGDM calibration searches that require 6 or 24 generations and must be repeated when the operator or noise level changes.The reported FB-GDM run takes 165s, versus 122s for ΠGDM; its 35% per-run overhead is attributed to K=100 variational updates.Source: F. Computational cost · p. 8, Table IV · p. 9

V. CONCLUSION AND DISCUSSION

  • FB-GDM matches ground-truth ΠGDM oracles in distribution and remains close to per-problem oracles across operator, noise, and image-distribution changes.The inferred precisions adapt to each observation without supervision; the method requires only the observation and forward operator, with cost comparable to one ΠGDM pass.Source: §V · p. 9
  • Future work will test FB-GDM on linear or locally linearized scientific imaging problems, including accelerated MRI, X-ray CT, and astronomical reconstruction.These settings are identified as ones where ground-truth images are typically unavailable.Source: §V · p. 9

APPENDIX A PROOF OF THE CLOSED-FORM CONDITIONAL SCORE

The proof derives the conditional score by combining the Gaussian denoising approximation with the observation likelihood, then differentiating the joint distribution and applying Bayes’ rule.

A. General conditional score

  • The derivation targets the conditional score ∇xt log p(xt | y) under the model assumptions.The proof begins by setting up the quantities used to derive this score.Source: A. General conditional score · p. 9
  • The working model assumes a Gaussian observation likelihood and a Gaussian denoising approximation.The likelihood uses working variance σ2_b.Source: A. General conditional score · p. 9
  • The Markov chain xt → x0 → y gives conditional independence of y and xt given x0.Equivalently, p(y | x0, xt) = p(y | x0).Source: A. General conditional score · p. 9
  • Using that conditional independence, the proof writes the joint distribution in terms of the prior and likelihood factors.The joint-distribution expression includes an integral over x0.Source: A. General conditional score · p. 10
  • Differentiating the joint distribution with respect to xt acts only on p(x0 | xt) and p(xt).The proof applies gradient identities to these terms.Source: A. General conditional score · pp. 9–10
  • Bayes’ rule relates p(y | x0) p(x0 | xt) to p(y | xt) p(x0 | xt, y), enabling the conditional posterior to enter the score derivation.The proof then substitutes this identity and divides by the joint-distribution expression to establish the result.Source: A. General conditional score · p. 10, Conclusion · p. 10

B. Gaussian closed form

Under Gaussian assumptions, the posterior over x0 given xt and y is Gaussian, yielding a closed-form posterior mean used in the conditional score. Fixing the precisions connects this formulation to ΠGDM.

B. Gaussian closed form

  • Under the Gaussian assumptions, p(x0 | xt, y) is Gaussian and its expectation has the closed form µpost.The posterior mean is given by equation (20).Source: B. Gaussian closed form · p. 10
  • The conditional score uses the closed-form posterior expectation together with the prior score ∇xt log p(xt).The score expression is identified as equation (54).Source: B. Gaussian closed form · p. 10
  • The derivation recovers Corollary 1 from the Gaussian closed form.Source: B. Gaussian closed form · p. 10

APPENDIX B CONNECTION WITH ΠGDM

  • To compare FB-GDM with ΠGDM, the appendix replaces the Jeffreys priors with Dirac priors centered on ΠGDM settings.This fixes the precisions rather than inferring them.Source: Appendix B · p. 10
  • Under those Dirac priors, Proposition 3 gives the conditional score of FB-GDM.Source: Appendix B · p. 10
  • The measurement term’s scale is fixed at 1 for FB-GDM, whereas ΠGDM weights it by a task-tuned factor.The stated ΠGDM factor includes w and the diffusion-dependent terms.Source: Appendix B · p. 10
  • Rewriting µpost with the matrix inversion lemma yields the measurement term used in the comparison with ΠGDM guidance.Source: Appendix B · p. 10
Loading 2609.29216v1…