PaperlayerView exact revision on arXiv

Source-linked AI summary

Getting Your Guidance Weights Right in diffusion and flow-matching posterior sampling

Liam Moroy, Jean-François Giovannelli, Yoann Altmann, Steve McLaughlin, Frédéric Champagnat, Guillaume Bourmaud

arXiv:2610.03503v1cs.LG

TL;DR

Training-free posterior sampling uses pretrained unconditional models, but guidance weights are often left to heuristics and must be retuned when measurement settings change. This paper calibrates time-dependent weights offline with least-squares methods and SIMCA, reporting improved results across diffusion and flow-matching methods and faster diffusion sampling.

  • Problem

    Guidance weights balance prior information from an unconditional network with measurement consistency, but are often tuned heuristically and require retuning when measurement settings change.

    Source: §1 · p. 2

  • Method

    The paper uses per-time-step least squares and greedy SIMCA to calibrate guidance weights offline, with SIMCA using one minibatch of sampling trajectories.

    Source: §1 · p. 2

  • Results

    SIMCA matches or surpasses state-of-the-art training-free methods across multiple inverse problems and datasets using diffusion and flow-matching backbones.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Diffusion samplers reduce sampling steps from 1000 to 50 without significant degradation in reconstruction quality.

    Source: §1 · p. 2

    Simulation-free calibration does not guarantee close-to-posterior sampling trajectories because the guidance is limited to two parameters and its components are not necessarily optimal.

    Source: §4.1 · pp. 4–5

Abstract

from arXiv · show

Training-free posterior sampling methods, also known as Plug-and-Play methods, leverage pretrained unconditional diffusion or flow-matching models to solve inverse problems. Most existing approaches rely on guidance weights to balance, at each time step, prior information from the unconditional score or velocity network with measurement consistency, yet the tuning of these weights is often not discussed and is largely left to heuristics. We introduce a simple and principled offline strategy for automatically tuning these guidance weights. Our key observation is that, at each time step, the conditional denoising score-matching objective for diffusion models, or the conditional flow-matching objective for flow-matching models, is a least-squares objective. Therefore, when the conditional prediction is expressed as a weighted sum of the unconditional network output and a measurement-guidance term, optimizing over these weights reduces to a two-dimensional linear least-squares problem. The resulting time-dependent guidance weights can be optimized offline for a given measurement operator, noise level and sampler at the cost of a single minibatch of sampling trajectories, without retraining or fine-tuning the pretrained generative model. Instantiated with the standard Tweedie-based measurement-consistency term, our approach improves posterior sampling and achieves state-of-the-art reconstruction performance across diffusion- and flow-matching-based methods. Moreover, the optimized guidance weights enable diffusion samplers to reduce the number of sampling steps from 1000 to 50 with no significant degradation in reconstruction quality. Code will be made available.

1 INTRODUCTION

Training-free posterior sampling avoids retraining for new measurement settings, but its guidance weights are commonly tuned heuristically. The paper proposes offline calibration and reports improved reconstruction performance and faster diffusion sampling.

1 INTRODUCTION

  • Training-free methods use unconditionally pretrained models to approximate posterior samples without retraining for each measurement operator or noise level.The paper motivates this approach because acquisition settings vary at inference time and model training is computationally expensive.Source: §1 · p. 1
  • Guidance weights balance the unconditional network’s prior information against measurement consistency, but are often tuned heuristically.Changing sensor numbers, locations, or acquisition settings can require time-consuming weight retuning.Source: §1 · p. 2
  • At each time step, the conditional diffusion or flow-matching objective yields a two-dimensional linear least-squares problem for the guidance weights.The solution provides a simulation-free strategy for calibrating time-dependent weights.Source: §1 · p. 2
  • SIMCA greedily calibrates each time step while fixing weights calibrated at earlier steps, using one minibatch of sampling trajectories.It is proposed because simulation-free calibration alone does not guarantee improved posterior trajectories.Source: §1 · p. 2
  • SIMCA matches or surpasses state-of-the-art training-free methods across multiple inverse problems and datasets using diffusion and flow-matching backbones.With the standard Tweedie-based measurement-consistency term, reported gains span both model families.Source: §1 · p. 2
  • Diffusion samplers reduce sampling steps from 1000 to 50 without significant degradation in reconstruction quality.This is a 20× reduction in inference cost.Source: §1 · p. 2

2 BACKGROUND ON TRAINING-FREE POSTERIOR SAMPLING

The background describes diffusion and Gaussian flow-matching models as transporting noise toward data, then frames inverse sampling as approximating a conditional field using an unconditional model and measurement guidance. The paper focuses on an additive guidance term and considers several sampling dynamics.

Unconditional diffusion models and (Gaussian) flow-matching

  • Diffusion and Gaussian flow-matching models generate data by transporting Gaussian noise along marginals formed by interpolating noise and data.The discussion focuses on independent coupling for establishing equivalence between diffusion and Gaussian flow matching, while experiments also use velocity networks trained with independent and OT couplings.Source: §2 · p. 2
  • The noise schedule runs from approximately data at t=0 to Gaussian noise at t=1, allowing sampling to start from pure noise.The stated schedule has α0≈1, σ0≈0, α1≈0, and σ1≈1.Source: §2 · p. 2
  • Diffusion models regress a score with denoising score matching, whereas flow-matching models regress a velocity with conditional flow matching.The unified notation uses the time derivatives of the interpolation coefficients.Source: §2 · p. 3
  • Sampling integrates the corresponding dynamics backward on a time grid, with the number of steps trading sample quality against computational cost.The paper considers DDPM, reverse-SDE Euler-Maruyama, and the Euler step of the flow ODE.Source: §2 · p. 3

Inverse problems and Training-free posterior sampling

  • An inverse problem seeks an unknown signal from a degraded observation, with the paper considering a known linear measurement operator.Training-free posterior sampling approximates the conditional field using an unconditionally pretrained network and a measurement-consistency mechanism.Source: §2 · p. 3

Measurement-consistency mechanism

  • The paper approximates the conditional field as a weighted sum of the unconditional network output and a measurement-guidance term.Its experiments use the Tweedie-based term, proportional to the gradient of the squared measurement residual evaluated at the Tweedie estimate.Source: §2 · p. 3

3 RELATED WORK

Prior methods set measurement guidance heuristically or derive its scale from an assumed posterior covariance, while fixing the pretrained-field weight at one. Other adaptive approaches use heuristic schedules or a single coupled parameter rather than fitting weights to a measurable conditional-field criterion.

Guidance weights in training-free posterior sampling

  • Existing additive samplers use heuristic measurement weights or derive them from an assumed posterior covariance, and all fix the pretrained-field weight at one.The covariance-based methods also change the measurement term and its effective scale; the paper’s ablations suggest the scale accounts for much of their benefit.Source: §3 · p. 4

Adaptive guidance

  • Adaptive guidance has also used heuristic time-varying weights or a single coupled parameter estimated online against an intractable posterior score.The paper instead describes minimizing a measurable conditional-field approximation error in closed form; its Monte Carlo-moment solve follows Analytic-DPM’s approach to unconditional reverse variance.Source: §3 · p. 4

4 METHOD

The method calibrates time-dependent guidance weights by reducing their estimation to small linear least-squares problems. Its simulation-based strategy accounts for the approximate posterior encountered during sampling and requires only one minibatch of trajectories.

4 METHOD

  • The strategy targets efficient offline calibration of time-dependent weights for a specified measurement operator and noise level, avoiding iterative optimizers that would add calibration cost.It uses a pretrained unconditional score or velocity network and a user-specified measurement-consistency term.Source: §4 · p. 4

4.1 SIMULATION-FREE CALIBRATION STRATEGY

  • Replacing the unconditional network output with the weighted guidance expression yields a linear least-squares problem, allowing each timestep’s two weights to be estimated independently from a small minibatch.The procedure is simulation-free and does not require sampling model trajectories.Source: §4.1 · p. 4
  • Simulation-free calibration does not guarantee trajectories close to the true posterior; its two-parameter restriction also uses components that are not necessarily optimal.The paper motivates simulation-based calibration in response to these limitations.Source: §4.1 · pp. 4–5

4.2 SIMULATION-BASED CALIBRATION STRATEGY

  • Simulation-based calibration replaces the forward-process density with the approximate posterior encountered during sampling, which depends on guidance weights from later timesteps.The resulting objective remains linear in the current timestep’s weights but depends nonlinearly on later weights.Source: §4.2 · p. 5
  • SIMCA optimizes each timestep’s two guidance weights through a 2 × 2 linear system, using a greedy strategy that holds previously calibrated weights fixed.The calibration depends on the sampler choice.Source: §4.2 · p. 5

4.3 PRACTICAL CONSIDERATIONS

  • One minibatch of sampling trajectories suffices for SIMCA, and calibrated weights add no inference-time overhead; they must be recomputed when the measurement or sampling setup changes.The procedure applies to additive guidance without retraining, architectural changes, or sampler modifications.Source: §4.3 · pp. 5–6

5 EXPERIMENTS

Experiments show that SIMCA performs strongly across diffusion and flow-matching benchmarks, with results varying by task and sampler budget. Ablations find that calibration can use very few images and that simulation-based calibration outperforms the simulation-free alternative.

5.1 EXPERIMENTAL SETUP

  • Evaluation follows DAPS protocols for diffusion and Flower protocols for flow matching, with calibration performed offline on images disjoint from test images.Baselines use published benchmark metrics under matching operators, noise levels, and test images, with hyperparameters grid-searched to maximize PSNR; each sampler configuration and discretization is calibrated separately.Source: §5.1 · p. 6

5.2 MAIN RESULTS

  • On FFHQ, calibrated guidance schedules vary by measurement operator rather than following one constant or shared hand-designed curve.The schedules are shown for three operators.Source: §5.2 · p. 6
  • Against DAPS, SIMCA has 31% lower LPIPS and 43% lower FID on FFHQ random inpainting, while box-inpainting results are mixed.It also surpasses DAPS on all four FFHQ super-resolution metrics and on PSNR, SSIM, and LPIPS for motion deblurring; on box inpainting, SSIM and LPIPS improve over every baseline while PSNR trails DDNM and FPS-SMC. Figure 2 compares the box-inpainting completions.Source: §5.2 · p. 7, Figure 2 · p. 7
  • Reducing diffusion sampling from 1000 to 50 steps cuts inference cost 20× with bounded perceptual cost; 500 steps halves the budget with no metric cost.At 50 steps, PSNR and SSIM improve again, reaching the table’s best values on four of five FFHQ tasks, while LPIPS and FID degrade moderately.Source: §5.2 · p. 7
  • SIMCA matches or beats state-of-the-art methods on almost every metric, task, and dataset; FFHQ random-inpainting LPIPS drops from 0.203 to 0.068 and FID from 69.20 to 18.31.On ImageNet random inpainting, LPIPS drops from 0.297 to 0.109 and FID from 87.53 to 32.08.Source: §5.2 · p. 6, Table 1 · p. 8
  • On CelebA, SIMCA leads 14 of 15 columns and improves over Flower on every task; on AFHQ-Cat, it leads all three metrics on both inpainting tasks.On AFHQ-Cat it also leads on super-resolution PSNR and SSIM, while PnP-GS, Flower, and OT-ODE perform best on denoising and deblurring.Source: §5.2 · p. 7, Table 2 · p. 8, Table 3 · p. 9

5.3 ABLATION STUDIES

  • Both simulation-free and simulation-based calibration improve on the uncalibrated sampler, but SIMCA is uniformly better, especially on DPS perceptual metrics.The comparison is on FFHQ random inpainting.Source: §5.3 · p. 8
  • A single calibration image suffices; solving one weight pair takes 0.11 seconds, and a full schedule takes seconds to minutes.The calibration set-size study varies image count on FFHQ random and box inpainting.Source: §5.3 · p. 8, Table 5 · p. 9

N LPIPS FID LPIPS FID Time (s)

After simulation-based calibration, three measurement-consistency terms achieve nearly indistinguishable results. Sampler performance at reduced step counts varies by sampler, with Euler–Maruyama degrading sharply below 100 steps.

Measurement-consistency comparison

  • LPIPS falls from 0.103 to 0.069 and FID from 33.71 to 19.16, a 43% reduction for the same number of samples.The same ordering holds for ΠGDM and TMPD.Source: N LPIPS FID LPIPS FID Time (s) · p. 9

Choice of the measurement-consistency term

  • After calibration, DPS, ΠGDM, and TMPD reach 32.91, 32.97, and 32.84 dB, with LPIPS of 0.069, 0.069, and 0.070.Uncalibrated, the terms span 25.10 to 27.89 dB and 0.156 to 0.248 LPIPS.Source: N LPIPS FID LPIPS FID Time (s) · p. 9
  • After calibration, the advantage of more elaborate measurement surrogates over the isotropic Gaussian term almost vanishes in these settings.The passage characterizes the uncalibrated advantage as effective scaling rather than expressiveness.Source: N LPIPS FID LPIPS FID Time (s) · p. 9

Sampler and discretization

  • DDPM improves in PSNR down to 50 steps, whereas Euler–Maruyama collapses below 100 steps, losing 14 dB by 20.Each sampler and step-count configuration is calibrated separately.Source: N LPIPS FID LPIPS FID Time (s) · p. 9

6 CONCLUSION

SIMCA calibrates guidance weights offline by reducing per-step guidance optimization to a two-dimensional linear least-squares problem. Across four datasets and multiple inverse problems, it matches or outperforms state-of-the-art methods using diffusion and flow-matching methods.

6 CONCLUSION

  • SIMCA reduces guidance optimization at each sampling step to a two-dimensional linear least-squares problem by exploiting conditional score- and flow-matching objectives.The method tunes guidance weights for training-free posterior sampling.Source: §6 · p. 9
  • The resulting weights are optimized once for a given inverse-problem setting, without retraining or fine-tuning the generative model.Source: §6 · p. 9
  • Across four datasets and multiple inverse problems, SIMCA consistently matches or outperforms state-of-the-art methods using diffusion and flow-matching methods.Source: §6 · p. 9

USE OF AI TOOLS

The appendix states the authors’ limited use of language-model tools, discusses related guidance approaches, and presents calibrated weights and step-count reconstructions. Recalibrating each discretization yields visually similar reconstructions on well-posed operators, while masked completions vary for box inpainting.

USE OF AI TOOLS

  • The authors used a large language model only to polish writing, draft figure and table captions, and assist with LaTeX formatting.They state that it was not used for research ideation, deriving or verifying theoretical results, or writing experimental code.Source: USE OF AI TOOLS · p. 10

A EXTENDED RELATED WORK

  • Full-dimensional control methods require differentiation through sampling dynamics at substantial cost, outside the target of guidance selection in seconds.The passage contrasts this with SIMCA’s intended use when a new measurement setup requires guidance to be set quickly.Source: A EXTENDED RELATED WORK · p. 13
  • SIMCA addresses exposure bias by obtaining the unconditional network-output scaling from an explicit criterion rather than a hand-tuned schedule.The passage identifies exposure bias as a mismatch between forward marginals and inference-time states.Source: A EXTENDED RELATED WORK · p. 13

B CALIBRATED WEIGHTS

C EFFECT OF THE STEP COUNT

  • At K = 1000, 500, and 50, reconstructions on four well-posed operators are visually almost indistinguishable when recalibrated at each discretization.The operators are Gaussian deblurring, motion deblurring, super-resolution, and random inpainting.Source: C EFFECT OF THE STEP COUNT · p. 14
  • For box inpainting, different step counts produce different masked-region completions while leaving the unmasked part identical.The passage attributes this variation to the masked region not being determined by the measurement.Source: C EFFECT OF THE STEP COUNT · p. 14

D ROBUSTNESS TO THE BACKBONE AND THE SAMPLER.

In flow-matching experiments, calibrated OT-coupled and independently coupled backbones behave almost identically, while renoising outperforms plain Euler integration on CelebA inpainting. The supplied figure caption identifies a separate FFHQ box-inpainting comparison across three step counts.

D ROBUSTNESS TO THE BACKBONE AND THE SAMPLER.

  • Calibrated OT-coupled and independently coupled flow-matching backbones differ by at most 0.2 dB and 0.002 LPIPS on CelebA.Source: D ROBUSTNESS TO THE BACKBONE AND THE… · p. 14
  • The renoising sampler outperforms plain Euler integration by 3.7 dB on CelebA random inpainting and 3.0 dB on box inpainting.The passage describes this as a sampler property, while Euler rows still improve on most flow-matching baselines.Source: D ROBUSTNESS TO THE BACKBONE AND THE… · p. 14
  • Figure 5 compares SIMCA on FFHQ box inpainting at K = 1000, 500, and 50 steps, recalibrated at each discretization.Source: Figure 5 · p. 15

E SAMPLER AND DISCRETIZATION

Sampler choice and discretization strongly affect reconstruction as the step budget shrinks. With weights recalibrated for each configuration, DDPM at 50 steps offers the strongest reported distortion results on two tasks at one-twentieth the nominal cost.

E SAMPLER AND DISCRETIZATION

  • At 50 steps, recalibrated DDPM gives the table’s best distortion metrics on random inpainting and super-resolution at one-twentieth the nominal cost.The reported recommendation is DDPM with weights recalibrated for that step count.Source: E SAMPLER AND DISCRETIZATION · p. 15
  • As the budget shrinks, DDPM improves in PSNR through 50 steps and fails only below 20, while Euler-Maruyama collapses below 100 and DDIM remains usable at 10.Euler-Maruyama’s explicit reverse-SDE discretization is reported unstable at large step sizes; DDIM remains at a consistently lower level.Source: E SAMPLER AND DISCRETIZATION · p. 15
  • Guidance weights depend on the sampler and timestep count, so transferring them across configurations rather than recalibrating forfeits the reported gains.Source: E SAMPLER AND DISCRETIZATION · p. 15

F EXPERIMENTS - TECHNICAL DETAILS

The experiments use pretrained diffusion and flow-matching backbones across multiple datasets and inverse-problem tasks. The section also reports sampler and step-count comparisons and illustrates recalibrated FFHQ sampling at three discretizations.

F EXPERIMENTS - TECHNICAL DETAILS

  • Diffusion uses released pixel-space DDPM checkpoints for FFHQ and ImageNet at 256 × 256, while flow matching uses Flower’s U-Net velocity networks on CelebA and AFHQ-Cat.The flow experiments include both OT-coupled and independently coupled velocity networks.Source: F EXPERIMENTS - TECHNICAL DETAILS · pp. 15, 17
  • At K = 50, DDPM alone remains stable and attains the table’s best distortion metrics on FFHQ random inpainting and super-resolution.SIMCA is recalibrated independently for each sampler and step count; stochastic samplers outperform deterministic counterparts at every budget.Source: Table 6 · p. 16
  • Figures 6–9 show SIMCA on FFHQ random inpainting, Gaussian deblurring, motion deblurring, and super-resolution at K = 1000, 500, and 50, recalibrated at each discretization.Source: Figure 6 · p. 17, Figure 7 · p. 18, Figure 8 · p. 19
  • Evaluation uses 100 held-out images per dataset and reports PSNR, SSIM, and LPIPS on both benchmarks, plus FID on diffusion.The diffusion datasets are FFHQ and ImageNet at 256 × 256; flow datasets are CelebA at 128 × 128 and AFHQ-Cat at 256 × 256.Source: F EXPERIMENTS - TECHNICAL DETAILS · p. 17
  • Diffusion tests five inverse problems with σm = 0.05 noise, while flow matching tests five Flower benchmark tasks using unchanged operators and noise levels.Diffusion tasks include ×4 super-resolution, Gaussian and motion deblurring, box inpainting, and random inpainting; flow tasks include denoising, Gaussian deblurring, super-resolution, random inpainting, and box inpainting.Source: F EXPERIMENTS - TECHNICAL DETAILS · p. 17

Related papers

Loading 2610.03503v1…