PaperlayerView exact revision on arXiv

Source-linked AI summary

Parallel Diffusion Models of Operator and Image for Blind Inverse Problems

Hyungjin Chung, Jeongsol Kim, Sehui Kim, Jong Chul Ye

arXiv:2211.10656v1cs.CVcs.LGstat.ML

TL;DR

Blind inverse problems remain challenging because the forward operator is unknown and must be estimated jointly with the image. BlindDPS constructs diffusion priors for the image and operator and performs parallel reverse diffusion guided by measurement consistency. It establishes state-of-the-art performance on blind deblurring and imaging through turbulence, while remaining applicable when the forward functional form is known.

  • Problem

    Diffusion inverse-problem solvers were limited to known, fixed forward operators, whereas blind problems require jointly estimating the operator and image.

  • Method

    BlindDPS constructs multiple diffusion processes for the image and forward-operator parameters and couples their parallel reverse diffusion through approximate likelihood guidance.

  • Results

    BlindDPS establishes state-of-the-art performance on blind deblurring and imaging through turbulence, including settings with heavy degradation and measurement noise.

  • Takeaways & Limitations

    The method provides a coarse-to-fine diffusion strategy applicable to blind inverse problems whose forward functional forms are known.

  • Takeaways & Limitations

    BlindDPS is typically less robust than non-blind reconstruction, can diverge, and does not yet solve cases where the forward mapping’s functional form is unknown.

Abstract

from arXiv · show

Diffusion model-based inverse problem solvers have demonstrated state-of-the-art performance in cases where the forward operator is known (i.e. non-blind). However, the applicability of the method to blind inverse problems has yet to be explored. In this work, we show that we can indeed solve a family of blind inverse problems by constructing another diffusion prior for the forward operator. Specifically, parallel reverse diffusion guided by gradients from the intermediate stages enables joint optimization of both the forward operator parameters as well as the image, such that both are jointly estimated at the end of the parallel reverse diffusion procedure. We show the efficacy of our method on two representative tasks -- blind deblurring, and imaging through turbulence -- and show that our method yields state-of-the-art performance, while also being flexible to be applicable to general blind inverse problems when we know the functional forms.

1. Introduction

Blind inverse problems require jointly estimating an unknown forward operator and latent image, making them less stable than non-blind reconstruction. BlindDPS addresses this gap with parallel diffusion priors for the image and operator, achieving state-of-the-art results across blind deblurring and turbulence imaging.

  • Problem: Blind inverse problems jointly estimate the latent image and unknown forward operator, making them considerably harder than non-blind problems.Joint minimization is typically much less stable.
  • Motivation: Diffusion inverse-problem solvers had primarily addressed settings with known and fixed forward operators.Known operators allow likelihood computation while optimizing only the latent image.
  • Motivation: Unknown operators require priors for both the image and operator, while conventional patch-based and sparsity priors often model their distributions inaccurately.The paper identifies shortcomings in both prior modeling and optimization stability.
  • BlindDPS: BlindDPS constructs multiple diffusion processes and runs reverse diffusion in parallel to estimate the image and operator from noisy initialization.Cross-talk between the paths is enforced through the approximate likelihood and measurement.
  • Scope and results: BlindDPS provides a coarse-to-fine diffusion strategy and applies to known forward-model structures, including blind deblurring and imaging through turbulence.The authors report state-of-the-art performance and generalizability across these inverse problems.

2. Background

The background formulates diffusion posterior sampling for known forward operators and then extends the blind inverse-problem setup to unknown parameterized operators. It motivates BlindDPS by highlighting limitations in conventional priors and optimization procedures.

  • Diffusion models: Variance-preserving diffusion gradually transforms data into Gaussian noise, while reverse diffusion uses score functions to sample from the data prior.The noise schedule maps pdata to an approximately standard Gaussian distribution.
  • Diffusion posterior sampling: Diffusion posterior sampling combines the measurement likelihood gradient with the prior score to sample latent images consistent with corrupted observations.The likelihood term may be approximated using the denoised intermediate estimate.
  • Blind inverse problems: The standard posterior-sampling method is limited to a fixed forward model and therefore cannot directly solve blind inverse problems.Blind problems require estimating the forward-model parameter together with the image.
  • Blind inverse problems: Blind inverse problems use a parameterized forward model in which both the operator parameter and latent image are unknown.Classical optimization introduces separate regularization functions that can be interpreted as negative log priors.
  • Forward models: The framework covers blind deconvolution and turbulence imaging, with the latter represented using a tilt operator parameterized by a tilt vector field.Kernel magnitude and polarity constraints help remove scale ambiguity between the kernel and image.
  • Challenges: Conventional blind-inverse optimization is limited by imperfect prior functions and unstable, difficult-to-tune minimization.The paper identifies both prior accuracy and optimization quality as determinants of success.

3. BlindDPS

BlindDPS places diffusion priors on both the image and unknown forward-model parameters, then couples parallel reverse-diffusion processes through posterior likelihood gradients. Independence enables separate reverse processes, while measurement-based guidance and kernel regularization produce a coarse-to-fine blind reconstruction procedure.

  • Dual diffusion priors: BlindDPS uses diffusion priors for both the image p(x0) and forward-model parameter p(ϕ), estimating each prior through its score function.This extends diffusion-based posterior sampling beyond known operators to blind settings.
  • Parallel reverse diffusion: Independence of x0 and k0 permits two separate reverse diffusion processes, one for the image and one for the convolution kernel.Their posterior dependence is then introduced through likelihood-gradient terms involving the measurement.
  • Measurement guidance: The reverse processes use gradients of the approximate time-conditional log likelihood, whose analytic tractability makes the coupled equations numerically solvable for Gaussian measurements.The resulting procedure combines posterior guidance with ancestral sampling.
  • Kernel regularization: BlindDPS augments the kernel diffusion prior with ℓ0/ℓ1 regularization to promote sparsity and stabilize kernel reconstruction.The regularization strength is λ, while the choice of Rk depends on the dataset.
  • Coarse-to-fine interpretation: Reverse diffusion provides a smooth coarse-to-fine evolution through Gaussian scale-space, replacing the abrupt, ad-hoc schedules commonly used in blind deconvolution.This interpretation connects the method to conventional multiscale optimization while retaining a continuous evolution schedule.

4. Experiments

Experiments evaluate BlindDPS on blind deblurring and imaging through turbulence using image, kernel, and perceptual metrics. BlindDPS outperforms comparison methods in the reported qualitative and quantitative results, while ablations examine its diffusion and sparsity priors.

  • Experimental setup: BlindDPS is evaluated on FFHQ and AFHQ blind deblurring, and on FFHQ and ImageNet imaging through turbulence.The experiments use motion and Gaussian blur, with turbulence modeled through kernel blur and tilt maps.
  • Evaluation: The evaluation measures FID, LPIPS, and PSNR for reconstructed images, plus MSE and MNC for kernel estimation.The reported settings include Gaussian measurement noise with σ = 0.02 and step size α = 0.3.
  • Blind deblurring: BlindDPS accurately captures sharp images and kernels under aggressive motion blur, while competing methods often produce blur, inaccurate kernels, or failed reconstructions.Figure 4 compares measurements, Pan-DCP, MPRNet, SelfDeblur, BlindDPS, and ground truth across FFHQ and AFHQ motion and Gaussian deblurring.
  • Results: BlindDPS establishes state-of-the-art performance across the reported blind deblurring metrics and outperforms comparison methods in most turbulence cases.For turbulence, the largest advantages are reported on perceptual metrics such as FID and LPIPS.
  • Results: For PSNR, BlindDPS often slightly underperforms supervised learning approaches because recovering high-frequency details can penalize distortion metrics.This limitation is reported for reconstructions from heavy degradations.
  • Ablation studies: The diffusion prior for the forward model is crucial: replacing it with a uniform kernel prior produces heavily distorted reconstructions and poorly estimated kernels.A separate ablation studies the additional sparsity regularization applied to kernels.

5. Discussion and Related Works

The discussion positions BlindDPS within diffusion-based inverse-problem methods and reports its broader applicability alongside practical limitations. The method extends to turbulence but remains sensitive to tuning, computationally heavier, and limited to known forward-model forms.

  • Related works: Prior diffusion inverse-problem methods addressed non-blind settings, whereas BlindDPS extends posterior-sampling ideas to blind inverse problems.The discussion identifies BlindDPS as the first method in this line to show posterior sampling scaling to blind settings.
  • Imaging through turbulence: BlindDPS reconstructs turbulent images using a setup whose rows contain FFHQ and ImageNet examples compared with ILVR, MPRNet, TSR-WGAN, and ground truth.The figure organizes each row into measurement, competing methods, BlindDPS, and ground truth.
  • Limitations: Joint minimization over multiple factors is typically less robust than non-blind reconstruction and can diverge when parameters are incorrectly tuned.The stated limitation applies to components such as the kernel, tilt map, and image.
  • Limitations: In turbulence experiments, the tilt map is often estimated incorrectly even when the kernel and ground-truth image are accurately estimated.This identifies a component-specific failure pattern rather than a general failure of all reconstructed quantities.
  • Limitations: Inference slows as additional component-specific score functions require extra forward and backward passes, with the number of score functions scaling linearly with added parameters.The method does not yet solve the truly blind case where the forward mapping's functional form is unknown.

6. Conclusion

The paper proposes BlindDPS for jointly estimating forward-operator parameters and images in blind inverse problems. It theoretically constructs multiple reverse SDEs and reports state-of-the-art performance on blind deblurring and imaging through turbulence under heavy degradation and noise.

  • Contribution: BlindDPS jointly estimates the forward measurement operator's parameters and the image to be reconstructed.The framework targets blind inverse problems rather than only cases with a known operator.
  • Method: The method constructs multiple reverse SDEs with component-specific score functions to approximate posterior sampling for blind inverse problems.The construction supports separate modeling of the factors involved in the measurement process.
  • Results: BlindDPS establishes state-of-the-art performance on blind deblurring and imaging through turbulence, even with heavy degradation and measurement noise.The conclusion summarizes the paper's results across both evaluated blind inverse-problem tasks.

A. Proofs

The proof develops tractable posterior approximations for jointly diffused image and operator variables, then extends the construction to turbulence imaging. The approximation error decreases as diffusion noise increases.

  • Posterior approximation: BlindDPS approximates p(y|x_t, k_t) using posterior means x̂_0 and k̂_0 computed from the intermediate diffusion states.The image and kernel posterior means are obtained efficiently through one forward pass of their respective score networks.
  • Approximation analysis: The Jensen gap quantifies the error between the expected likelihood and the likelihood evaluated at posterior means.The proof derives an upper bound for this gap under Gaussian diffusion assumptions.
  • Approximation analysis: 0 approximation error is approached as the noise level σ increases sufficiently.This establishes the regime in which the posterior-mean likelihood approximation becomes increasingly accurate.
  • Imaging through turbulence: For imaging through turbulence, the same reverse-SDE construction is extended to image, kernel, and tilt-map variables.The turbulence formulation uses a score function trained on tilt maps and an ancestral-sampling algorithm returning all three estimates.

C.1. Diffusion prior for the forward model

The forward-model prior is a central design choice in blind diffusion sampling: replacing a uniform kernel prior with a learned diffusion prior enables more accurate modeling of higher-dimensional operators. The method’s estimates evolve through a coarse-to-fine process, with error minima occurring before the final reverse-diffusion steps.

  • Diffusion prior for the forward model: Uniform-prior estimation far underperforms the proposed diffusion prior, especially when the unknown operator has higher-dimensional parameters.The comparison supports using richer operator priors beyond the scalar-parameter setting where uniform priors may be feasible.
  • Estimation dynamics: BlindDPS admits a Gaussian scale-space interpretation in which intermediate denoised estimates provide a coarse-to-fine reconstruction path.This connects the reverse-diffusion evolution to coarse-to-fine optimization strategies.
  • Estimation dynamics: The MSE reaches its minimum at about 400/1000 iterations for images and 200/1000 iterations for kernels.Later steps mainly fill in high-frequency image details and improve perceptual quality.
  • Applications: The method is evaluated for blind deblurring and imaging through turbulence using forward models constructed for convolutional and tilt-based operators.The turbulence setting uses a tilt vector field φ and a tilt-blur forward model.

F.2. Compute time

The compute analysis measures the additional reconstruction cost associated with using multiple score functions at inference time.

  • Compute scaling: Inference-time score-function count scales linearly with the number of forward-model components.Blind deblurring uses two score networks, while imaging through turbulence uses three; wall-clock time is measured on a single RTX 2080ti GPU.

F.3. Comparison methods

The comparison methods span optimization-based, self-supervised, diffusion-based, and adversarial approaches, with implementation choices adapted to each task. Kernel-size mismatch requires post-processing for several baselines.

  • Optimization-based methods: Pan-DCP uses a dark channel prior and eight coarse-to-fine optimization stages for facial blur images.The implementation uses the official method and advised parameters.
  • Turbulence comparisons: TSR-WGAN is adapted from spatiotemporal 3D data to single-frame turbulence imaging with a 2D U-Net-like architecture.Other training settings follow the original method’s defaults.
  • Evaluation alignment: Several kernel-estimating baselines support only odd-sized kernels, so 65×65 estimates are cropped to match the even-sized ground-truth kernels.The reported discrepancy is described as having only a marginal practical effect.

G. Further Experiments

The supplementary experiments cover blind deblurring and imaging through turbulence across FFHQ, AFHQ, and ImageNet 256 × 256 datasets. They compare BlindDPS with task-specific baselines and ground truth.

  • The supplementary figures provide additional blind deblurring and turbulence-imaging results.
  • Blind motion deblurring is evaluated on FFHQ and AFHQ 256 × 256 datasets against Pan-DCP, MPRNet, SelfDeblur, and ground truth.
  • Blind Gaussian deblurring is evaluated on FFHQ and AFHQ 256 × 256 datasets against Pan-DCP, MPRNet, SelfDeblur, and ground truth.
  • Imaging through turbulence is evaluated on FFHQ and ImageNet 256 × 256 datasets against ILVR, MPRNet, TSR-WGAN, and ground truth.
Loading 2211.10656v1…