PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Based Unpaired Data Learning for Inverse Problems

Chenglong Bao, Yiming Dang, Chenguang Duan, Yuling Jiao, Defeng Sun

arXiv:2609.01370v1cs.CV

TL;DR

Paired data are difficult to obtain for many deep learning inverse problems, motivating methods that can use unpaired marginal data. LUD-DIF decouples the joint ELBO into two diffusion processes under weak coupling, analyzes the resulting bias, and performs strongly on image denoising and super-resolution tasks.

  • Problem

    Deep learning inverse solvers depend on paired samples, but acquiring paired data can be prohibitively expensive or infeasible in applications such as medical and astronomical imaging.

  • Method

    LUD-DIF uses variational inference to decouple the joint ELBO into two independently trainable diffusion processes using unpaired marginal data.

  • Results

    LUD-DIF achieves outstanding performance on natural image denoising and super-resolution datasets and the best performance on most metrics for real-world noise modeling.

  • Takeaways & Limitations

    LUD-DIF provides a feasible diffusion-based training algorithm for inverse problems with unpaired data and demonstrates effectiveness on image restoration tasks.

  • Takeaways & Limitations

    LUD-DIF requires a known or approximate forward operator, and reducing the error from the weak-coupling assumption remains open.

Abstract

from arXiv · show

Data is important in many deep learning-based inverse problem solvers. However, obtaining sufficient paired data in many scenarios remains highly challenging, while unpaired data is cheap. To maximize data utilization, this paper proposes LUD-DIF, a diffusion-based approach for solving inverse problems with unpaired data. Starting from the evidence lower bound (ELBO) of the joint distribution, we decouple it into two independent diffusion processes under the weak-coupling assumption. The method provides theoretical support from a variational inference perspective, derives the loss function, quantitatively analyzes the error bound introduced by the assumption, and offers a theorem-motivated heuristic for hyperparameter selection. Experimental results demonstrate that LUD-DIF achieves outstanding performance on multiple image inverse problems, validating its effectiveness and generalization capability in unpaired inverse problem settings.

1 Introduction

Inverse problems are difficult to solve with deep learning when paired observations are unavailable, despite unpaired marginal data being more accessible. LUD-DIF addresses this gap by decoupling a joint ELBO into independently trainable diffusion processes and validating the approach on image restoration tasks.

  • Motivation: Inverse problems are often ill-posed because observation noise is unknown and forward operators can be nonlinear or low-rank.Traditional approaches rely on priors such as sparsity, smoothness, or low-rank structure, or on Gaussian-noise Bayesian assumptions.
  • Motivation: Deep learning inverse solvers require many paired samples, which can be prohibitively expensive or infeasible to acquire in applications such as medical and astronomical imaging.Unpaired learning instead seeks to exploit the marginal distributions p(x) and p(y) without paired samples.
  • Method: LUD-DIF constructs proxy pairs through diffusion alignment and derives a trainable dual-path conditional diffusion objective from a marginal–conditional decomposition of the joint ELBO.The formulation uses only unpaired marginal samples and quantifies bias from the weak-coupling approximation.
  • Method: The method decouples the joint ELBO under a reasonable data-prior assumption, yielding a feasible diffusion-based training algorithm for unpaired inverse problems.This contribution is framed from a variational-inference perspective.
  • Theory and tuning: LUD-DIF quantitatively analyzes assumption-induced theoretical error and proposes a theorem-motivated heuristic for selecting hyperparameters (S, V).The heuristic is intended to provide a theoretically grounded practical strategy for model tuning.
  • Validation: LUD-DIF achieves outstanding performance on natural image denoising and super-resolution datasets.These experiments validate the method across multiple image inverse-problem tasks.

2 Preliminaries

Diffusion models learn data distributions by adding noise through a fixed forward process and learning a reverse generative process. Their ELBO objective can be rewritten into a tractable decomposition used for training.

  • Diffusion models: Diffusion models approximate an unknown data distribution by gradually adding noise and learning to reverse the process.The reverse process generates samples from a noise distribution through learned transition kernels.
  • Forward process: The forward process adds noise through a fixed Markovian chain governed by a prescribed variance schedule.The schedule controls the noise injected at each diffusion step.
  • Forward process: Iterating the forward recursion yields a closed-form conditional marginal distribution for the noisy state U_t given the original state U_0.The notation defines the cumulative product of the schedule parameters through ᾱ_t.
  • Forward process: As ᾱ_T → 0, the final conditional distribution p_U_T|U_0 converges to a standard Gaussian N(0, I_d).This Gaussian endpoint supplies the starting distribution for reverse generation.
  • ELBO formulation: Diffusion models are trained by optimizing the evidence lower bound of the log-likelihood, which Markovian structure rewrites into a tractable decomposition.The decomposition includes a constant forward-process term and a reconstruction term.
  • ELBO formulation: The reconstruction term uses a Gaussian posterior transition p_U_t−1|U_t,U_0 with mean μ̃_t and variance β̃_t I_d.The displayed derivation leads to a mean-squared-error loss function.

3 The LUD-DIF method

LUD-DIF formulates unpaired inverse problems as joint-distribution learning with a tractable variational diffusion model. It replaces inaccessible paired-data objectives with weakly coupled marginal-data objectives whose consistency follows under the weak-coupling assumption.

  • 3.1 Problem Formulation: The inverse problem seeks to recover x from y, but unknown complex noise makes estimating the joint distribution pX0,Y0 difficult.The forward operator may also be nonlinear or low-rank, contributing to ill-posedness.
  • 3.1 Problem Formulation: LUD-DIF reduces the forward model to conditional noise modeling by treating A(X0) as an effective signal variable.An approximate unbiased forward operator can be used by absorbing its residual into effective noise.
  • 3.2 Learning joint distribution via diffusion models: The variational reverse process factors into an X-reverse kernel and a Y-reverse kernel conditioned on the less-diffused structural state and noisy observation.This tractable Gaussian factorization is a restriction of the variational family, not a true conditional-independence claim.
  • 3.2 Learning joint distribution via diffusion models: The joint ELBO decomposes into marginal and conditional terms, allowing the marginal term to use pX0 samples while the conditional term uses weak-coupling samples.The decomposition is established under the stated variational family.
  • 3.1 Problem Formulation: Unpaired learning provides only marginal samples, so pX0|Y0 and the joint distribution are not uniquely accessible.This motivates constructing a computable surrogate coupling from marginal data.
  • 3.2 Learning joint distribution via diffusion models: The weak-coupling time steps S and V balance alignment accuracy against information loss from perturbing the observation.Small steps favor perturbation accuracy, whereas larger steps improve alignment but degrade retained information.
  • 3.2 Learning joint distribution via diffusion models: Under weak coupling, the computable weak conditional loss equals the ideal paired-data loss, and the two diffusion models can be trained separately.The equality is stated as Ljoint(θ) = Lweak(θ), while separate training improves conditional-model training efficiency.

4 The Bias Analysis without Assumption 3.4

The paper quantifies bias from relaxing weak coupling by bounding discrepancies between true and weakly coupled posteriors. The resulting bounds separate alignment and perturbation errors and motivate practical selection of diffusion time steps.

  • 4 The Bias Analysis without Assumption 3.4: The bias analysis controls weakly coupled loss error through the expected total variation distance between true and weakly coupled posteriors.Learning and sampling errors are excluded because pX0|XV is treated as exact in this analysis.
  • 4 The Bias Analysis without Assumption 3.4: The posterior discrepancy decomposes into alignment error from differing noisy representations and perturbation error from information loss in YS.These correspond to the two approximations underlying weak coupling.
  • 4 The Bias Analysis without Assumption 3.4: Posterior stability is adopted as a regularity assumption, with sufficient conditions covering Gaussian and broad classes of non-Gaussian noise.A positive noise density with bounded log-density Hessian yields a TV-Lipschitz posterior map under the stated data assumptions.
  • 4 The Bias Analysis without Assumption 3.4: Theorem 4.5 provides explicit upper bounds for both perturbation and alignment errors in terms of S, V, and the noise under stated assumptions.The bounds rely on data, noise, network-growth, and posterior-stability conditions.
  • 4 The Bias Analysis without Assumption 3.4: Theorem 4.5 and Corollary 4.6 bound the weakly coupled loss and conditional generative error under the same assumptions.The corollary concerns the weakly coupled conditional distribution pweakY0|X0.
  • 4 The Bias Analysis without Assumption 3.4: Variance alignment and simplified heuristic objectives guide selection of S and V, while Gaussian white noise permits S = 0 and eliminates the perturbation term.For practical noise, σX ≈ σY is used, and V is determined by a variance-alignment constraint.
  • 4 The Bias Analysis without Assumption 3.4: Figure 1 compares LUD-VAE and LUD-DIF using per-image KL divergence scatter plots and generated-noise distribution curves on BSDS300.Points below y = x indicate samples where LUD-DIF performs worse; selected samples lie in the 25%–50% best range for both methods.

5 Experimental Results

LUD-DIF is evaluated on simulated noise, real-world smartphone noise, and image super-resolution, using generated degradations for downstream restoration. Results show strong noise modeling and restoration performance, with limitations on multimodal and discrete noise distributions.

  • Experiments cover simulated noise modeling, SIDD real-world noise modeling, and image super-resolution.The evaluation includes natural-image noise distributions, smartphone noise, and pseudo-degraded images for super-resolution.
  • 5.1 Simulated Image Noise: On simulated noise, LUD-DIF captures Poisson characteristics effectively and outperforms LUD-VAE for extreme salt-and-pepper patterns, while Gaussian-mixture reconstruction remains imprecise.The Gaussian-mixture limitation is attributed to stronger non-Gaussianity and higher-order moment discrepancy; smoothing improves salt-and-pepper visual quality, but KL divergence remains relatively high.
  • 5.2 Real-world Noise Modeling: On SIDD, LUD-DIF achieves the best performance on most noise-generation metrics and substantially improves noisy-image generation quality across all three metrics.Weak-Unpaired training slightly outperforms Strong-Unpaired training, while the unpaired-versus-paired gap is consistent with the weak-coupling analysis.
  • LUD-DIF uses separate clean-image and conditional noisy-image diffusion models, with generated pairs supporting degradation evaluation and restoration training.A clean sample is diffused to an aligned state, then the conditional reverse process generates the corresponding degraded observation.
  • 5.3 Image Super-Resolution: For AIM19 and NTIRE20 super-resolution, LUD-DIF achieves the best AKLD and FID, the best LPIPS on both datasets, and the best SSIM on NTIRE20.Generated degraded images are paired with high-resolution images to train ESRGAN for downstream evaluation.

6 Conclusion and Future Work

LUD-DIF uses two independently trainable diffusion processes under a weak-coupling assumption, while analyzing the resulting error bound and selecting hyperparameters heuristically. Its main limitations are dependence on a known or approximate forward operator and unresolved reduction of weak-coupling error.

  • Under the weak-coupling assumption, the joint ELBO decouples into two independently trainable diffusion processes.
  • The method quantitatively analyzes the error bound introduced when weak coupling is assumed and provides a theorem-motivated hyperparameter-selection heuristic.
  • Experiments on simulated and real-world image noise and image super-resolution demonstrate LUD-DIF’s effectiveness.
  • LUD-DIF requires a known or approximate forward operator, limiting its use in blind inverse problems and semantic modality translation.
  • How to reduce the error introduced by the weak-coupling assumption remains open.

A.2 Proof of Proposition 3.3

The proof expands the joint ELBO into terms and uses forward and reverse factorization to derive the required decomposition, with constants independent of model parameters separated out.

  • The joint ELBO is expanded into terms A1, A2, and A3 for separate analysis.
  • Term A1 is constant with respect to θ and can therefore be excluded from optimization.
  • The analysis of A2 follows directly from the reverse factorization of the diffusion process.
  • Term A3 is handled using both forward-posterior and reverse factorization identities.
  • Substituting the derived expressions completes the decomposition, while C(z0) is independent of θ and computable using the standard Gaussian KL formula.

A.3 Proof of Theorem 3.7

The proof establishes the weak-coupling theorem by comparing objective terms, defining an auxiliary weakly coupled variable, and applying conditional-independence and diffusion-process identities.

  • The proof compares objectives (3.13) and (3.17) and reduces the theorem to showing their required equality.
  • A weakly coupled variable is introduced to construct the auxiliary process used in the comparison.
  • The auxiliary process shares the original forward process, preserving the relevant diffusion dynamics.
  • Conditional-independence relations and the Chapman–Kolmogorov equation simplify the transition and posterior terms.
  • The remaining objective discrepancy is bounded using the triangle inequality, assumptions on the model, and expectation inequalities.

A.5 Proof of Theorem 4.5

The theorem’s proof bounds the weak-coupling error through Gaussian smoothing, conditional-noise comparisons, and posterior stability, with an extension to signal-dependent noise. Experiments additionally test whether generated noise samples are independent.

  • Lemma A.2 establishes finite KL divergence and a Gaussian-smoothed KL-divergence bound under bounded density and finite-second-moment conditions.
  • The proof estimates perturbation error using conditional independence, Jensen’s inequality, and diffusion-time normalization.
  • Alignment error is decomposed into mean, variance, and Gaussian-approximation discrepancies.
  • Pinsker’s inequality and Gaussian KL bounds control the component discrepancies before they are combined into the final theorem bound.
  • The analysis extends to signal-dependent noise by using a trace-matched Gaussian reference and assuming posterior Lipschitz stability.
  • Proposition A.3 states that, under the trace-averaged alignment condition, the loss gap remains controlled by posterior mismatch.
  • Generated noise samples have Pearson correlation coefficients consistently below 0.05, supporting their independence, although some high-noise images show correlated regions.

B.2 Training and Sampling Hyperparameters

Training losses for both diffusion networks stop decreasing after 100k iterations, while validation FID is tracked to guide conditional-network sampling choices. Sampling quality is best at 200 steps with η = 0.5, but simulated-pair generation prioritizes faster sampling while preserving stochasticity.

  • Training iterations: Both diffusion-network losses stop decreasing after 100k iterations.The conditional diffusion network’s validation FID is also monitored every 50 iterations for sampled validation results.
  • Sampling settings: 200 sampling steps with η = 0.5 yields the best generation quality on the SIDD validation set.The evaluation varies both the number of sampling steps and the random term η.
  • Sampling settings: Training-set sampling prioritizes faster generation while preserving stochasticity rather than using the validation-set optimum unchanged.This selection is made when generating simulated paired data for training.

B.3 Analysis of Degradation Level

The degradation-level analysis evaluates effective diffusion noise τS across datasets and sampling configurations. Results reveal a weak-coupling trade-off: increasing perturbation can improve alignment initially but eventually causes information loss and worse FID, with dataset-dependent optima.

  • Effective diffusion noise: For “n2n,” increasing τS worsens FID because additional Gaussian perturbations disrupt structural details.The n2n condition already approaches the weak-reconstruction condition used during training when τS is small.
  • Effective diffusion noise: “c2n” FID initially declines as alignment improves, then rises as excessive perturbation causes information loss.This behavior illustrates the weak-coupling trade-off described in the paper.
  • Dataset dependence: The τS yielding optimal FID on AIM19 is significantly larger than the optimum on NTIRE20.The difference is qualitatively consistent with dataset-dependent noise statistics.
  • Sampling steps: One-step sampling is used for AIM19 and NTIRE20 because their relatively small noise variance concentrates pX0|XV around clean images.This choice improves efficiency for subsequent conditional-diffusion training.
  • Sampling steps: On SIDD, final-generation FID improves rapidly within the first few steps and reaches a near-optimal range around 4–6 steps.More sampling steps do not further improve quality and may degrade it.

B.5 Role of the Two-Path Diffusion Architecture

The two-path architecture can be reduced to a direct clean reconstruction baseline or used to generate clean–noisy pairs through its conditional path. Table 6 evaluates whether explicit noisy-domain modeling adds information beyond weak reconstruction.

  • N2C baseline: The clean-domain diffusion model alone maps a noisy observation to a clean reconstruction in the N2C baseline.The pipeline is y0 → yS ≡ xV → x0.
  • Two-path architecture: The full architecture combines clean-domain diffusion with a learned conditional diffusion model for Y | X.This explicitly models both the clean-domain process and the noisy-domain conditional path.
  • C2N generation: In C2N mode, generated noisy observations form clean–noisy pairs used to train the downstream DnCNN denoiser.Table 6 compares this setup with direct weak reconstruction to assess the value of the conditional path.
Loading 2609.01370v1…