PaperlayerView exact revision on arXiv

Source-linked AI summary

Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction

Hyungjin Chung, Byeongsu Sim, Jong Chul Ye

arXiv:2112.05146v2eess.IVcs.CVcs.LGstat.ML

TL;DR

Diffusion models are slow for inverse problems because conventional sampling starts from Gaussian noise and uses long reverse trajectories. CCDF instead forward-diffuses a better initialization, uses alternating reverse diffusion and data consistency, and obtains accelerated reconstruction across several tasks, including MRI at t0=0.02 and 50× acceleration. The paper notes that selecting t0 remains task- and corruption-dependent.

  • Problem

    Diffusion models provide strong inverse-problem performance but are critically slow to sample because they conventionally start from Gaussian noise and use long reverse trajectories.

  • Method

    CCDF forward-diffuses a better initial estimate to t0<T, then alternates conditional reverse diffusion with a non-expansive data-consistency step.

  • Results

    CCDF significantly accelerates reconstruction across super-resolution, inpainting, and MRI, including MRI at t0=0.02 with 50× acceleration and performance outperforming score-MRI.

  • Takeaways & Limitations

    Pre-trained feed-forward neural networks can be synergistically combined with diffusion models to enable accurate, stable, and faster inverse-problem reconstruction.

  • Takeaways & Limitations

    Choosing t0 varies across tasks and corruption levels, lacks a clear rule, and currently relies mostly on trial and error.

Abstract

from arXiv · show

Diffusion models have recently attained significant interest within the community owing to their strong performance as generative models. Furthermore, its application to inverse problems have demonstrated state-of-the-art performance. Unfortunately, diffusion models have a critical downside - they are inherently slow to sample from, needing few thousand steps of iteration to generate images from pure Gaussian noise. In this work, we show that starting from Gaussian noise is unnecessary. Instead, starting from a single forward diffusion with better initialization significantly reduces the number of sampling steps in the reverse conditional diffusion. This phenomenon is formally explained by the contraction theory of the stochastic difference equations like our conditional diffusion strategy - the alternating applications of reverse diffusion followed by a non-expansive data consistency step. The new sampling strategy, dubbed Come-Closer-Diffuse-Faster (CCDF), also reveals a new insight on how the existing feed-forward neural network approaches for inverse problems can be synergistically combined with the diffusion models. Experimental results with super-resolution, image inpainting, and compressed sensing MRI demonstrate that our method can achieve state-of-the-art reconstruction performance at significantly reduced sampling steps.

1. Introduction

Diffusion models offer strong generative and inverse-problem performance but are slow to sample. CCDF accelerates conditional diffusion by starting from forward-diffused, better-initialized estimates and leveraging contraction during reverse diffusion.

  • Conditional inverse problems: Conditional diffusion supports inverse problems including super-resolution, inpainting, and MRI reconstruction, but task-specific redesigns reduce flexibility.The paper builds on approaches that retain a general training procedure for multiple tasks.
  • Motivation: Diffusion models achieve state-of-the-art generative performance but require slow sampling from Gaussian noise.The standard reverse process gradually denoises a Gaussian sample through many iterations.
  • CCDF: CCDF accelerates conditional diffusion by studying contraction in the reverse diffusion path.Its conditional strategy alternates reverse diffusion with a non-expansive data-consistency operation.
  • CCDF: CCDF forward-diffuses an initial estimate only to t0<T before reverse diffusion, avoiding the standard start from Gaussian noise at t=T.The forward diffusion requires no neural-network passes, while reverse diffusion reduces error exponentially through contraction.
  • Evaluation: Experiments across super-resolution, inpainting, and MRI reconstruction show significant acceleration of diffusion-based inverse-problem models.The paper evaluates qualitative and quantitative reconstruction behavior while reducing t0.

2. Background

Score-based diffusion constructs data through a Gaussian-noising forward process and reverses it using a learned score. The paper presents continuous SDEs and discrete DDPM, DDIM, and SMLD formulations underlying these processes.

  • Continuous diffusion: The forward diffusion gradually perturbs data with Gaussian noise until reaching a tractable distribution that is approximately spherical Gaussian.The process is modeled by an Itô SDE with drift and diffusion coefficients.
  • Reverse diffusion: The reverse-time SDE denoises samples backward using the score function, which is approximated by a neural network trained through score matching.Denoising score matching replaces the marginal score with a conditional perturbation-kernel score.
  • SDE variants: VP-SDE uses drift −1/2 β(t)x and diffusion √β(t), with β(t) increasing over the noise scale.These continuous dynamics correspond to discrete DDPM under linear time discretization.
  • SDE variants: VE-SDE uses zero drift and diffusion √(d[σ^2(t)]/dt), where σ(t) is a monotonically increasing noise scale.The associated discrete formulation is SMLD, whose forward step adds scaled Gaussian noise.
  • Discrete models: DDPM and DDIM define discrete reverse-diffusion updates, while DDIM removes the additive noise term to enable fewer iterations.SMLD provides another discrete diffusion and reverse-diffusion formulation based on score estimates and noise-scale differences.

3. Main Contribution

CCDF combines one-step forward diffusion from a better inverse-problem estimate with conditional reverse diffusion and data consistency. Stochastic contraction explains why shorter paths can converge rapidly, especially with neural-network initialization.

  • 3.1. The CCDF Algorithm: CCDF starts reverse diffusion at N′=Nt0<N after forward-diffusing a better initialization, reducing the number of reverse steps.The algorithm forward-diffuses to N′ and then performs conditional reverse diffusion down to i=0.
  • 3.1. The CCDF Algorithm: Conditional diffusion alternates standard reverse diffusion with a non-expansive data-consistency mapping.The mapping satisfies ||A x−A x′||≤||x−x′||, and linear A includes gradient descent or POCS examples.
  • Assumptions: The method assumes a linear data-consistency operator A, with task-specific algorithms provided separately.The paper identifies this operator assumption in its algorithmic formulation.
  • 3.2. Fast Convergence Principle of CCDF: Forward diffusion increases initialization error, but conditional reverse diffusion decreases it exponentially faster through stochastic contraction.This trade-off yields a shortcut path and a suitable intermediate timestep N′.
  • 3.2. Fast Convergence Principle of CCDF: For any 0<µ≤1, a minimum N′ exists such that the final error satisfies ε̄0,r≤µε0, and N′ decreases as ε0 becomes smaller.The theorem formalizes that shorter sampling paths can achieve a tighter error bound.
  • 3.2. Fast Convergence Principle of CCDF: Pre-trained neural-network initialization enables smaller t0 values and, for MRI reconstruction, reaches t0=0.02 while outperforming score-MRI with 50× acceleration.The experiments cover super-resolution, inpainting, and MRI reconstruction, using NN-initialized estimates unless otherwise specified.

4. Experiments

Experiments evaluate CCDF on super-resolution, inpainting, and MRI reconstruction, varying initialization and reverse-diffusion duration. Across these tasks, CCDF maintains or improves reconstruction quality with substantially fewer iterations than full reverse diffusion.

  • Experimental settings: CCDF is evaluated on super-resolution, inpainting, and MRI reconstruction using quantitative and qualitative comparisons.The experiments use AFHQ and FFHQ face datasets for vision tasks and fastMRI for MRI reconstruction.
  • Initialization and convergence: Small initialization error enables stable reverse-diffusion convergence at small t0, whereas random initialization requires larger t0 to reach a feasible solution.Figure 3 compares random, vanilla, and neural-network initialization through their associated error scales.
  • Super-resolution: CCDF preserves or improves FID as iterations decrease, while ILVR image quality degrades when using fewer iterations.The comparison uses the ×8 super-resolution task and reports CCDF against ILVR under reduced sampling budgets.
  • Super-resolution: CCDF achieves results on par with or better than ILVR while reducing diffusion steps from 1000 to 100, 200, or 300 for ×4, ×8, and ×16 super-resolution.The corresponding starting times are t0 = 0.1, 0.2, and 0.3, respectively.
  • Inpainting: Inpainting with CCDF reconstructs realistic images within 20 steps and outperforms score-SDE and SN-PatchGAN in FID.The compared full reverse-diffusion baseline uses 20 steps from T = 1, while CCDF uses 20 steps from t0 = 0.2.
  • MRI reconstruction: MRI reconstruction reaches strong results with t0 = 0.02 and 20 iterations, outperforming score-POCS despite its full reverse diffusion.The paper reports that score-POCS with 20 iterations produces an infeasible noisy image, whereas CCDF remains feasible.

5. Discussion

The paper distinguishes CCDF from SDEdit by applying forward diffusion to inverse problems within a rigorous acceleration framework. It also identifies initialization quality and task-dependent corruption as central factors in choosing the starting time.

  • Discussion: CCDF differs from SDEdit by rigorously showing that better initialization can accelerate reverse diffusion for inverse problems.The paper frames this as a distinct use of forward-diffused data rather than signal destruction for high-fidelity synthesis.
  • Limitations: The optimal t0 varies substantially across tasks and corruption levels because ε0 is not known beforehand.The authors therefore identify trial-and-error selection as a practical limitation and propose adaptive selection as future work.

6. Conclusion

The paper accelerates conditional diffusion for inverse problems by exploiting stochastic contraction and initializing reverse diffusion from forward-diffused data. It reports acceleration together with increased stability and performance across diverse applications.

  • Conclusion: CCDF replaces random Gaussian initialization with forward-diffused data from a better initialization, such as a one-step neural-network correction.The method studies stochastic contraction to justify this shortcut path theoretically.
  • Conclusion: Theoretical analysis and diverse applications support acceleration alongside increased stability and performance for conditional diffusion inverse problems.

Supplementary Material

The supplementary material defines contraction and states a stochastic-contraction theorem for noisy difference equations with contracting drift and bounded diffusion. Its corollaries extend the mean-square bound to non-expansive data-consistency mappings.

  • Mathematical Preliminaries: A contraction mapping satisfies ||f(x)−f(y)|| ≤ λ||x−y|| for a factor λ below one.The supplementary text relates this property to λ-Lipschitz continuity.
  • Stochastic Contraction: The stochastic difference-equation theorem assumes contracting f and bounded Tr(g(x,i)Ig(x,i)) across states and iterations.The system includes independent zero-mean unit-variance Gaussian noise vectors.
  • Stochastic Contraction: The theorem bounds trajectory disagreement by a noise-dependent term plus λ^{2i} times the initial disagreement.The initial-condition contribution therefore decays geometrically with iteration index i.
  • Data Consistency: With a non-expansive linear data-consistency mapping, the same mean-square bound holds with the noise term scaled by τ.The corollary defines τ from Tr(A^T A)/n.
  • Proof: The proof composes the linear data-consistency map with the stochastic update and bounds its Jacobian and noise covariance contributions.
  • Score Conditions: The score-function appendix derives the contraction condition from the parameterized score’s Jacobian, including the Gaussian forward-diffusion representation.The forward process is written as x_i = a_i x_0 + b_i z with z sampled from a standard Gaussian.

B. Proof of Theorem 1

The proof analyzes the reverse process after truncating its starting time to t0, then derives contraction rates for DDPM, VE-SDE, and DDIM parameterizations. These rates support fewer reverse-diffusion steps, with DDIM eliminating the noise contribution.

  • Setup: Starting reverse diffusion at t0 changes the step count from N to N′ = Nt0 while preserving the stochastic difference equation’s contraction property.The reverse index is reparameterized into a forward-direction index for analysis.
  • DDPM: For DDPM, the reverse-step Jacobian yields a contraction rate whose maximum is below one under 0 < αi, ¯αi < 1.The proof uses the decreasing behavior of ¯αi.
  • VE-SDE: For VE-SDE, the reverse-step Jacobian is governed by the ratio of adjacent noise-level differences, with σi increasing in i.
  • DDIM: DDIM can be reparameterized as a discrete VE-SDE without noise terms, so its contraction analysis uses the corresponding deterministic formulation.The supplementary text states that the noise constant becomes C = 0.

C. Proof of Theorem 2

The proof bounds the reverse conditional diffusion error using a recursive stochastic-contraction inequality, showing that sufficiently small initialization error requires fewer reverse steps. It applies this argument to DDPM, SMLD, and DDIM schedules under stated variance and non-expansiveness conditions.

  • Stochastic contraction: Under uniformly bounded contraction and noise constants, the recursion reduces to the stated finite-step bound.The uniform bounds replace step-dependent λ_j and C_j with λ and C.
  • Stochastic contraction: The recursive bound decomposes error into a contracted initial-error term and accumulated stochastic terms.The contraction factors multiply across reverse steps, while noise contributes a weighted sum.
  • DDPM: For DDPM, normalized images give ε0 ≤ n, and the chosen N′ guarantees the reverse-diffusion error is at most μ ε0.The proof separately bounds the propagated error and accumulated noise, each contributing at most half of the target bound.
  • DDPM: The DDPM contraction factor is bounded exponentially, yielding a sufficient condition on N′β_N′ and showing that N′ decreases as ε0 decreases.The proof explicitly concludes that better initialization permits fewer reverse iterations.
  • SMLD and DDIM: For SMLD and DDIM, variance-schedule conditions similarly ensure contraction to μ ε0, with the required N′ decreasing as initialization error decreases.The DDIM argument uses C_j = 0, while the SMLD argument assumes bounded minimum and maximum variances.

D. Implementation detail

CCDF starts reverse diffusion at N′ = Nt0 after a forward diffusion step, then alternates reverse updates with measurement-consistency projections. The implementation instantiates this scheme for VP/DDPM, DDIM, and VE/PC MRI reconstruction.

  • Core procedure: CCDF uses N′ = Nt0 < N reverse steps, where t0 determines the acceleration level.The initial state is obtained by forward-diffusing an estimate before reverse sampling begins.
  • Data consistency: Super-resolution uses a normalized blur operator, while inpainting uses a diagonal measured-location operator, both with σmax(A) ≤ 1.These operators implement the non-expansive data-consistency requirement.
  • VP diffusion: The VP implementation alternates a learned reverse-diffusion update with measurement consistency for super-resolution and inpainting.The consistency step replaces the measured component using the forward-propagated observation.
  • DDIM: The same trained score function can be paired with DDIM by modifying the unconditional update step.Algorithm 2 combines forward initialization, DDIM reverse updates, and measurement consistency.
  • MRI reconstruction: MRI reconstruction uses VE-SDE predictor-corrector sampling and Fourier-domain measurement consistency.The consistency operator inserts the sampled k-space measurements through F^-1DF, whose norm is non-expansive because the Fourier transform is orthonormal.

E.1. Super-resolution

Experiments evaluate CCDF for super-resolution and compare it with diffusion and feed-forward baselines across datasets and sampling schedules. The reported results show preserved or improved reconstruction quality with substantially fewer iterations.

  • Comparison study: 100, 200, and 300 diffusion steps for ×4, ×8, and ×16 SR respectively match the comparison using 1000 ILVR steps without quality degradation.The authors attribute this result to CCDF’s contraction property.
  • CCDF + DDIM: 5 reverse diffusion iterations produce high-fidelity ×8 and ×16 SR samples when CCDF is combined with DDIM.The experiment uses FFHQ and reduces t0 while fixing N = 50 for the proposed method.
  • Quantitative comparison: With fewer sampling steps, ILVR+DDIM FID worsens exponentially, whereas CCDF improves the metric relative to full diffusion with 50 total steps.The same qualitative trend is reported for DDPM experiments.
  • Inpainting: For inpainting, the best t0 depends on mask size: t0 = 0.1 works for relatively small boxes, while larger boxes require more diffusion steps.The comparison uses FID scores across varying t0 values.
  • Inpainting: CCDF improves imperfect neural-network initialization for inpainting and produces highly realistic reconstructions compared with SN-PatchGAN and 1000-step score-SDE.The neural network initialization serves as the starting point for CCDF.

F. Validity of assumption

The paper justifies its score-function assumption theoretically near zero noise and tests it numerically on fastMRI. The measured score error remains low over the regime used by the contraction analysis but increases at high noise levels.

  • Theoretical justification: The optimal denoising-autoencoder reconstruction function asymptotically yields the score function as σ approaches zero.Because σ approaches zero as t approaches zero, the assumed score relation holds near t = 0.
  • Numerical validation: Figure F.1 measures the average score-function error norm on 1K fastMRI test samples across diffusion time.The figure evaluates the discrepancy between the learned score and the conditional score target.
  • Scope of validity: The error norm stays mostly low but increases when t > 0.5, while most contraction analysis uses t ∈ [ε, 0.5].The authors therefore describe the assumption as practical for the analyzed time regime.
  • Additional evaluations: Additional figures test CCDF across SR factors and inpainting masks while varying t0, extending the empirical validity checks beyond the fastMRI score-error experiment.These figures compare reconstruction outputs across datasets and task settings.
Loading 2112.05146v2…