PaperlayerView exact revision on arXiv

Source-linked AI summary

Denoising Diffusion Restoration Models

Bahjat Kawar, Michael Elad, Stefano Ermon, Jiaming Song

arXiv:2201.11793v3eess.IVcs.CVcs.LG

TL;DR

Image-restoration posterior samplers face a trade-off between the generality of unsupervised methods and the efficiency of supervised, problem-specific training. DDRM uses variational inference with pre-trained diffusion models to solve general linear inverse problems efficiently, achieving competitive or superior results across tasks and datasets while remaining applicable beyond the observed ImageNet distribution.

  • Problem

    Unsupervised posterior samplers handle general inverse problems but are inefficient, whereas efficient supervised approaches are problem-specific.

  • Method

    DDRM is an unsupervised posterior-sampling solver that conditions pre-trained unconditional or class-conditional diffusion models on measurements for general linear inverse problems.

  • Results

    DDRM achieves competitive performance in as low as 20 NFEs, mostly outperforms neural-network baselines on ImageNet restoration, and works on natural images outside the observed training distribution.

  • Takeaways & Limitations

    A single pre-trained diffusion model can support efficient posterior sampling across denoising, super-resolution, deblurring, inpainting, and colorization.

  • Takeaways & Limitations

    Future work includes extending DDRM to nonlinear inverse problems, unknown degradation operators, and improved self-supervised training.

Abstract

from arXiv · show

Many interesting tasks in image restoration can be cast as linear inverse problems. A recent family of approaches for solving these problems uses stochastic algorithms that sample from the posterior distribution of natural images given the measurements. However, efficient solutions often require problem-specific supervised training to model the posterior, whereas unsupervised methods that are not problem-specific typically rely on inefficient iterative methods. This work addresses these issues by introducing Denoising Diffusion Restoration Models (DDRM), an efficient, unsupervised posterior sampling method. Motivated by variational inference, DDRM takes advantage of a pre-trained denoising diffusion generative model for solving any linear inverse problem. We demonstrate DDRM's versatility on several image datasets for super-resolution, deblurring, inpainting, and colorization under various amounts of measurement noise. DDRM outperforms the current leading unsupervised methods on the diverse ImageNet dataset in reconstruction quality, perceptual quality, and runtime, being 5x faster than the nearest competitor. DDRM also generalizes well for natural images out of the distribution of the observed ImageNet training set.

1 Introduction

Image restoration can be formulated as linear inverse problems, but existing posterior-sampling approaches trade generality for efficiency. DDRM addresses this trade-off with an efficient unsupervised diffusion-based solver that works across inverse problems and measurement-noise settings.

  • Super-resolution, deblurring, inpainting, colorization, and compressive sensing can be posed as linear inverse problems with potentially noisy measurements.
  • Existing approaches combine learned image priors with degradation-model likelihoods, then often solve posterior optimization or sampling iteratively.
  • DDRM uses a denoising diffusion generative model conditioned on measurements and the inverse problem to sample the posterior through a variational-inference objective.
  • DDRM can achieve competitive results in as low as 20 NFEs while using pre-trained unconditional diffusion models across linear inverse problems.
  • On ImageNet, DDRM mostly outperforms neural-network baselines for noiseless super-resolution and deblurring in PSNR and KID, while requiring at least 50× fewer NFEs when second-best.

2 Background

Linear inverse problems recover signals from measurements using a known degradation model and can incorporate learned generative priors. Diffusion models provide a denoising-based generative framework that supports this prior modeling.

  • A linear inverse problem seeks to recover x from measurements y produced by a known linear degradation matrix H and additive Gaussian noise.
  • A learned prior pθ(x) and likelihood p(y|x) define a posterior pθ(x|y) that can be sampled or optimized to recover the signal.
  • Diffusion models learn a data distribution through a Markov chain that progressively transforms noisy states into an image sample.
  • For Gaussian diffusion processes, the variational lower-bound objective reduces to a denoising autoencoder objective.
  • The diffusion network is trained to recover a noiseless observation from a noisy state, with coefficients determined by the diffusion distribution.

3 Denoising Diffusion Restoration Models

DDRM resolves the efficiency–generality dilemma by conditioning a diffusion process on measurements for general linear inverse problems while reusing pre-trained diffusion models. Its spectral-space construction separates measured information from missing content and supports efficient sampling.

  • 3 Denoising Diffusion Restoration Models: Unsupervised posterior samplers apply to general problems but are inefficient, whereas supervised solvers are efficient but problem-specific.
  • 3 Denoising Diffusion Restoration Models: DDRM is an unsupervised solver for general linear inverse problems with or without measurement noise and competitive performance against unsupervised solvers.
  • 3.1 Variational Objective for DDRM: DDRM defines a measurement-conditioned Markov diffusion chain whose final state x0 is the restored output and whose variational distribution yields an ELBO objective.
  • 3.2 A Diffusion Process for Image Restoration: Using the SVD of H, DDRM performs diffusion in spectral space, denoising noisy measured components and synthesizing information absent from the measurements.
  • 3.2 A Diffusion Process for Image Restoration: The method assumes m ≤ n for useful degradation models, while also stating that it works for m > n.
  • 3.2 A Diffusion Process for Image Restoration: The spectral-space updates use the measurement information when singular values are nonzero and unconditional generation when corresponding singular values are zero.
  • 3.3 “Learning” Image Restoration Models: Pre-trained DDPM models approximate the DDRM objective, allowing the same inverse-problem-independent diffusion model to be reused by modifying H and its SVD.
  • 3.3 “Learning” Image Restoration Models: DDRM can use skipped diffusion steps, requiring about 20 NFEs instead of about 1000 for SNIPS, while supporting denoising, super-resolution, deblurring, inpainting, and colorization.

4 Related Work

Related work develops unsupervised inverse-problem solvers using learned priors, but many approaches rely on iterative procedures or narrow data settings. Diffusion-based methods improve noise handling, while DDRM targets greater efficiency and broader applicability.

  • Unsupervised restoration methods use clean-image data during training while leaving the degradation model unspecified until inference.
  • Many solvers incorporate neural denoisers into iterative optimization schemes such as steepest descent, fixed-point methods, or ADMM.
  • Latent-space generative methods can perform impressively on specific image classes but have not shown broad success on diverse datasets such as ImageNet.
  • Diffusion-based inverse-problem methods often handle significant measurement noise but typically require hundreds or thousands of iterations and lack validation on diverse datasets.
  • DDRM addresses efficiency and noisy measurements, while related methods such as ILVR and a score-based solver have narrower noise-handling capabilities.

5 Experiments

Experiments evaluate DDRM across multiple datasets, inverse problems, resolutions, and noise settings using pretrained diffusion models. On ImageNet, DDRM achieves strong restoration results with few steps, and its advantages increase under measurement noise while extending to varied samples and out-of-distribution images.

  • Experimental Setup: Experiments use pretrained diffusion models from CelebA-HQ, LSUN, and ImageNet, including class-conditional DDRM-CC for models requiring labels.The models cover 256 × 256 and 512 × 512 images, with ImageNet validation images used for corresponding models.
  • Experimental Setup: The benchmark includes super-resolution, deblurring, colorization, and inpainting, with optional additive white Gaussian measurement noise.Super-resolution uses block averaging, deblurring uses a uniform kernel, colorization averages RGB channels, and inpainting masks image regions or drops 50% of pixels.
  • Quantitative Experiments: Evaluation emphasizes PSNR and SSIM for faithfulness and KID for resulting image quality on diverse ImageNet images.The reported ImageNet experiments focus on 256 × 256 images and average metrics across the evaluation set.
  • Quantitative Experiments: With only 20 steps, DDRM outperforms all baselines across both noiseless 4× super-resolution and deblurring in every reported metric except noiseless-deblurring KID.SNIPS achieves better KID in that exception but requires 50× more NFEs; DGP and DDRM-CC use ground-truth class labels.
  • Quantitative Experiments: Under significant measurement noise, DGP, RED, and SNIPS fail to produce viable results, making DDRM’s comparative advantage more substantial.The noisy 4× super-resolution comparison uses σy = 0.05.
  • Qualitative Experiments: DDRM produces high-quality reconstructions across tested datasets and problems, samples multiple outputs for one input, and restores out-of-distribution USC-SIPI images.The USC-SIPI examples are 256 × 256 images restored with an ImageNet diffusion model.

6 Conclusions

DDRM is a general sampling-based solver for linear inverse problems that uses diffusion generative models as learned priors and requires few NFEs. It works across several restoration tasks, including noisy settings and images outside the observed training distribution, while non-linear and unknown-degradation cases remain future work.

  • DDRM uses unconditional or class-conditional diffusion generative models as learned priors for general linear inverse problems.
  • 20 NFEs can suffice for DDRM, compared with 1000 NFEs for SNIPS.
  • DDRM scales to denoising, super-resolution, deblurring, inpainting, and colorization.
  • DDRM succeeds on natural images outside the distribution of the observed training set and on inverse problems with significant noise.
  • Future work includes non-linear inverse problems, unknown degradation operators, and self-supervised training techniques.

C Proofs

The proofs establish the Gaussian structure of the conditional distributions used by DDRM and show that its ELBO objective can be rewritten as a DDPM or DDIM denoising objective. They also justify using pre-trained diffusion models and efficient SVD-based computations for several degradation operators.

  • Propositions and theorems: DDRM’s conditional distributions satisfy the proposition’s stated Gaussian relationships, enabling marginalization over future diffusion states.
  • Propositions and theorems: The ELBO objective of DDRM can be rewritten in the form of the DDPM or DDIM objective.
  • Proof structure: The proof decomposes the objective across spectral dimensions and cases determined by singular values and noise levels.
  • Efficient SVD computations: For denoising and inpainting, the degradation SVD can be applied with linear space complexity using identity, permutation, and slicing operations.
  • Efficient SVD computations: For super-resolution and colorization, structured operators and Kronecker products provide corresponding SVD constructions.

D.5 Deblurring

The deblurring implementation exploits separable convolution and Kronecker-product structure to construct and apply the degradation SVD efficiently. On zero-padded ImageNet deblurring, DDRM improves substantially over the pseudo-inverse in the noiseless case, while noise makes the pseudo-inverse unusable.

  • D.5 Deblurring: Separable blurring is represented as H = A_r ⊗ A_c, allowing the 2D operator’s SVD to be built from 1D SVDs.
  • D.5 Deblurring: Kronecker-product structure and permutations yield a valid, sorted SVD for the blurring operator.
  • D.5 Deblurring: Θ(n) space complexity suffices to simulate multiplication by the SVD factors without storing the full matrices.
  • D.5 Deblurring: 35.64dB PSNR for DDRM exceeds the 32.41dB achieved by the pseudo-inverse on noiseless ImageNet-1K deblurring.
  • D.5 Deblurring: With noise, the pseudo-inverse amplifies noise and does not produce a valid output; the reported zero-padded setup also applies to strided convolutions.

E Ablation Studies on Hyperparameters

The ablations examine noise-injection hyperparameters and timestep schedules. They identify a PSNR–KID trade-off for η and find that 20 and 100 timesteps have similar performance under matched settings.

  • η and η_b: Increasing η_b generally raises PSNR and lowers KID, while increasing η generally raises both PSNR and KID except at η = 1.0.
  • η and η_b: η_b = 1 and η = 0.85 are selected to balance PSNR and KID.
  • Timestep schedules: The wall-clock time is roughly linear in the number of NFEs, motivating timestep reduction.
  • Timestep schedules: DDRM with 20 or 100 timesteps has similar performance when other hyperparameters are identical, with DDRM (20) slightly better in FID and KID.

F Experimental Setup of DGP, RED, and SNIPS

The experiments compare DGP, RED, and SNIPS on ImageNet 1K super-resolution and deblurring, using shared or specified diffusion-model settings and varying iteration counts. DDRM-20 is compared with these baselines in reconstruction quality, perceptual quality, efficiency, and noisy-measurement behavior.

  • Experimental setup: DGP, RED, and SNIPS are evaluated on 256 × 256 ImageNet 1K images for 4× super-resolution and deblurring without measurement noise.The evaluation follows the setup described for the main-paper comparison.
  • RED: RED uses the shared diffusion model as its denoiser, operates in the luminance channel, and is tuned with RED-SD and λ = 500.For noiseless measurements, σ0 is set to 0.01 because RED divides by it.
  • RED: RED is initialized from bicubic-upsampled images for super-resolution and blurry images for deblurring, with reported comparisons using 100 and 500 steps respectively.The best PSNR is selected for each problem across tested step counts.
  • Comparison: DDRM with 20 steps outperforms RED in PSNR and perceptual quality, while RED reaches near-best super-resolution PSNR in 20 steps.The comparison identifies a quality advantage for DDRM despite the shorter DDRM run.
  • SNIPS: SNIPS uses the same diffusion model with τ = 1, while its direct SVD computation is replaced by an efficient implementation for 256 × 256 comparisons.Higher τ values are not explored because runtime would become less comparable.
  • Runtime: Runtime is approximately linear in NFEs, with DDRM, RED, and SNIPS near 0.09 s/it and DGP near 0.11 s/it on an Nvidia RTX 3080 GPU.Because per-iteration differences are negligible, NFEs are used as a hardware-independent runtime proxy; all experiments use pretrained models without training.

H ILVR as a special case of DDRM

The ILVR update can be expressed in the spectral space of the degradation operator and matches a special DDRM case for noiseless measurements. The equivalence holds with η = ηb = 1, while ILVR’s experiments specifically use bicubic downscaling.

  • ILVR formulation: ILVR applies iterative updates using a diffusion model that predicts x from x_{t+1} and t + 1 under noiseless measurements y = Hx.The formulation assumes a variance-exploding diffusion model, with the variance-preserving case equivalent by construction.
  • Spectral derivation: The updates are rewritten through the SVD of H, using the diagonal structure of Σ†Σ to simplify the spectral-space expression.The derivation uses orthogonal matrices U and V and the pseudoinverse Σ†.
  • Spectral derivation: Σ†Σ contains zeros for zero singular values and ones otherwise, while Σ† has zero rows where singular values vanish.These properties support the simplification of the update expression.
  • Equivalence: The resulting distribution is exactly Equation 8 of the main paper when η = ηb = 1 and σy = 0.This identifies the noiseless ILVR distribution with a specific DDRM setting.
  • Equivalence: ILVR initializes xT from a Gaussian distribution, whereas DDRM uses Equation 7; their variances match and the mean difference has negligible effect because the variance is much larger.The resulting ILVR form is therefore a DDRM instance for noiseless linear inverse problems.
  • Scope: ILVR experiments tested bicubic downscaling matrices, although the method can theoretically handle any linear degradation H when measurements are noise-free.The stated scope excludes noisy measurements from this theoretical extension.

I Additional Results

Additional figures illustrate DDRM across datasets, inverse problems, noise levels, and sampled-solution diversity. The examples include face, bedroom, cat, USC-SIPI, ImageNet super-resolution, deblurring, and denoising results.

  • Additional results: Additional experiments cover 4× super-resolution and deblurring with noise, alongside figures showing DDRM’s versatility and sample diversity.The added examples extend the reported evaluations across inverse problems, datasets, and noise levels.
  • Face restoration: Figures 7 and 8 show pairs of recovered 256 × 256 face images from noisy low-resolution or degraded inputs with σy = 0.1.Both examples use a 20-step DDRM.
  • Sample diversity: Figure 9 shows six recovered face samples together with the sampled solution’s mean and standard deviation scaled by 4.The degraded inputs use σy = 0.1 and a 20-step DDRM.
  • Cross-dataset restoration: Figures 10–12 show recovered bedroom, cat, and USC-SIPI images using 20-step DDRM with σy = 0.05.The USC-SIPI examples use an ImageNet diffusion model.
  • ImageNet restoration: Figures 13 and 14 present uncurated ImageNet triplets for noisy 4× super-resolution and deblurring, ordered as original, degraded, and restored images.Both tasks use σy = 0.05 and DDRM-20 on 256 × 256 ImageNet 1K images.
  • Denoising: Figure 15 compares denoising at σy = 0.75 and shows DDRM’s denoiser restoring more fine details, such as hair, than an MMSE denoiser.The denoiser is the diffusion function used by DDRM, with t chosen to minimize |σt − σy|.
Loading 2201.11793v3…