PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model

Yinhuai Wang, Jiwen Yu, Jian Zhang

arXiv:2212.00490v2cs.CV

TL;DR

Existing image restoration methods are often task-specific or struggle with the ill-posed balance between data consistency and realistic detail. DDNM uses an off-the-shelf diffusion model to refine only null-space contents for zero-shot linear restoration, while DDNM+ extends it to noisy and difficult real-world cases. The methods achieve state-of-the-art zero-shot restoration across diverse tasks and support applications such as old photo restoration.

  • Problem

    Image restoration is ill-posed, and existing approaches are limited in generalizing across degradation operators while balancing data consistency and realistic details.

  • Method

    DDNM fixes the range-space implied by the observation and refines only null-space contents during reverse diffusion using a pretrained diffusion model, without extra training or network modification.

  • Results

    DDNM outperforms state-of-the-art zero-shot methods across super-resolution, colorization, compressed sensing, inpainting, and deblurring, while DDNM+ supports noisy restoration and improves hard-task quality.

  • Takeaways & Limitations

    The framework supports diverse linear and complex real-world restoration tasks, including old photo restoration, without degradation-specific training.

  • Takeaways & Limitations

    DDNM fails on noisy restoration and can yield poor realness for particular pseudo-inverses, motivating DDNM+ and its time-travel enhancement.

Abstract

from arXiv · show

Most existing Image Restoration (IR) models are task-specific, which can not be generalized to different degradation operators. In this work, we propose the Denoising Diffusion Null-Space Model (DDNM), a novel zero-shot framework for arbitrary linear IR problems, including but not limited to image super-resolution, colorization, inpainting, compressed sensing, and deblurring. DDNM only needs a pre-trained off-the-shelf diffusion model as the generative prior, without any extra training or network modifications. By refining only the null-space contents during the reverse diffusion process, we can yield diverse results satisfying both data consistency and realness. We further propose an enhanced and robust version, dubbed DDNM+, to support noisy restoration and improve restoration quality for hard tasks. Our experiments on several IR tasks reveal that DDNM outperforms other state-of-the-art zero-shot IR methods. We also demonstrate that DDNM+ can solve complex real-world applications, e.g., old photo restoration.

1 INTRODUCTION

Image restoration is difficult because degraded observations are ill-posed, while existing model-based and end-to-end approaches struggle to combine realistic details, generalization, and flexibility across degradations. DDNM addresses this gap with a zero-shot diffusion framework that refines null-space contents for diverse linear restoration tasks.

  • Image restoration seeks a high-quality image from a degraded observation, but its ill-posed nature makes the problem long-standing.
  • Model-based methods enforce data fidelity with hand-designed priors, yet those priors often fail to produce realistic details.
  • End-to-end neural restoration requires paired degraded and ground-truth images and can perform poorly under degradation-domain shift.
  • Generative-prior methods can struggle to balance realness and data consistency, motivating a different decomposition of the restoration problem.
  • DDNM refines only null-space contents during reverse diffusion, using an off-the-shelf diffusion model without extra training or network modification.
  • DDNM outperforms state-of-the-art zero-shot restoration methods across diverse tasks, while DDNM+ supports noisy restoration and challenging real-world applications.

2 BACKGROUND

DDPM learns to reverse a gradual noising process, transforming random noise into samples from the image distribution. Its reverse process uses a neural network to predict the noise at each timestep.

  • DDPM defines T-step forward and reverse processes, with the forward process gradually adding random noise to data.
  • The forward process represents a noised image x_t through predefined factors β_t and a Gaussian distribution.
  • The reverse process samples x_t−1 from the posterior p(x_t−1|x_t, x_0), derived using Bayes theorem.
  • DDPM trains a neural network Z_θ to predict the noise ϵ_t at each timestep from a noised image and timestep.
  • Iterative reverse sampling converts random noise x_T into clean images distributed like the training dataset.
  • For linear operators, A†A projects onto the range-space and (I−A†A) projects onto the null-space, yielding x ≡ A†Ax +(I−A†A)x.

3 METHOD

DDNM combines linear range-null decomposition with diffusion sampling: it fixes the range-space to preserve data consistency and iteratively generates compatible null-space content. DDNM+ extends this procedure to noisy restoration and difficult cases where the measured range-space is insufficient for globally realistic results.

  • Image restoration imposes consistency with the degradation operator and realism under the ground-truth image distribution.
  • A general consistent solution is x̂ = A†y +(I−A†A)x̄, so the null-space component can vary without changing the measurement.
  • DDNM: DDNM estimates a clean image at each diffusion timestep, fixes its range-space to A†y, and leaves the null-space for diffusion refinement.
  • DDNM: Iterative sampling removes disharmony between fixed range-space contents and generated null-space contents, producing a final result that satisfies consistency.
  • DDNM: DDNM uses a pretrained denoising network to solve diverse linear degradations without task-specific training or optimization.
  • Practical operators: Practical pseudo-inverses support inpainting, colorization, and super-resolution, while composed operators extend the method to complex restoration such as old photos.
  • DDNM+: DDNM+ addresses noisy restoration by scaling range-space correction and injected noise, with parameters chosen to preserve the diffusion noise definition.
  • DDNM+: DDNM performs poorly when certain pseudo-inverses yield overly local range-space contents, including large-scale SR, low-ratio compressed sensing, and large-mask inpainting.

4 EXPERIMENTS

The experiments evaluate DDNM and DDNM+ across standard, noisy, difficult, and real-world linear image-restoration tasks. DDNM outperforms prior zero-shot methods, while DDNM+ improves denoising, restoration robustness, and challenging applications.

  • Evaluation on DDNM: Experiments compare DDNM with state-of-the-art zero-shot methods on five noise-free tasks: super-resolution, deblurring, colorization, compressed sensing, and inpainting.The evaluation uses bicubic downsampling, Gaussian blur, average grayscale, Walsh-Hadamard sampling, and text masks, respectively.
  • Evaluation on DDNM: DDNM achieves significantly better Consistency and Realness than ILVR, RePaint, and DDRM using the same pretrained denoising models and sampling steps.It also far exceeds previous GAN-prior methods DGP and PULSE.
  • Evaluation on DDNM+: DDNM+ decently removes observation noise, whereas DDNM fully inherits the noise contained in the degraded observation.This comparison evaluates denoising performance on three noisy restoration tasks with the time-travel trick disabled.
  • Evaluation on DDNM+: The time-travel trick significantly improves overall performance, especially Realness measured by FID, on 32× super-resolution, colorization, and 10% compressed sensing.DDNM+ and DDNM use roughly equal sampling steps and computational consumption in this comparison.
  • Evaluation on DDNM+: DDNM+ robustly handles arbitrary scales of linear image-restoration tasks and shows excellent robustness across diverse tasks in qualitative comparisons.The paper presents this as the first image-restoration method with this capability to the authors’ knowledge.
  • Real-world applications: DDNM+ is demonstrated on real-world colorization and old photo restoration using approximate linear degradation operators and pseudo-inverses.These applications are presented as challenging real-world examples.

5 RELATED WORK

Related image-restoration methods include supervised and zero-shot diffusion approaches, as well as null-space methods based on learned priors. Supervised diffusion methods require task-specific training, while prior zero-shot and null-space approaches address generality and consistency through different mechanisms.

  • Diffusion-based methods: Diffusion-based image-restoration methods are divided into supervised and zero-shot categories.This distinction organizes recent approaches by whether task-specific training is used.
  • Supervised methods: Supervised diffusion methods train task-specific models, often with synthetic image pairs, and therefore cannot generalize to different degradation operators.Examples include SR3 for super-resolution and a residual-prediction diffusion model for deblurring.
  • Zero-shot methods: Zero-shot diffusion methods guide reverse diffusion using unmasked regions, gradients, or low-frequency references for inpainting and general inverse problems.Representative methods include Song and Ermon’s approaches, ILVR, and RePaint.
  • Null-space methods: Null-space image-inverse methods learn or explore missing null-space contents using DNNs, GAN priors, or editable super-resolution procedures.These approaches include separate range/null-space learning and range-null decomposition applied to GAN-prior super-resolution.

6 CONCLUSION & DISCUSSION

The paper presents a unified zero-shot framework for linear image-restoration tasks and identifies extensions to other data types and output sizes. The Mask-Shift trick enables arbitrary desired output sizes while maintaining coherent reconstructions.

  • Conclusion and discussion: The framework offers a unified zero-shot approach to linear image-restoration tasks and may inform general inverse problems.The paper states that extension to video, audio, and point-cloud inverse problems is theoretically possible when suitable diffusion training data exist.

7 MASK-SHIFT TRICK

The Mask-Shift trick extends DDNM to arbitrary output sizes while preserving coherence across shifted image divisions. The method also retains low computational cost by avoiding costly optimization and, when possible, SVD.

  • 7 MASK-SHIFT TRICK: The Mask-Shift trick enables DDNM to solve restoration tasks with arbitrary desired output sizes, including 4K images.
  • 7 MASK-SHIFT TRICK: Naively dividing large inputs into tiles and concatenating independently restored outputs creates significant block artifacts between divisions.
  • 7 MASK-SHIFT TRICK: Overlapping shifted DDNM runs replace intermediate regions so the final reconstruction remains coherent between shifts.
  • Efficiency: DDNM has nearly the same consumption as the original diffusion models and avoids optimization for minimizing the data-consistency residual.
  • Efficiency: DDNM does not necessarily need SVD because hand-constructed A and A† for several tasks have negligible computation and memory cost.

B COMPARING DDNM WITH SUPERVISED METHODS

Compared with supervised restoration methods, DDNM is zero-shot across diverse tasks and more robust to degradation modes. Experiments also report significantly better performance on certain datasets and resolutions.

  • B COMPARING DDNM WITH SUPERVISED METHODS: DDNM is zero-shot for diverse tasks, whereas supervised methods require separate training for each task.
  • B COMPARING DDNM WITH SUPERVISED METHODS: DDNM is robust to degradation modes, while supervised methods have poor generalized performance.
  • B COMPARING DDNM WITH SUPERVISED METHODS: DDNM yields significantly better performance on certain datasets and resolutions, including ImageNet at 256x256.

C LIMITATIONS

DDNM+ extends restoration to noisy and real-world degradations, but the framework remains constrained by operator knowledge, diffusion inference speed, denoiser quality, randomness, and manual noise-level selection.

  • C LIMITATIONS: Existing diffusion models make DDNM inference slow, despite the method adding negligible computational cost.The limitation concerns the speed of the pretrained diffusion models rather than extra computation introduced by DDNM.
  • C LIMITATIONS: DDNM requires an explicit degradation operator, and approximating an unavailable operator may work but is not optimal.This requirement can be challenging for some restoration tasks.
  • C LIMITATIONS: The theoretical framework supports linear operators; nonlinear operators may lack linearly separable range and null spaces.The stated obstacle is that nonlinear operators may not satisfy the distributive property needed for the decomposition.
  • C LIMITATIONS: DDNM inherits diffusion randomness, which supports diversity but can sometimes produce undesirable results.The limitation is a variability trade-off rather than a complete failure of restoration.
  • C LIMITATIONS: Restoration quality is limited by the pretrained denoiser’s network capacity and training dataset.The paper notes that existing diffusion models do not outperform StyleGANs for 1024×1024 FFHQ/AFHQ synthesis.

G VISUALIZATION OF THE INTERMEDIATE RESULTS

DDNM’s intermediate states show that fixed range-space content provides meaningful structure early, while reverse diffusion progressively supplies null-space details and high-frequency information.

  • Intermediate results: The rectified clean estimate already contains meaningful content early because DDNM fixes the range-space component A†y.At t = 0, additional image details emerge beyond A†y; these details are identified as null-space contents.
  • Intermediate results: The reverse diffusion process restores images progressively from low-frequency contours to high-frequency details.
  • Comparison with related methods: DDNM operates on the noise-free x0|t domain rather than noisy xt, yielding strict range-space preservation and data consistency.
  • Comparison with related methods: RePaint and ILVR can be interpreted as special cases of range-null space decomposition under their respective degradation operators.
  • Comparison with related methods: Unlike DDRM’s SVD-dependent matrix formulation, DDNM supports arbitrary linear operator forms when AA†A = A is satisfied.The paper notes that pseudo-inverses can be obtained in multiple ways, including Fourier transforms for convolutional operators.

H.3 OTHER DIFFUSION-BASED IR METHODS

The paper contrasts DDNM with diffusion-based restoration methods and shows how its range-null decomposition can complement task-specific diffusion models while supporting additional constraints.

  • SR3: DDNM can be inserted into SR3, improving restoration quality by ensuring data consistency through range-null space decomposition.
  • SR3: SR3 requires paired data and a modified conditional denoiser, whereas DDNM remains fully zero-shot across diverse restoration tasks.
  • SDE: SDE guides each latent xt toward a condition-dependent objective before returning it to the reverse diffusion process.
  • Additional constraints: DDNM is compatible with additional constraint sources, including classifier guidance applied to diffusion models pretrained on ImageNet.
  • Noisy restoration: For noisy restoration, the method adjusts introduced noise so the total noise remains within the prescribed diffusion noise level.
  • Noisy restoration: The parameter η can improve robustness, but the paper states that this sacrifices some range-space information.

J ADDITIONAL RESULTS

Additional experiments provide quantitative comparisons with DDRM and visual results across clean, noisy, and real-world restoration settings without using the time-travel trick.

  • Quantitative results: Additional quantitative comparisons between DDNM and DDRM are reported in Table 5.
  • Clean restoration: Additional DDNM visual results cover CelebA and ImageNet across super-resolution, deblurring, inpainting, colorization, and compressed sensing.
  • Noisy restoration: DDNM+ visual results demonstrate noisy image restoration on CelebA and ImageNet.
  • Experimental setting: The additional visual results in these experiments do not use the time-travel trick.
  • Real-world restoration: DDNM is also applied to restoring real-world photos collected from the internet.
Loading 2212.00490v2…