PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiscale Structure Guided Diffusion for Image Deblurring

Mengwei Ren, Mauricio Delbracio, Hossein Talebi, Guido Gerig, Peyman Milanfar

arXiv:2212.01789v3cs.CV

TL;DR

Image-conditioned diffusion models’ robustness to out-of-domain blurry images is unclear, particularly given their lack of intermediate constraints. This paper adds regression-derived multiscale structure guidance to the diffusion model. Its single-dataset-trained model reports stronger perceptual quality across test sets while maintaining competitive distortion metrics.

  • Problem

    Image-conditioned DPMs lack intermediate constraints, and their robustness to out-of-domain images is unclear despite synthetic training data differing from real-world blurry images.

    Source: §1 · p. 1

  • Method

    The method adds multiscale guidance from regression-network features trained to predict sharp-image structure, conditioning the icDPM alongside the blurry input.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across four test sets, the model significantly improves perceptual scores while remaining within < 0.08 dB of the best PSNR and < 0.001 of the best SSIM with sample averaging.

    Source: §4.4 · p. 7

  • Takeaways & Limitations

    The authors report more consistent deblurring with fewer artifacts on unseen data, alongside high perceptual realism.

    Source: §4.4 · p. 7

    The model is restricted to GoPro training data, which does not adequately cover real-world cases such as saturated regions in poor light.

    Source: §5 · p. 9

Abstract

from arXiv · show

Diffusion Probabilistic Models (DPMs) have recently been employed for image deblurring, formulated as an image-conditioned generation process that maps Gaussian noise to the high-quality image, conditioned on the blurry input. Image-conditioned DPMs (icDPMs) have shown more realistic results than regression-based methods when trained on pairwise in-domain data. However, their robustness in restoring images is unclear when presented with out-of-domain images as they do not impose specific degradation models or intermediate constraints. To this end, we introduce a simple yet effective multiscale structure guidance as an implicit bias that informs the icDPM about the coarse structure of the sharp image at the intermediate layers. This guided formulation leads to a significant improvement of the deblurring results, particularly on unseen domain. The guidance is extracted from the latent space of a regression network trained to predict the clean-sharp target at multiple lower resolutions, thus maintaining the most salient sharp structures. With both the blurry input and multiscale guidance, the icDPM model can better understand the blur and recover the clean image. We evaluate a single-dataset trained model on diverse datasets and demonstrate more robust deblurring results with fewer artifacts on unseen data. Our method outperforms existing baselines, achieving state-of-the-art perceptual quality while keeping competitive distortion metrics.

1. Introduction

Image deblurring is ill-posed, and image-conditioned diffusion models’ robustness on out-of-domain images remains unclear. The paper adds multiscale structure guidance to improve icDPM robustness and reports stronger perceptual quality with competitive distortion metrics across test sets.

Motivation

  • Image deblurring is an ill-posed inverse problem, while regression methods can produce over-smoothed outputs and perceptual methods may trade distortion for visual quality.GANs can improve perception but may suffer from instability, mode collapse, and artifacts.Source: §1 · p. 1
  • Image-conditioned DPMs use pairwise supervised data but impose no degradation model or intermediate constraints, leaving their robustness to domain shift unclear.Synthetic training data can differ from real-world blurry images.Source: §1 · p. 1

Contribution

  • The method adds multiscale sharp-structure guidance at intermediate icDPM layers, using learned regression features to inform restoration alongside the blurry input.The guidance is intended to retain salient sharp structures and improve generalization.Source: §1 · p. 2
  • The authors report more plausible deblurring and fewer artifacts on unseen data, with state-of-the-art perceptual quality and distortion metrics on par with existing benchmarks.The contribution is evaluated using a model trained on a single dataset.Source: §1 · p. 2

2. Related Works

Prior deblurring work spans structural priors, supervised image-to-image models, and image-conditioned diffusion. The paper focuses on domain generalization through image conditioning rather than explicit adaptation between specified domains.

Image deblurring methods

  • Traditional deblurring uses variational optimization and handcrafted structural priors, such as edges and shapes, to preserve important image features while removing blur.The paper’s design principle is a learned guidance acting as an implicit structural bias.Source: §2 · p. 2
  • Deep deblurring uses pixel-wise losses, which can yield over-smoothed images, while perceptual and adversarial losses improve visual fidelity with possible distortion-score compromises.Architectural directions include attention, multiscale designs, and multistage frameworks.Source: §2 · p. 2

Diffusion models

  • Image-conditioned DPMs have been applied to restoration by concatenating the corrupted observation at input level, but generalization to unseen domain shifts remains underexplored.The proposed training process equips an icDPM UNet with multiscale structure guidance.Source: §2 · p. 2, Figure 2 · p. 3

Generalization to unseen domains

  • Generalization approaches include more representative training data and explicit domain adaptation, which may require retraining for a new dataset; this method instead changes image conditioning without explicit adaptation between specified domains.The paper presents these as distinct directions that can be combined.Source: §2 · p. 3

3. Method

The method trains an image-conditioned diffusion model on paired sharp and blurry images, adding regression-derived multiscale structure features to its intermediate representations. Guidance and denoising losses are optimized end-to-end.

3.1. Overview

  • The training setup assumes paired sharp images and blurry observations, while the goal is to recover clean images from observations drawn from an unseen distribution.The DPM iteratively denoises a noisy target; the conditional model also receives the blurry input.Source: §3.1 · p. 3

3.2. Multiscale structure guidance

  • At scales k = 1, 2, 3, grayscale inputs are downsampled and noised before a regression-driven network extracts features constrained to predict the sharp target at each scale.This transformation suppresses color and domain-specific degradation cues while retaining coarse structures.Source: §3.2 · p. 4
  • The extracted guidance features are added as biases to corresponding diffusion-encoder feature maps, with convolutions matching the encoder’s feature counts.The learned guidance is drawn from latent features of a regression network predicting the sharp target’s luma at lower resolutions.Source: Figure 3 · p. 3, §3.2 · p. 4

3.3. Training loss

  • A multiscale regression loss trains the guidance features toward transformed sharp targets, using mean squared error at each scale and averaging across scales.A single convolutional layer projects the guidance features toward the clean-image target.Source: §3.3 · p. 4
  • The icDPM denoising loss predicts noise from the noisy image, blurry input, scheduler, and multiscale guidance; both losses jointly optimize the guidance and diffusion networks.At inference, sampling begins with Gaussian noise and iteratively recovers the clean image conditioned on the blurry input and guidance.Source: §3.3 · p. 4

4. Experiments

Experiments evaluate a GoPro-trained model on in-domain and unseen datasets, testing guidance effectiveness, generalization, and design choices. The reported results show stronger perceptual quality on unseen data while retaining competitive distortion metrics.

4.1. Setup and metrics

  • The evaluation trains on GoPro and tests generalization on Realblur-J, REDS, and HIDE, treated as unseen test sets.Realblur-J is described as having the largest domain gap with GoPro.Source: §4.1 · pp. 4–5

4.2. Implementation details

  • Training warms up with regression loss, then increases denoising-loss weight to 1 over 60k iterations; inference samples under varied parameters.The experiments use a fully convolutional UNet so inference can run at arbitrary image resolutions.Source: §4.2 · p. 5

4.3. Effectiveness of the guidance

  • For GoPro-trained models tested on GoPro and Realblur-J, guidance consistently improves in- and out-of-domain performance, while increasing unguided icDPM capacity improves in-domain results but worsens out-of-domain results.Guided models consistently outperform baseline DPMs under varying sampling parameters.Source: §4.3 · pp. 5–6, Table 2 · p. 6

4.4. Deblurring results

  • <0.08 dB from best PSNR and <0.001 from best SSIM, Ours-SA improves perceptual scores averaged across four test sets.The model is trained only on GoPro; the four datasets are GoPro, HIDE, Realblur-J, and REDS.Source: Table 3 · p. 6, §4.4 · p. 7
  • On unseen datasets, the method reduces artifacts and produces more perceptually realistic outputs than the compared GAN, diffusion, and regression baselines.The reported comparisons include GoPro, HIDE, Realblur-J, and REDS, with all models trained only on GoPro.Source: §4.4 · p. 7, Figure 6 · p. 7, Figure 7 · p. 7

4.5. Perceptual Study

  • In a Relblur-J pairwise user study, raters preferred the proposed method over existing solutions, with a significant preference gap between the method with guidance and without guidance (icDPM).The study averaged 750 ratings from 25 raters across 30 unique 512×512 image pairs.Source: §4.5 · p. 8

4.6. Additional modeling choices

  • Latent-space guidance outperforms input-level concatenation, and multiscale guidance and grayscale regression targets further improve results.The ablations compare matched diffusion UNets using the same sampling parameters.Source: §4.6 · p. 9
  • Addition is the best-performing tested guidance operation on GoPro PSNR; removing regression loss lowers PSNR from 31.19 to 30.56.The authors also report that addition is more memory-efficient than concatenation and prevents the guidance from being simply neglected.Source: §4.6 · p. 9

5. Discussion

The discussion presents learned multiscale structure guidance as an implicit bias for more robust icDPM deblurring. It also identifies training-data quality and the method’s icDPM scope as boundaries for the work.

5. Discussion

  • The proposed learned multiscale structure guidance acts as an implicit bias intended to enhance icDPM deblurring robustness.The authors state that its limitations require further investigation.Source: §5 · p. 9
  • Training-dataset quality and realism ultimately bound model capability; the GoPro setup does not adequately cover scenarios such as saturated, poorly lit regions.The authors suggest that larger, more diverse training sets could benefit the method.Source: §5 · p. 9
  • The work specifically studies robustness for icDPM conditional generation; applying guidance to regression models remains future work and may increase parameters.The authors also identify integration with other regression backbones as requiring further investigation.Source: §5 · p. 9

A.1. Effectiveness of the guidance on GoPro, HIDE and REDS

Supplementary perception-distortion plots report that guidance improves perceptual quality and distortion relative to icDPM across network capacities. The models are trained on GoPro and compared on in-domain and out-of-domain datasets.

A.1. Effectiveness of the guidance on GoPro, HIDE and REDS

  • Guidance yields consistently better perceptual quality and lower distortions than icDPM across the reported network capacities.The plots cover both in-domain GoPro and out-of-domain HIDE and REDS.Source: Figure 10 · p. 15
  • For small networks, icDPM-S with Guide-S is reported above icDPM-S in the perception-distortion plots.The models are compared on GoPro, HIDE, and REDS.Source: Figure 10 · p. 15
  • For large networks, icDPM-L with Guide-S or Guide-L is reported above icDPM-L in the plots.All models are trained only on GoPro.Source: Figure 10 · p. 15

A.2. Additional visual results

The supplementary section provides additional deblurring examples across in-domain and out-of-domain datasets. It also describes training-data limits and failure cases on night scenes.

A.2. Additional visual results

  • Additional qualitative deblurring results are provided for GoPro, Realblur-J, REDS, and HIDE.The text identifies GoPro as in-domain and the other three as out-of-domain.Source: §A.2 · p. 15

A.3. Failure cases

  • GoPro’s limited daytime, well-lit outdoor coverage and less realistic blur synthesis constrain generalization to scenes such as low-light images with saturated regions.The authors also note that GoPro’s ground-truth images are relatively low quality.Source: §A.3 · p. 16
  • All methods fail to remove blur from the shown night scenes, especially when night streaks are present.The authors suggest more realistic training datasets could further increase model generalization.Source: §A.3 · p. 16

B. Additional Ablation

The additional ablations find that input-level concatenation improves deblurring over guidance-only conditioning, while adversarial cross-domain alignment offers no extra benefit and may hurt performance.

B. Additional Ablation

  • The tested guidance-only conditioning removes input-level concatenation and uses regression-network intermediate representations as the diffusion condition.The authors considered this as a possible way to improve domain generalization, while noting a risk of losing input detail.Source: B. Additional Ablation · p. 16
  • Input concatenation performs better in-domain and out-of-domain than omitting it, so the final model retains concatenation alongside guidance features.The comparison is reported in Table 11.Source: B. Additional Ablation · p. 16, Table 11 · p. 16
  • Adversarial cross-domain alignment of guidance features provided no extra benefits and could hurt performance.The authors speculate that GAN training instability or a suboptimal formulation may explain the result.Source: B. Additional Ablation · p. 17, Figure 11 · p. 17

C.1. Architectures

The appendix details the diffusion and guidance architectures, inference sampling choices, and computational-cost accounting. It also notes that faster diffusion sampling could be incorporated in future work.

C.1. Architectures

  • The diffusion UNet replaces encoder residual blocks with guided residual blocks that incorporate multiscale structure features extracted from downsampled grayscale blurry images.The guidance network predicts clean counterparts at multiple resolutions and uses the last residual block’s output as the guidance feature.Source: Figure 12 · p. 18, §C.1 · p. 18

C.2. Inference

  • Inference settings were selected by grid-searching diffusion steps and the upper bound of noise variance, excluding combinations that yielded pure noise or blank images.Continuous noise-level sampling during training permits different inference noise schedulers and potentially different distortion–perception trade-offs.Source: §C.2 · p. 19, Table 12 · p. 19

C.3. Computational cost

  • FLOPs are reported for a 720 × 1280 × 3 input, with diffusion-network costs measured per single diffusion step.Sampling-speed optimization is out of scope, though the authors note that recent diffusion-sampling acceleration methods could be incorporated.Source: Table 13 · p. 19, §C.3 · p. 19

C.4. Benchmark results

The appendix describes a consistent computation of benchmark comparisons and provides deblurring examples across REDS, HIDE, GoPro, and Realblur-J, including Realblur-J failure cases.

C.4. Benchmark results

  • Benchmark results were computed consistently, using author-provided results when possible and official models or code with GoPro checkpoints for specified comparisons.For Realblur-J cross-domain comparisons, author-provided results were used for DvSR, UFormer, and Restormer; REDS results came from official models with GoPro checkpoints.Source: §C.4 · p. 19
  • Figures 13–16 show REDS [56] deblurring examples for MPRNet [96], HINet [6], DeblurGAN-v2 [38], Restormer [94], UFormer [89], icDPM without guidance, and the guided method.These captions identify the compared methods but do not state which outputs perform better.Source: Figure 13 · p. 20, Figure 14 · p. 21, Figure 15 · p. 22
  • Across Figures 17–22, HIDE [76] (17–20) and GoPro [57] (21–22) deblurring examples feature MPRNet [96], MIMO UNet+ [7], SAPHNet [84], Restormer [94], UFormer [89], DvSR [90], and Ours.Source: Figure 17 · p. 24, Figure 18 · p. 25, Figure 19 · p. 26
  • Figures 25–28 show Realblur-J [67] deblurring examples comparing the proposed method with UNet [70], MPRNet [96], DeblurGAN-v2 [38], Restormer [94], UFormer [89], and DvSR [90].The captions identify the compared methods but do not report an outcome.Source: Figure 25 · p. 32, Figure 26 · p. 33, Figure 27 · p. 34
  • Figures 29–32 present Realblur-J [67] failure cases in low-light scenes or conditions, night scenes, and scenes with strong light streaks.The captions do not specify the cause or frequency of these failures.Source: Figure 29 · p. 36, Figure 30 · p. 37, Figure 31 · p. 38
Loading 2212.01789v3…