PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Integration Diffusion Model for Realistic Image Deblurring

Zheng Chen, Yulun Zhang, Ding Liu, Bin Xia, Jinjin Gu, Linghe Kong, Xin Yuan

arXiv:2305.12966v4cs.CV

TL;DR

Image deblurring requires recovering sharp images from complex blur, while diffusion models face costly inference and regression methods can lose details. HI-Diff generates priors in a highly compact latent space and hierarchically integrates them into a regression-based Transformer. It outperforms state-of-the-art methods on synthetic and real-world blur datasets.

  • Problem

    Image deblurring must recover sharp images from complex, non-uniform blur, while diffusion models require costly inference and regression methods can have limited detail reconstruction.

  • Method

    HI-Diff performs diffusion in a highly compact latent space to generate priors, then hierarchically integrates them into a regression-based Transformer deblurring process.

  • Results

    HI-Diff outperforms state-of-the-art deblurring methods on synthetic and real-world blur datasets.

  • Takeaways & Limitations

    The approach combines diffusion-generated detail priors with regression-based restoration while retaining computational efficiency and improving generalization in complex blurry scenarios.

Abstract

from arXiv · show

Diffusion models (DMs) have recently been introduced in image deblurring and exhibited promising performance, particularly in terms of details reconstruction. However, the diffusion model requires a large number of inference iterations to recover the clean image from pure Gaussian noise, which consumes massive computational resources. Moreover, the distribution synthesized by the diffusion model is often misaligned with the target results, leading to restrictions in distortion-based metrics. To address the above issues, we propose the Hierarchical Integration Diffusion Model (HI-Diff), for realistic image deblurring. Specifically, we perform the DM in a highly compacted latent space to generate the prior feature for the deblurring process. The deblurring process is implemented by a regression-based method to obtain better distortion accuracy. Meanwhile, the highly compact latent space ensures the efficiency of the DM. Furthermore, we design the hierarchical integration module to fuse the prior into the regression-based model from multiple scales, enabling better generalization in complex blurry scenarios. Comprehensive experiments on synthetic and real-world blur datasets demonstrate that our HI-Diff outperforms state-of-the-art methods. Code and trained models are available at https://github.com/zhengchen1999/HI-Diff.

1 Introduction

Image deblurring must recover sharp images despite complex, non-uniform blur that limits traditional priors and regression-based detail reconstruction. HI-Diff combines compact-latent diffusion priors with regression and hierarchical integration to improve efficiency, distortion accuracy, and generalization.

  • Complex, non-uniform real-world blur is difficult to model with specific handcrafted priors, causing poor generalization.
  • Regression-based methods achieve strong distortion-based metrics but tend to recover fewer high-frequency details.
  • Diffusion models reconstruct high-fidelity images through iterative denoising but require many inference steps and remain computationally expensive.
  • HI-Diff performs diffusion in a highly compact latent space, integrates its priors into a regression-based model, and uses hierarchical fusion for complex blur.
  • Extensive experiments on synthetic and real-world blur datasets demonstrate superior HI-Diff performance over state-of-the-art deblurring methods.

2 Related Work

Prior image-deblurring methods include handcrafted optimization, CNN, Transformer, and diffusion approaches. Diffusion models improve restoration quality but retain substantial inference complexity, while Transformer-based regression methods remain limited in detail recovery.

  • Traditional Methods: Traditional methods formulate deblurring as optimization with handcrafted natural-image or blur-kernel priors, limiting performance in complex situations.
  • Deep CNN-based Methods: CNN-based methods improve deblurring through multi-scale, recurrent, patch-based, and hierarchical architectures.
  • Transformer-based Methods: Transformer methods model long-range dependencies and improve deblurring performance through channel or orientation-specific attention mechanisms.
  • Transformer-based Methods: Transformer-based regression methods remain limited in recovering high-frequency image details because their reconstruction behavior is conservative.
  • Diffusion Models: Diffusion models generate samples through stochastic iterative denoising and have been applied to restoration tasks including deblurring.
  • Diffusion Models: Diffusion-based restoration methods still face high inference complexity, especially for high-resolution image deblurring.

3 Method

HI-Diff combines latent diffusion with a regression-based Transformer deblurring model. It generates compact prior features, integrates them hierarchically through cross-attention, and jointly trains diffusion and restoration components to guide deblurring.

  • Framework: HI-Diff integrates a diffusion-generated prior feature hierarchically into a Transformer-based regression model for image deblurring.The prior feature is fused into the Transformer at multiple scales through hierarchical integration.
  • Stage One: Latent Compression: Stage one compresses the blurry image and ground truth into a highly compact latent prior representation used to guide Transformer restoration.The latent token count N is much smaller than the image spatial size H×W, reducing the subsequent diffusion model's computational burden.
  • Hierarchical Integration Module: The hierarchical integration module applies cross-attention between Transformer intermediate features and prior features, then adds the fused output to the intermediate representation.Queries come from reshaped intermediate features, while keys and values are projected from the prior feature.
  • Hierarchical Integration Module: Multiple-scale prior features are created by downsampling the latent prior so they can adapt to Transformer features at different resolutions.The multi-scale design addresses the difficulty of adapting a single-scale prior to complex, non-uniform blur.
  • Stage Two: Latent Diffusion Model: Stage two trains a conditional latent diffusion model to generate the prior feature from Gaussian noise using the blurry image as a condition.The model encodes the blurry image into a condition latent, predicts noise during reverse denoising, and produces the predicted prior feature used by the Transformer.
  • Training Strategy: HI-Diff jointly trains diffusion and Transformer components because predicted-prior deviations can otherwise create a mismatch that restricts deblurring performance.During joint training, the predicted prior feature from the complete reverse process guides Transformer through the hierarchical integration module.

4 Experiments

Experiments evaluate HI-Diff on synthetic and real-world datasets, ablate its diffusion prior, hierarchical integration, joint training, and iteration count, and compare accuracy, visual quality, and complexity. HI-Diff improves deblurring results while using a compact diffusion prior and achieves strong cross-dataset performance and efficiency.

  • Data and Evaluation: Experiments use GoPro, HIDE, RealBlur, and RWBI, with ablations conducted on GoPro and comparisons spanning synthetic and real-world blur.The evaluation includes specified training and testing datasets, while the ablation setup fixes GoPro experiments at 100K iterations per stage.
  • Ablation Study: The diffusion prior improves PSNR by 0.28 dB over the Restormer baseline while adding only 4.86M Params and 8.22G FLOPs.Visual comparisons also show sharper textures and more complete structures than the baseline without the prior.
  • Ablation Study: Multi-scale hierarchical integration improves PSNR by 0.24 dB over single-scale guidance and restores better-deblurred images in visual comparisons.The ablation attributes the single-scale model’s lower performance to poorer adaptation to complex blurry situations.
  • Ablation Study: Joint training outperforms split training by 1.51 dB on PSNR, demonstrating the importance of jointly training the diffusion model and Transformer.Split training optimizes the diffusion model alone before combining it with the Transformer for evaluation.
  • Ablation Study: Diffusion performance essentially converges at 8 iterations, whereas one iteration cannot generate a reasonable prior feature.The compact latent space contains only N=16 tokens, supporting effective prior generation with few iterations.
  • Quantitative Results: HI-Diff outperforms all compared methods across four benchmark datasets, including gains of 0.25 dB over Stripformer on GoPro and 0.41 dB and 0.24 dB over Restormer on GoPro and HIDE.All models in this comparison are trained only on GoPro, and the evaluated datasets include GoPro, HIDE, RealBlur-R, and RealBlur-J.
  • Quantitative Results: On real-world datasets, HI-Diff gains 0.33 dB over Stripformer on RealBlur-J and 0.09 dB and 0.19 dB over Restormer on RealBlur-R and RealBlur-J.These results are reported for a GoPro-trained model evaluated on RealBlur datasets.
  • Quantitative Results: When trained and tested on RealBlur, HI-Diff gains 1.17 dB on RealBlur-R and 1.22 dB on RealBlur-J over Stripformer.Table 3 reports PSNR/SSIM comparisons using corresponding-dataset training and testing.

5 Conclusion

HI-Diff combines diffusion-generated prior features with a regression-based deblurring method and hierarchical multi-scale fusion. Experiments on synthetic and real-world blur datasets show that it outperforms state-of-the-art methods.

  • HI-Diff generates prior features with a diffusion model during regression-based image deblurring.The regression-based method preserves the general distribution, while diffusion-generated priors enhance deblurred-image details.
  • Diffusion operates in a highly compact latent space to improve computational efficiency.
  • The hierarchical integration module fuses prior and Transformer image features across scales for better generalization in complex blurry scenarios.
  • Extensive experiments on synthetic and real-world blur datasets demonstrate that HI-Diff outperforms state-of-the-art methods.
Loading 2305.12966v4…