Source-linked AI summary
DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior
Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Bo Dai, Fanghua Yu, Wanli Ouyang, Yu Qiao, Chao Dong
TL;DR
Blind image restoration must reconstruct realistic content from low-quality images with unknown and varied degradations. DiffBIR separates degradation removal from information regeneration, using IRControlNet and controllable restoration guidance; experiments report superiority across blind super-resolution, face restoration, and denoising tasks, while the authors note a 50-step computational cost.
Problem
Blind restoration must handle general degradations, while directly using low-quality images as diffusion conditions can destabilize generation and induce artifacts.
Method
DiffBIR uses task-specific restoration modules for degradation removal, IRControlNet for diffusion-based information regeneration, and training-free region-adaptive guidance for quality–fidelity control.
Results
DiffBIR achieves state-of-the-art performance across blind super-resolution, blind face restoration, and blind image denoising on synthetic and real-world datasets.
Takeaways & Limitations
The two-stage pipeline provides a unified route to realistic blind restoration while allowing users to tune the balance between realness and fidelity.
Takeaways & Limitations
Restoring one low-quality image requires 50 sampling steps, making the method computationally expensive.
Abstract
from arXiv · showhide
We present DiffBIR, a general restoration pipeline that could handle different blind image restoration tasks in a unified framework. DiffBIR decouples blind image restoration problem into two stages: 1) degradation removal: removing image-independent content; 2) information regeneration: generating the lost image content. Each stage is developed independently but they work seamlessly in a cascaded manner. In the first stage, we use restoration modules to remove degradations and obtain high-fidelity restored results. For the second stage, we propose IRControlNet that leverages the generative ability of latent diffusion models to generate realistic details. Specifically, IRControlNet is trained based on specially produced condition images without distracting noisy content for stable generation performance. Moreover, we design a region-adaptive restoration guidance that can modify the denoising process during inference without model re-training, allowing users to balance realness and fidelity through a tunable guidance scale. Extensive experiments have demonstrated DiffBIR's superiority over state-of-the-art approaches for blind image super-resolution, blind face restoration and blind image denoising tasks on both synthetic and real-world datasets. The code is available at https://github.com/XPixelGroup/DiffBIR.
1. Introduction
Blind image restoration targets realistic reconstruction from general degradations, extending classic restoration beyond constrained settings. DiffBIR addresses this challenge with a unified two-stage framework that removes degradation before regenerating lost information.
- Classic restoration methods assume simple, known degradations, which limits their generalization to real-world degraded images.
- Blind image restoration seeks realistic reconstruction for general images with general degradations, including applications such as old photo and film restoration.
- Existing blind super-resolution methods robustly remove degradations but often fail to generate realistic details because of limited generative ability.
- DiffBIR decouples blind restoration into degradation removal and information regeneration within a unified framework.Task-specific restoration modules remove degradations, while a generation module regenerates lost image content.
- IRControlNet uses a text-to-image latent diffusion prior with reliable, diversified condition images to stabilize realistic content generation.The conditions are produced by an additional restoration module rather than directly using noisy low-quality images.
- DiffBIR introduces training-free region-adaptive restoration guidance to let users trade off quality and fidelity during sampling.
2. Related Work
Related work spans blind super-resolution and zero-shot restoration, including GAN-, transformer-, feature-matching-, and diffusion-prior approaches. DiffBIR is motivated by the gap between restoration methods requiring degradation assumptions and the complex, entangled degradations in blind restoration.
- Blind Image Super-Resolution: Blind image super-resolution methods model increasingly complex degradations using strategies such as degradation shuffling, high-order degradation modeling, transformer backbones, and feature matching.
- Zero-shot Image Restoration: Zero-shot restoration leverages pretrained prior networks without supervision, progressing from GAN latent-code search to diffusion-based approaches.
- Motivation: Directly conditioning diffusion generation on low-quality images is unstable because degradation and content information are entangled, producing artifacts.
3. Method
DiffBIR separates blind image restoration into degradation removal and information regeneration, using task-specific restoration modules followed by an IRControlNet diffusion-based generation module. Region-adaptive guidance then balances fidelity and generated detail during inference.
- Motivation and Framework: DiffBIR treats blind restoration as conditional generation but separates degradation removal from image-content regeneration to avoid unreliable degraded conditions.The two stages are independently optimized and cascaded, with generation conditioned on restored image content rather than distracting degradation.
- Restoration Module: Task-specific restoration modules remove image-independent degradations without generating new content, producing condition images for the generation module.Separate modules address differing degradation processes and datasets across blind denoising and blind face restoration.
- Restoration Module: The training restoration module uses synthesized low-quality/high-quality pairs across a broad degradation range, then is discarded at inference because it cannot handle complex real-world degradations.Its role is condition preprocessing for training the generation module, not final real-world restoration.
- Generation Module: IRControlNet encodes the restored condition with the fixed pretrained VAE encoder and combines condition encoding, a condition network, and feature modulation.Projecting the condition into the latent space improves control of the latent diffusion prior; a ControlNet-style variant was 3dB lower in PSNR on average.
- Generation Module: Including the noisy latent at each timestep helps IRControlNet converge and generate detailed results, while alternative initialization and architecture choices can impair convergence or performance.Variant 2 favored fidelity metrics but produced smoother results with fewer texture details; Variant 3 had the worst performance across metrics.
- Restoration Guidance: Region-adaptive restoration guidance applies a training-free, user-controlled denoising adjustment that preserves fidelity in flat regions while allowing more generated content in high-frequency regions.A gradient-density weight map assigns larger loss weights to weak-gradient regions, and larger guidance scales push outputs closer to the restoration-module result.
4. Experiments
Experiments evaluate DiffBIR across blind super-resolution, face restoration, and denoising tasks on synthetic and real-world data, including comparisons, ablations, and guidance analysis. Results support the unified two-stage design, IRControlNet, and tunable fidelity–realism control.
- Experimental setup: DiffBIR is evaluated on BSR, BFR, and BID datasets, using both synthetic and real-world benchmarks and task-specific restoration models during inference.The evaluation includes DIV2K-Val, DRealSR, RealSR, RealSRSet, Real47, LFW-Test, WIDER-Test, and mixed real-world denoising data.
- BSR comparisons: DiffBIR with s = 0 significantly outperforms all baseline methods on MUSIQ, MANIQA, and CLIP-IQA for synthetic BSR.At s = 1, it obtains the best PSNR and SSIM while MANIQA and CLIP-IQA remain top-3; s = 0.5 is suggested as a balance between quality and fidelity.
- BSR comparisons: On RealSRSet and Real47, DiffBIR with s = 0 obtains the best scores across all reported metrics for real-world BSR.Visual comparisons describe sharper results than GAN-based methods and more realistic details than diffusion-based methods.
- Ablation studies: Ablation visualizations show that omitting the restoration module harms fidelity maintenance, while IRControlNet addresses ControlNet’s color-shift problem.These comparisons connect degradation removal with fidelity and condition encoding with color consistency.
- Ablation studies: Removing the restoration module causes a noticeable performance drop across IQA and reference-based metrics on synthetic and real-world datasets.The ablation compares the proposed two-stage pipeline with direct diffusion-model finetuning on synthesized training pairs.
- Ablation studies: IRControlNet achieves higher PSNR scores than ControlNet, while the proposed degradation model improves restored-result quality through better use of generative capabilities.The degradation-model comparison is evaluated on RealSRSet.
5. Conclusion and Limitations
The paper concludes that DiffBIR provides realistic blind image restoration by combining a unified two-stage framework with Stable Diffusion prior knowledge. It reports strong results across BSR, BFR, and BID, while noting substantial sampling cost and possible extension to other BIR tasks.
- DiffBIR uses pre-trained Stable Diffusion prior knowledge in a unified framework for realistic blind image restoration.
- Extensive experiments validate DiffBIR’s superiority over existing state-of-the-art methods for BSR, BFR, and BID.
- Restoring one low-quality image requires 50 sampling steps, making DiffBIR computationally expensive.
- The two-stage restoration pipeline may be feasible for other blind image restoration tasks, motivating further exploration.
6. Appendix
The appendix analyzes IRControlNet variants, restoration guidance, efficiency, and visual comparisons across blind restoration tasks. Results show a fidelity–quality trade-off, region-adaptive control, and competitive diffusion efficiency.
- 6.1. Comparison with More Variants: Variant 5 and Variant 6 improve PSNR and SSIM but obtain worse MANIQA than IRControlNet.Both variants trade generation quality for fidelity through stronger feature control.
- 6.1. Comparison with More Variants: IRControlNet generates more vivid textures than Variant 2, which tends to produce over-smoothed results.
- 6.3. More Details about Restoration Guidance: Restoration guidance modifies the clean latent at each denoising step and requires no additional training.The method updates the clean latent before sampling the previous noisy latent.
- 6.4. More Quantitative and Qualitative Comparisons for BSR on Synthetic Datasets: On DRealSR and RealSR, DiffBIR outperforms baselines across IQA metrics at guidance scale s=0 and balances quality and fidelity at s=1.At s=1, it surpasses baselines in MANIQA and CLIP-IQA and is comparable to GAN-based methods in PSNR.
- 6.5. Quantitative Comparisons for Efficiency: DiffBIR is about 1.8x faster than StableSR and about 1.6x faster than PASD among diffusion-based baselines.GAN-based methods remain more efficient but perform significantly worse than diffusion-based methods.