DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior

Motivation

• Blind image restoration seeks realistic reconstruction of general images under general degradations, extending classic restoration beyond settings with simple, known degradations.The paper identifies blind super-resolution, blind denoising, and blind face restoration as representative tasks, with applications such as old photo and film restoration.Source: §1 · pp. 1–2

Related work

• BSR and BID methods remove real-world degradations but often fail to generate realistic details, while BFR methods rely on a restricted face-image space and fixed input size.The paper cites BSRGAN [73] and Real-ESRGAN for BSR, SCUNet [74] for BID, and CodeFormer [77] and GFPGAN [55] for BFR.Source: §1 · p. 2
• Diffusion-prior zero-shot restoration methods such as DDRM [23], DDNM [57], and GDP [14] handle defined degradations but do not generalize well to unknown ones.The paper distinguishes realistic reconstruction on general images from restoration under general degradations.Source: §1 · p. 2

Contributions

• IRControlNet uses a pretrained text-to-image latent diffusion prior and reliable, diversified condition images to generate realistic details across different restoration modules.The conditions are produced by an MSE-based restoration module trained with simple, wide-range degradation, avoiding distracting noise and artifacts in the original low-quality images.Source: §1 · p. 2
• Training-free region-adaptive restoration guidance balances fidelity and quality by guiding low-frequency regions toward a high-fidelity image while preserving generative freedom in high-frequency regions.Users tune a guidance scale to move smoothly between fidelity and quality preferences.Source: §1 · p. 2
• DiffBIR is the first unified two-stage framework to achieve state-of-the-art performance across BSR, BFR, and BID, separating degradation removal from information regeneration.Different restoration modules handle degradation removal for each task, while one generation module regenerates lost content.Source: §1 · p. 2

2.1. Blind Image Restoration

• BSR methods model practical degradations (BSRGAN [73], RealESRGAN), use SwinIR-GAN [29] or FeMaSR [5], or apply diffusion (StableSR [52], PASD [66]); they require retraining for other restoration tasks.StableSR uses a time-aware encoder, while PASD injects pixel-level condition information through PACA.Source: §2.1 · p. 3
• Face restoration combines GAN priors [4] [18] [55] [65], HQ codebooks [16] [59] [77], and diffusion priors [61] [63] [67], but these methods are limited to faces.The cited HQ-codebook approaches exploit VQ dictionary learning [13] [50].Source: §2.1 · p. 3
• Blind denoising methods address unknown noise with DnCNN [71], GCBD, CBDNet [17], and VDNet [68], but typically produce smooth results.Their approaches include multi-level Gaussian denoising, GAN-based noise modeling, realistic noise synthesis with real noisy-clean pairs, and joint noise estimation and denoising.Source: §2.1 · p. 3

2.2. Zero-shot Image Restoration.

• Zero-shot restoration evolved from searching pretrained GAN latent spaces [2] [9] [36] [39] to DDPM-based methods, but remains unsatisfactory on real-world low-quality images.ZIR leverages pretrained prior networks in an unsupervised manner.Source: §2.2 · p. 3
• DDRM [23] handles linear restoration using an SVD-based approach, DDNM [57] uses range-null-space decomposition for sampling, and GDP [14] estimates degradation during inference for guidance.GDP is inspired by classifier guidance [11].Source: §2.2 · p. 3

3.1. Motivation and Framework

• DiffBIR separates degradation removal from content regeneration because degraded LQ conditions entangle content and degradation, destabilizing generation and causing artifacts.The independently optimized stages allow flexible restoration modules while conditioning generation on image content rather than degradation.Source: §3.1 · pp. 3–4

3.2. Restoration Module

• Task-specific restoration modules remove degradations without generating content; a separately trained RM creates diverse condition images for generation training and is discarded at inference.The task-specific modules are off-the-shelf BIR models trained with MSE, while the preprocessing RM uses a large degradation range and cannot handle complex real-world degradations.Source: §3.2 · p. 4

3.3. Generation Module

• IRControlNet encodes the condition with Stable Diffusion’s fixed VAE, concatenates its latent with noisy zt in a copied UNet condition network, and modulates frozen denoiser features.Only the condition network and feature modulation are trained; modulation is applied to middle-block and skipped features, with zero convolutions for stability.Source: §3.3 · pp. 4–5
• IRControlNet’s VAE condition encoding preserves input color and outperforms ControlNet by 3dB on average in PSNR.The authors attribute this to encoding the condition in the same latent space where image generation occurs.Source: §3.3 · p. 6
• IRControlNet converges fastest; removing zt favors fidelity but yields lower IQA and smoother textures, while random initialization performs worst across metrics.Changing modulation from skipped to decoder features gives comparable convergence and quantitative results, but requires more parameters and computation.Source: §3.3 · p. 6, Figure 5 · p. 6, Table 1 · p. 6

3.4. Restoration Guidance

• Training-free restoration guidance updates the clean latent at every sampling step, weighting flat regions more strongly toward the restored image while allowing more generated content in high-frequency regions.A tunable guidance scale controls the fidelity trade-off: larger values push the decoded result closer to the guidance image.Source: §3.4 · pp. 6–7

4.1. Datasets, Implementation, Metrics

• DiffBIR is trained on roughly 15M filtered LAION2B-EN images, then evaluated with task-specific restoration models and a shared IRControlNet across BSR, BFR, and BID datasets.The evaluation uses PSNR, SSIM, and LPIPS for synthesized data, no-reference IQA metrics, and FID for BFR; inference uses 50 spaced-DDPM sampling steps.Source: §4.1 · p. 7

4.2. Comparisons with State-of-the-Art Methods

• For BSR, DiffBIR (s = 0) leads all reported real-world metrics and synthetic IQA metrics, while s = 1 gives the best synthetic PSNR and SSIM.At s = 1, MANIQA and CLIP-IQA still rank in the top three; the paper recommends tuning the guidance scale to balance quality and fidelity. On real-world data, DiffBIR produces sharper, more realistic details than GAN- and diffusion-based methods.Source: Table 2 · p. 8, Table 3 · p. 8, Figure 7 · p. 8
• For BFR, DiffBIR achieves the highest reported FID on LFW and Wider, leads CLIP-IQA and MANIQA, and has MUSIQ scores close to the highest.Despite not finetuning IRControlNet on a face dataset such as FFHQ, DiffBIR restores challenging hand and side-face regions that competing methods distort or fail to recover.Source: Table 4 · p. 8, Figure 8 · p. 8, §4.2 · p. 9
• For real-world BID, DiffBIR significantly outperforms baselines across all metrics and removes noise while generating realistic textures.SwinIR and SCUNet-GAN remove noise but produce smoothed results without vivid texture details.Source: Table 5 · p. 9, §4.2 · p. 9, Figure 9 · p. 9

4.3. Ablation Studies

• Removing the Restoration Module causes a noticeable drop across IQA and reference-based metrics and produces less realistic results than the two-stage pipeline.The one-stage model treats degradation as semantic information, leading to facial distortion, colorful backgrounds, and unusual eye shapes.Source: §4.3 · p. 9, Figure 10 · p. 9, Table 6 · p. 10
• Compared with ControlNet, IRControlNet avoids the color shifts shown in visual comparisons and achieves higher PSNR on BSR.The paper attributes ControlNet’s color shifts to the lack of explicit color-consistency regularization during training.Source: §4.3 · pp. 9–10, Figure 10 · p. 9, Table 7 · p. 10
• Using DiffBIR’s wide-range degradation model yields better restoration quality than the smaller-range RealESRGAN degradation model on RealSRSet.The authors report that the wider range improves utilization of the generation module’s capabilities.Source: Table 8 · p. 10, §4.3 · p. 10

• Experiments validate DiffBIR’s superiority over existing state-of-the-art methods for BSR, BFR, and BID.Source: §5 · p. 10
• DiffBIR is a unified blind image restoration framework that leverages the prior knowledge of pre-trained Stable Diffusion to achieve realistic restoration results.Source: §5 · p. 10
• Restoring one low-quality image requires 50 sampling steps, making DiffBIR computationally expensive; its two-stage pipeline may also be explored for other blind image restoration tasks.Source: §5 · p. 10

6.1. Comparison with More Variants

• Variants 5 and 6 improve PSNR and SSIM but lower MANIQA than IRControlNet, trading generation quality for fidelity through stronger feature control.Variant 5 controls middle, decoder, and skipped features; Variant 6 uses an SFT layer [53] to modulate skipped features, while IRControlNet uses restoration guidance and add-on control to preserve generation capability.Source: §6.1 · p. 13, Table 9 · p. 13
• IRControlNet generates more vivid textures than Variant 2, which tends to produce over-smoothed results.The appendix presents visual comparisons of the two methods.Source: §6.1 · p. 14, Figure 12 · p. 14

6.2. More Details of Training RM

• For generation-module training, the authors modify SwinIR as a restoration module and train it on synthetic low- and high-quality image pairs using a first-order degradation model.They expand the blur and downsampling ranges because these operations contribute most to information loss; σ, r, δ, and q are sampled from {0.1:12}, {1:12}, {0:15}, and {30:100}, respectively.Source: §6.2 · p. 14

6.3. More Details about Restoration Guidance

• Restoration guidance adjusts the predicted clean latent at each denoising step toward the first-stage IRM using a region-adaptive image-space MSE, without additional training.Its weight map uses gradient intensity from the IRM; the method estimates this at patch level by summing pixel gradient magnitudes and mapping the result with tanh.Source: §6.3 · p. 14

6.4. More Quantitative and Qualitative Comparisons for BSR on Synthetic Datasets

• At guidance s=0, DiffBIR significantly outperforms baselines on all IQA metrics; at s=1, it surpasses them on MANIQA and CLIP-IQA, and is comparable to GAN-based methods on PSNR.At s=1, it also performs better than diffusion-based methods on PSNR, indicating a balance between quality and fidelity.Source: §6.4 · p. 15, Table 10 · p. 15
• On DIV2K-Val, only DiffBIR restores correct semantic details, while GAN-based methods oversmooth and diffusion-based baselines fail under severe degradation.Examples include recovering eyes behind a helmet, fireworks lines, and penguin wings.Source: §6.4 · p. 15, Figure 13 · p. 16

6.5. Quantitative Comparisons for Efficiency

• On 128 × 128 inputs at 4× super-resolution, DiffBIR is the most efficient diffusion baseline, running about 1.8x faster than StableSR and 1.6x faster than PASD.GAN-based methods are more efficient but perform significantly worse than diffusion-based methods.Source: §6.5 · p. 15, Table 11 · p. 15

6.6. More Real-world Visual Comparisons

• The paper provides additional visual comparisons on real-world datasets for the BSR, BID, and BFR tasks, shown in Figures 14, 15, and 16, respectively.Source: §6.6 · p. 15, Figure 14 · p. 17, Figure 15 · p. 17