Source-linked AI summary
DR2: Diffusion-based Robust Degradation Remover for Blind Face Restoration
Zhixin Wang, Xiaoyun Zhang, Ziying Zhang, Huangjie Zheng, Mingyuan Zhou, Ya Zhang, Yanfeng Wang
TL;DR
Blind face restoration models trained with predefined synthetic degradations can produce artifacts on severe or unseen degradations, while simulating every real-world degradation is difficult and expensive. DR2E uses a pretrained diffusion model to produce a coarse degradation-invariant prediction, then an enhancement module restores details. The paper reports that DR2E outperforms state-of-the-art methods on heavily degraded synthetic and real-world datasets.
Problem
Training on predefined synthetic degradations leaves blind face restoration methods without a robust mechanism for severe or unseen degradations, which can produce artifacts.
Source: §1 · pp. 1–2Method
DR2 uses diffused low-quality information to guide a pretrained DDPM toward a coarse, degradation-invariant prediction, which an enhancement module maps to a detailed output.
Source: §3.3 · p. 4, §5 · p. 8Results
DR2E outperforms state-of-the-art methods on heavily degraded synthetic and real-world datasets.
Source: §1 · p. 2Takeaways & Limitations
The two-stage framework combines degradation removal without synthetically degraded training data with enhancement for high-quality detail restoration.
Source: §5 · p. 8DR2 is less necessary and slower for slight degradation, may remove input details when degradation is very slight or absent, and requires manual choices of N and τ.
Source: §4.5 · p. 8
Abstract
from arXiv · showhide
Blind face restoration usually synthesizes degraded low-quality data with a pre-defined degradation model for training, while more complex cases could happen in the real world. This gap between the assumed and actual degradation hurts the restoration performance where artifacts are often observed in the output. However, it is expensive and infeasible to include every type of degradation to cover real-world cases in the training data. To tackle this robustness issue, we propose Diffusion-based Robust Degradation Remover (DR2) to first transform the degraded image to a coarse but degradation-invariant prediction, then employ an enhancement module to restore the coarse prediction to a high-quality image. By leveraging a well-performing denoising diffusion probabilistic model, our DR2 diffuses input images to a noisy status where various types of degradation give way to Gaussian noise, and then captures semantic information through iterative denoising steps. As a result, DR2 is robust against common degradation (e.g. blur, resize, noise and compression) and compatible with different designs of enhancement modules. Experiments in various settings show that our framework outperforms state-of-the-art methods on heavily degraded synthetic and real-world datasets.
1. Introduction
Blind face restoration methods trained on predefined synthetic degradations can produce artifacts on severe or unseen real-world degradations. DR2E uses a diffusion-based degradation remover followed by an enhancement module, and the paper reports that it outperforms state-of-the-art methods on heavily degraded synthetic and real-world datasets.
1. Introduction
- Face-restoration methods use geometric, reference, generative, and dictionary priors to recover facial shape, details, and textures.Examples include landmarks, parsing maps, and heatmaps [5], high-quality references [9] [26], and generative priors [39] [42].Source: §1 · p. 1
- Mismatch between synthetic training degradations and real-world inputs can make restoration networks produce artifacts on severe or unseen degradation.Artifacts occur when high-frequency information is missing or corrupted and is respectively undergenerated or mistakenly used during restoration.Source: §1 · pp. 1–2
- Adding Gaussian noise makes different degradation types less distinguishable while DDPM can still capture semantics; Figure 2 compares error distributions before and after diffusion.This pretrained DDPM property is used for degradation removal without training on synthetically degraded faces.Source: §1 · p. 2, Figure 2 · p. 2
- DR2E first transforms degraded inputs into coarse, visually clean, degradation-invariant predictions, then an enhancement module restores high-quality details.The enhancement module is compatible with different restoration-method designs.Source: §1 · p. 2
- DR2E outperforms state-of-the-art methods on heavily degraded synthetic and real-world datasets.The framework combines DR2 with an enhancement module.Source: §1 · p. 2
2. Related Work
Related work covers facial priors for blind restoration and diffusion models for image generation and super-resolution. The DR2E framework pairs a degradation remover with enhancement, and adopts a hybrid architecture to speed diffusion sampling.
Blind Face Restoration
Diffusion Models
- Diffusion super-resolution methods include concatenation-conditioned SR3 [35] and Latent Diffusion [33], while ILVR [6] controls pretrained DDPM generation for image translation.SR3 fixes degradation to simple downsampling, limiting its applicability to other degradation settings.Source: §2 · p. 2
- DR2E combines DR2 degradation removal with enhancement, and adopts a hybrid architecture like [47] to speed diffusion sampling.Its inference process uses diffused low-quality inputs as guidance before enhancement produces the final output.Source: Figure 3 · p. 3, §2 · p. 3
3. Methodology
DR2E separates blind face restoration into diffusion-based degradation removal that produces a coarse, degradation-invariant prediction and enhancement that restores high-quality details. DR2 uses iterative denoising with input conditioning, while the enhancement module can use different restoration backbones.
3.1. Preliminary
- DDPMs progressively add noise using a variance schedule and generate images by reversing that process with a learned denoising mean.The noisy state approaches a standard Gaussian when the diffusion timestep is sufficiently large.Source: §3.1 · p. 3
3.2. Framework Overview
- DR2E models restoration without a known degradation process by factoring it into degradation removal followed by enhancement, relying on an assumption that sufficiently noisy clean and degraded images become close in distribution.With suitable conditioning, DR2 predicts a coarse result that retains the clean image's semantics.Source: §3.2 · pp. 3–4
3.3. Diffusion-based Robust Degradation Remover
- DR2 starts diffusion from the degraded input, iteratively replaces low-frequency content with a filtered input signal, and truncates denoising to estimate a coarse output.The filter downsamples and upsamples the input; larger N and τ remove more degradation but reduce fidelity, while smaller values preserve fidelity but may retain degradation.Source: §3.3 · p. 4
3.4. Enhancement Module
- The enhancement module can use different restoration backbones, and its training data can be built by applying DR2 to clean images and then adding Gaussian blur.This data construction is recommended but not compulsory; the paper reports fewer artifacts and higher restoration quality with different backbones on heavily degraded CelebA-Test inputs.Source: §3.4 · pp. 4–5, Figure 4 · p. 5
4. Experiments
DR2E is evaluated on synthetic and real-world face-restoration datasets, with comparisons showing strong results on heavily degraded inputs. Experiments also examine diffusion-based alternatives, the effects of N and τ, and practical limitations.
4.1. Datasets and Implementation
- Testing covers CelebA-Test at 16×, 8×, and 4× across three degradation splits, plus four real-world datasets; parameters N and τ are grid-searched per dataset.WIDER-Critical and WIDER-Normal each contain 400 cases; CelebChild has 180 images and LFW-Test uses 1,000 images.Source: §4.1 · p. 5
4.2. Comparisons with State-of-the-art Methods
- On severe synthetic tasks, DR2 + VQFR and DR2 + SPAR achieve the best and second-best LPIPS and FID scores at 16× and 8×; on WIDER-Critical they rank first and second by FID, while DR2 + VQFR leads on mildly degraded real-world datasets.On 4× synthetic data, the methods achieve superior FID scores; at very low resolution, prior methods fail to complement adequate information while DR2 outputs are more visually pleasant.Source: Table 1 · p. 6, Table 2 · p. 6, §4.2 · p. 6
4.3. Comparisons with Diffusion-based Methods
- Concatenation-based diffusion is sensitive to degradation, while ILVR faces a fidelity–visual-quality trade-off; DR2 separates fidelity control from detail restoration through its enhancement module.Minor noise in the concatenated method’s second input harms performance; ILVR’s wide conditioning favors fidelity and its narrow conditioning favors visual quality.Source: §4.3 · pp. 6–7, Figure 6 · p. 7, Figure 7 · p. 7
4.5. Discussion and Limitations
- DR2 has slow sampling despite using only 0.25T steps, requires manual selection of N and τ, and may remove details from inputs with very slight or no degradation.The authors are exploring image-quality scores such as NIQE to automate parameter search.Source: §4.5 · pp. 7–8
4.4. Effect of Different N and τ
- Larger N and τ remove more degradation but reduce input fidelity, and quantitative results show PSNR and identity distance can peak together before outputs deviate as τ grows.For N = 2, PSNR stops increasing before identity distance reaches its optimum because Gaussian noise appears; optimal τ is larger for smaller N.Source: §4.4 · p. 7, Figure 9 · p. 8, Figure 8 · p. 8
5. Conclusion
DR2E pairs a pretrained DDPM-based degradation remover with an enhancement module for detail restoration. The framework uses diffused low-quality inputs without synthetically degraded training data, and comparisons report robustness and high restoration quality.
5. Conclusion
- DR2E combines a pretrained DDPM-based degradation remover with an enhancement module for detail restoration.DR2 uses diffused low-quality information to condition the generative process.Source: §5 · p. 8
- The DR2 transformation requires no synthetically degraded data for training.Source: §5 · p. 8
- Extensive comparisons demonstrate strong robustness and high restoration quality for DR2E.Source: §5 · p. 8
Appendix
The appendix adds ablations of DR2's initial-condition and iterative-refinement mechanisms, detailed control-parameter settings, and further qualitative comparisons.
Appendix
- Appendix A examines how initial condition and iterative refinement affect DR2's generative process.The ablations address the control and conditioning effects of these mechanisms.Source: Appendix · p. 11
- Appendix B reports DR2 control-parameter settings for each testing dataset.Source: Appendix · p. 11
- Appendix B adds qualitative comparisons across CelebA-Test splits with different degradation levels.Source: Appendix · p. 11
A. More Ablation Studies
The ablations isolate DR2's initial-condition and iterative-refinement effects by evaluating its outputs without an enhancement module. Bicubic-downsampled images serve as low-resolution ground truth because DR2 aims to produce clean but blurry intermediate results.
A. More Ablation Studies
- The ablations evaluate DR2 outputs without enhancement to avoid variation in enhancement-module structure, facial priors, and training strategies.Source: A. More Ablation Studies · p. 11
- Bicubic-downsampled images serve as ground-truth low-resolution images for evaluating DR2 output degradation removal and fidelity.Source: A. More Ablation Studies · p. 11
- DR2 is intended to produce clean but blurry intermediate results.Source: A. More Ablation Studies · p. 11
A.1. Conditioning Effect of Initial Condition with Iterative Refinement Enabled
With iterative refinement enabled, varying the initial-condition parameter ω has little reported effect on DR2 output quality or appearance, while the initial condition reduces denoising steps. Without refinement, its control is limited, especially at large ω.
A.1. Conditioning Effect of Initial Condition with Iterative Refinement Enabled
- With iterative refinement, DR2 output quality and fidelity are not evidently affected as ω varies when (N, τ) = (4, 300).The reported evaluation uses PSNR and identity distance (Deg) on CelebA-Test (8×, medium split).Source: §A.1 · p. 11
- With fixed iterative-refinement parameters, ω has little visual effect on DR2 outputs.Figure A1 uses CelebA-Test (8×, medium split) with (N, τ) = (4, 300).Source: §A.1 · p. 11, Figure A1 · p. 11
- The initial condition provides limited information compared with iterative refinement but significantly reduces total DR2 denoising steps.Source: §A.1 · p. 11
- Without iterative refinement, the initial condition provides limited generative-process control, especially when ω is big.With refinement, its effect on output quality is little as long as ω − τ is not too small.Source: Table A2 · p. 11, Table A1 · p. 11
A.2. Conditioning Effect of Initial Condition with Iterative Refinement Disabled
Without iterative refinement, DR2 depends on its initial condition to use input information, while increasing ω reduces input fidelity; iterative refinement helps preserve semantic information.
A.2. Conditioning Effect of Initial Condition with Iterative Refinement Disabled
- Without iterative refinement, DR2 relies solely on the initial condition for input information and generates images stochastically through DDPM denoising.In this setting, ω controls how much conditioning information is provided.Source: §A.2 · p. 11
- Figure A2 samples rows at τ = 300, 150, and 0; at τ = 0, truncated output is no longer needed.The DR2 outputs with iterative refinement use (N, τ) = (4, 300).Source: Figure A2 · p. 12
- More severely degraded datasets use larger N and τ, while less degraded datasets use smaller values.Table A3 lists settings for CelebA-Test, WIDER-Critical, WIDER-Normal, CelebChild, and LFW-Test.Source: Table A3 · p. 12
- PSNR and Deg are worse than in Tab. A1 and negatively correlated with ω when iterative refinement is disabled.The authors attribute the negative correlation to using less input information as ω increases.Source: §A.2 · pp. 11–12
- When ω ≥ 400, initial-condition noise covers input degradation, producing smooth, clean outputs that become less relevant to the input as ω rises.The authors note that iterative refinement is important for preserving semantic information.Source: §A.2 · p. 12
B.1. Controlling Parameter Settings
The experiments span synthetic upsampling splits at three degradation levels and four real-world datasets, with controlling parameters varied by dataset or split.
B.1. Controlling Parameter Settings
- Experiments cover mild, medium, and severe splits for 16×, 8×, and 4× upsampling, plus four real-world datasets.Different (N, τ) settings are used for each dataset or split.Source: §B.1 · p. 12
- Larger N and τ remove degradation more effectively but reduce fidelity; smaller values trade removal effectiveness for higher fidelity.The settings vary across datasets and splits.Source: §B.1 · p. 12
B.2. More Qualitative Comparisons
Qualitative comparisons show DR2’s robustness on severe degradation, while its transformation is less necessary for slight degradation.
B.2. More Qualitative Comparisons
- On severe degradation, previous methods fail while DR2 remains robust despite using no synthetic degraded training images.The comparison spans CelebA-Test splits and upsampling factors.Source: §B.2 · p. 12
- For heavily degraded ×16 and ×8 CelebA-Test inputs, previous methods produce more artifacts.Figures A1 and A2 show these qualitative comparisons.Source: Figure A1 · p. 13, Figure A2 · p. 13
- On mildly degraded data, DR2 produces results comparable to previous methods.Figure A3 shows the ×4 CelebA-Test comparison.Source: §B.2 · p. 12, Figure A3 · p. 13