PaperlayerView exact revision on arXiv

Source-linked AI summary

DifFace: Blind Face Restoration with Diffused Error Contraction

Zongsheng Yue, Chen Change Loy

arXiv:2212.06512v4cs.CV

TL;DR

Blind face restoration methods can fail under unseen, complex degradations and often require multiple difficult-to-balance losses. DifFace instead constructs a posterior through an L1-trained transition distribution and a reverse chain from a pretrained diffusion model, using error contraction to improve robustness. Experiments report strong performance on severe degradations and real-world restoration, while inference remains limited by iterative diffusion sampling.

  • Problem

    Existing BFR methods can deteriorate under degradation mismatch and require multiple constraints with laborious hyper-parameter tuning.

  • Method

    DifFace predicts an intermediate diffused state from the LQ image, then applies a reverse Markov chain from a pretrained diffusion model to estimate the HQ image.

  • Results

    DifFace is reported to be robust on severe degradations and achieves the best real-world performance on WIDER and LFW, while surpassing most recent BFR methods on WebPhoto.

  • Takeaways & Limitations

    Error contraction gives the transition distribution greater tolerance to restoration-backbone errors, supporting robustness to severe and complex degradations without complicated loss designs.

  • Takeaways & Limitations

    DifFace’s inference efficiency is limited by the iterative sampling process inherited from diffusion models, and it remains slower than CNN-based inpainting methods.

Abstract

from arXiv · show

While deep learning-based methods for blind face restoration have achieved unprecedented success, they still suffer from two major limitations. First, most of them deteriorate when facing complex degradations out of their training data. Second, these methods require multiple constraints, e.g., fidelity, perceptual, and adversarial losses, which require laborious hyper-parameter tuning to stabilize and balance their influences. In this work, we propose a novel method named DifFace that is capable of coping with unseen and complex degradations more gracefully without complicated loss designs. The key of our method is to establish a posterior distribution from the observed low-quality (LQ) image to its high-quality (HQ) counterpart. In particular, we design a transition distribution from the LQ image to the intermediate state of a pre-trained diffusion model and then gradually transmit from this intermediate state to the HQ target by recursively applying a pre-trained diffusion model. The transition distribution only relies on a restoration backbone that is trained with $L_2$ loss on some synthetic data, which favorably avoids the cumbersome training process in existing methods. Moreover, the transition distribution can contract the error of the restoration backbone and thus makes our method more robust to unknown degradations. Comprehensive experiments show that DifFace is superior to current state-of-the-art methods, especially in cases with severe degradations. Code and model are available at https://github.com/zsyOAOA/DifFace.

I. INTRODUCTION

Blind face restoration is difficult because real-world degradations are complex and mismatched with synthetic training data, while existing methods often require multiple carefully balanced losses. DifFace addresses these issues with a posterior-based diffusion approach that uses a transition distribution, error contraction, and a pretrained diffusion model.

  • Blind face restoration is an ill-posed inverse problem because one low-quality image can correspond to multiple high-quality faces.
  • Synthetic degradation assumptions often mismatch real degradations, causing existing restoration methods to lose performance on real-world inputs.
  • Existing methods combine fidelity, perceptual, adversarial, and face-specific constraints, making training complicated and hyper-parameter tuning laborious.
  • DifFace approximates p(x0|y0) with a transition distribution p(xN|y0) followed by a reverse Markov chain from an intermediate diffused state to the HQ image.
  • The transition distribution contracts restoration error, while the reverse chain reuses image priors from a pretrained diffusion model without retraining it on assumed degradations.
  • DifFace trains its restoration backbone with a simple L1 loss and reports a Markov chain that compresses predicted error by a factor below 1.

II. RELATED WORK

Prior work uses geometric, reference, generative, and diffusion priors to restore degraded faces, while diffusion models rely on forward corruption and learned reverse processes. DifFace builds on diffusion’s error-contraction property and pretrained generative prior.

  • BFR methods use geometric priors such as landmarks, parsing maps, component heatmaps, and 3D shapes to guide restoration.
  • Recent diffusion-based restorers condition diffusion models on LQ images, with related work also using pre-cleaning or latent-space inference to improve restoration or speed.
  • DiffBFR concurrently combines a pretrained diffusion model with identity restoration and texture enhancement modules for long-tail BFR distributions.
  • A diffusion model progressively destroys data through a forward Markov process and learns a reverse process to recover the data distribution.
  • DDPM predicts noise in noisy states, whereas DDIM generalizes it to a non-Markov process that can produce deterministic generation when η = 0.
  • Diffusion contracts the clean image by √αt, motivating DifFace’s use of an intermediate state whose residual to the HQ image is reduced.

IV. PROPOSED METHOD

DifFace restores HQ faces by exploiting image priors in a pretrained diffusion model and designing a posterior from LQ to HQ images. The method analyzes how the starting timestep affects the diffusion process.

  • DifFace aims to design a rational posterior distribution p(x0|y0) for restoring an HQ image x0 from its degraded LQ counterpart y0.
  • A pretrained diffusion model provides the reverse-diffusion transition used to reconstruct the HQ image from an intermediate state.
  • The diffusion model is trained with 1000 discrete steps, providing the timestep framework used by the restoration process.
  • The curves of κN and αN characterize how the starting timestep N changes the diffusion process.

A. Motivation

DifFace approximates the LQ-to-HQ posterior by predicting an intermediate diffused state and then applying pretrained reverse diffusion. Its starting timestep balances reconstruction fidelity against realism.

  • A. Motivation: The posterior p(x0|y0) is constructed from a transition distribution p(xN|y0) followed by reverse-diffusion transitions from timestep N to 1.
  • A. Motivation: Because pretrained diffusion supplies pθ(xt−1|xt), DifFace focuses on designing the transition distribution p(xN|y0).
  • A. Motivation: The target q(xN|x0) motivates approximating p(xN|y0), since the HQ image is unavailable during blind face restoration.
  • A. Motivation: For reasonable N values such as N < 500, the pretrained diffusion model can accurately reconstruct x0 after slight destruction by diffusion.
  • B. Design: DifFace models p(xN|y0) as a Gaussian whose mean uses a diffused estimator f(y0; w), while the final output is sampled from the entire Markov chain.
  • B. Design: The starting timestep N creates a realism–fidelity trade-off: larger N improves realism but introduces more noise and lowers fidelity to x0.

C. Algorithm

DifFace restores HQ images by first estimating a diffused intermediate state and then recursively sampling a pretrained diffusion model. Its posterior formulation contracts estimator errors, supports simple training, and extends to constrained restoration settings such as inpainting.

  • Posterior sampling: DifFace samples an intermediate xN from p(xN|y0), then applies the reverse Markov chain from t=N to 1 to obtain x0.The final restoration is generated by the full chain rather than directly by the diffused estimator.
  • Error contraction: Error contraction scales the diffused estimator’s predicted error by √αN, increasing tolerance to imperfect restoration under complex degradations.The authors argue this robustness also permits training the estimator with L1 loss on synthetic data.
  • Diffusion prior: A pretrained diffusion model supplies HQ image priors through recursive sampling, reducing dependence on manually synthesized degradation data.The diffusion model is trained unsupervised on HQ images and is not retrained for the assumed degradation model.
  • Design rationale: The constructed posterior is presented as a simple, robust BFR learning paradigm rather than a direct application of diffusion models.The formulation is designed specifically to mitigate robustness issues from complicated degradations.
  • Extension to inpainting: For inpainting, DifFace refines each estimated x0 according to the degradation constraint, balancing the diffusion prior and constraint with γ.The refinement is posed as an optimization problem with a closed-form solution when available, or one SGD step otherwise.

VI. EXPERIMENTAL RESULTS ON BFR

The experiments evaluate DifFace on BFR through component analyses and comparisons with state-of-the-art methods across synthetic and real-world datasets.

  • Experimental scope: The evaluation studies the starting timestep N, diffused estimator f(·; w), and DDIM hyper-parameter η before comparing DifFace with recent state-of-the-art methods.The comparisons cover both synthetic and real-world datasets.

A. Experimental setup

The experiments use synthetic and real-world face datasets, standard restoration metrics, multiple estimator backbones, and controlled analyses of timestep and sampling settings.

  • Datasets: DifFace is evaluated on CelebA-Test plus LFW, WebPhoto, and WIDER, which represent synthetic and varying real-world degradation settings.CelebA-Test contains 4,000 HQ images; the real-world datasets differ in degradation severity.
  • Metrics: The study measures PSNR, SSIM, LPIPS, identity score, landmark distance, and FID to assess restoration quality and identity preservation.On real-world data without ground truths, FID-F compares restored-image feature statistics with FFHQ.
  • Starting timestep: A larger starting timestep N increases realism but lowers fidelity, with strong performance reported for N in [400, 450].The experiments set N to 400 thereafter.
  • Diffused estimator: DifFace uses SRCNN or SwinIR as diffused estimators, with SwinIR producing more apparent details such as hairs.Even the simpler SRCNN backbone yields plausible HQ restorations, and subsequent experiments use SwinIR.
  • DDIM hyper-parameter: Lower η improves fidelity-oriented metrics but harms perceptual metrics, motivating the fixed setting η=0.5.The trade-off involves PSNR, IDS, and LMD versus SSIM, LPIPS, and FID.

C. Evaluation on Synthetic Data

On CelebA-Test, DifFace performs best or second best across five metrics and becomes more robust than CodeFormer and VQFR as degradation severity increases. Real-world evaluations also report strong FID-F performance, especially on WIDER and LFW, though non-reference metrics reveal a discrepancy.

  • DifFace achieves the best or second-best performance across all five CelebA-Test metrics.
  • As downsampling factors increase from 4 to 40, DifFace’s performance drops more gracefully and surpasses CodeFormer and VQFR under severe degradations.
  • DifFace achieves the best FID-F performance on WIDER and LFW and surpasses most recent BFR methods on WebPhoto.WebPhoto FID values may be unrepresentative because the dataset contains only 407 images.
  • On real-world examples, DifFace provides significantly better results than comparison methods when the LQ images are severely degraded.
  • DifFace performs weakly against GAN-based methods on WIDER non-reference metrics, while the diffusion model’s upper-bound scores also remain lower.The authors suggest these metrics may favor sharp images and artifacts over the smoother outputs encouraged by L2-trained diffusion models.

E. Multiple Predictions

DifFace can generate multiple diverse and plausible HQ reconstructions from one LQ input by varying diffusion random seeds. Its broader inpainting experiments show robustness to large occlusions and advantages over GAN-based methods, especially for severe masks.

  • E. Multiple Predictions: DifFace generates multiple diverse and plausible HQ solutions for one LQ image by changing the pretrained diffusion model’s random seed.
  • Inpainting comparison: Across four mask types, DifFace attains remarkable or comparable performance in averaged inpainting results.
  • Inpainting robustness: DifFace shows notable superiority on “Half” and “Expand” masks with large occlusions.
  • Inpainting comparison: Diffusion-based methods consistently outperform GAN-based counterparts on the inpainting task.
  • Qualitative results: Under large occlusions, diffusion-based methods produce more plausible results, while DifFace better preserves coherence in unmasked regions.

C. Hyper-parameter Analysis

DifFace’s inpainting refinement depends on the hyper-parameter γ, while the method’s diffusion sampling remains an efficiency limitation. Accelerated sampling can reduce runtime without evident performance loss under the reported setting.

  • Hyper-parameter γ: γ controls how strongly the degradation-model prior influences the inpainting optimization.
  • Hyper-parameter γ: With γ = 0, omitting the optimization produces visible incoherence between recovered and unmasked regions.
  • Inference efficiency: DifFace’s inference efficiency is limited by the iterative sampling process inherited from diffusion models.
  • Inference efficiency: For inpainting, DifFace is more efficient than the compared diffusion approaches but slower than CNN-based DeepFillv2 and LaMa.
  • Acceleration: Reducing sampling to 20 steps cuts runtime to 0.92s without evident performance loss, while remaining comparable to CodeFormer.

APPENDIX

The appendix describes the synthetic CelebA-Test construction, restoration backbones, inpainting estimator, and an additional super-resolution evaluation. DifFace is presented as a framework extending beyond blind face restoration and inpainting.

  • BFR data and degradation: CelebA-Test contains 4,000 CelebA-HQ images used to synthesize low-quality inputs with the manuscript’s degradation model.
  • BFR backbones: The BFR experiments use modified SRCNN and SwinIR restoration backbones for 512 × 512 images.
  • Inpainting estimator: The inpainting diffused estimator is borrowed from LaMa and uses fast Fourier convolution with an LQ image and mask as inputs.
  • Additional task: The study additionally evaluates DifFace on image super-resolution as a representative restoration task.
  • Additional task: The BFR-trained DifFace model can be directly applied to face image super-resolution by upsampling the LQ input.

1) Face Image Super-resolution:

DifFace is evaluated for face image super-resolution against SR3, showing comparable or better performance at 8x and stronger robustness at 16x while requiring less diffusion processing. Additional experiments compare DifFace with IIGDM on natural-image super-resolution and visualize restoration behavior across real-world tasks.

  • Face super-resolution: DifFace achieves better or comparable performance than SR3 on 8x face super-resolution, despite SR3 being specifically trained for bicubic upsampling.The comparison uses quantitative results in Table X and visual results in Fig. 14.
  • Face super-resolution: DifFace outperforms SR3 on both qualitative and quantitative comparisons when generalized to 16x face super-resolution.This result is presented as evidence of robustness to unknown degradations.
  • Face super-resolution: DifFace is more efficient than SR3 because it starts from the intermediate reverse-chain state xN rather than traversing the diffusion process from the beginning.SR3 passes through the whole reverse process, whereas DifFace initializes at an intermediate state.
  • Natural-image super-resolution: DifFace significantly surpasses IIGDM on quantitative and qualitative 4x natural-image super-resolution evaluations.The comparison is reported for 4x scaling from 64 to 256, with visual results shown in Fig. 15.
  • Natural-image super-resolution: DifFace preserves fidelity to the LQ image better than IIGDM, which the authors attribute to initialization from the diffusion model's intermediate state.IIGDM is evaluated with an unconditional diffusion model for a fairer practical comparison, producing lower overall performance than its original reported setting with image labels.
  • Additional evaluations: Visual comparisons also cover blind face restoration on three real-world datasets, different diffusion seeds on WIDER-Test, and face inpainting.These evaluations are shown in Figs. 16–18.
Loading 2212.06512v4…