PaperlayerView exact revision on arXiv

Source-linked AI summary

Blind Face Restoration via Deep Multi-scale Component Dictionaries

Xiaoming Li, Chaofeng Chen, Shangchen Zhou, Xianhui Lin, Wangmeng Zuo, Lei Zhang

arXiv:2008.00418v1cs.CV

TL;DR

Blind face restoration must recover realistic details from real low-quality images, while existing reference-based methods often require identity-matched high-quality references. DFDNet builds multi-scale dictionaries of facial components and transfers matched details through normalized, confidence-weighted progressive fusion, achieving promising results on synthetic and real degraded images without identity-belonging references.

  • Problem

    Reference-based face restoration methods generally require a high-quality reference image of the same identity, limiting their applicability to broader restoration scenes.

  • Method

    DFDNet builds K-means dictionaries of eye, nose, and mouth features from high-quality faces, then uses CAdaIN, feature matching, confidence scoring, and progressive multi-scale DFT to transfer details.

  • Results

    DFDNet achieves promising quantitative and qualitative performance on synthetic and real degraded images without requiring an identity-belonging high-quality reference.

  • Takeaways & Limitations

    Deep multi-scale component dictionaries provide flexible reference details for blind face restoration without identity-specific high-quality images.

  • Takeaways & Limitations

    The method depends on a fixed VggFace encoder matching the dictionary-generation network to keep input and dictionary features in the same feature space.

Abstract

from arXiv · show

Recent reference-based face restoration methods have received considerable attention due to their great capability in recovering high-frequency details on real low-quality images. However, most of these methods require a high-quality reference image of the same identity, making them only applicable in limited scenes. To address this issue, this paper suggests a deep face dictionary network (termed as DFDNet) to guide the restoration process of degraded observations. To begin with, we use K-means to generate deep dictionaries for perceptually significant face components (\ie, left/right eyes, nose and mouth) from high-quality images. Next, with the degraded input, we match and select the most similar component features from their corresponding dictionaries and transfer the high-quality details to the input via the proposed dictionary feature transfer (DFT) block. In particular, component AdaIN is leveraged to eliminate the style diversity between the input and dictionary features (\eg, illumination), and a confidence score is proposed to adaptively fuse the dictionary feature to the input. Finally, multi-scale dictionaries are adopted in a progressive manner to enable the coarse-to-fine restoration. Experiments show that our proposed method can achieve plausible performance in both quantitative and qualitative evaluation, and more importantly, can generate realistic and promising results on real degraded images without requiring an identity-belonging reference. The source code and models are available at \url{https://github.com/csxmli2016/DFDNet}.

1 Introduction

Blind face restoration seeks realistic high-quality details from real low-quality images without known degradation parameters, but reference-based methods typically require identity-matched high-quality images. DFDNet addresses this with multi-scale component dictionaries and progressive dictionary feature transfer, achieving promising results on synthetic and real degraded faces.

  • Blind face restoration recovers realistic details from real low-quality images without knowing degradation types or parameters.
  • Reference-based methods improve restoration but commonly rely on high-quality identity-matched references, limiting their applicability.
  • DFDNet builds deep dictionaries for left/right eyes, nose, and mouth from high-quality face images to provide component references without identity-matched images.
  • DFDNet achieves promising quantitative and qualitative performance on synthetic and real degraded images and is presented as flexible for face restoration applications.
  • The DFT block uses component AdaIN, confidence-weighted fusion, and progressive multi-scale transfer to adapt dictionary details to different inputs and degradation levels.

2 Related Work

Single-image face restoration methods often rely on geometric priors that do not directly supply facial details, while reference-based methods provide stronger detail guidance but require constrained references. DFDNet instead uses multi-scale component dictionaries to offer diverse references without identity matching.

  • 2.1 Single Image Restoration: Single-image restoration methods commonly use landmarks, heatmaps, parsing maps, or attributes, but these priors may not provide direct facial details.
  • 2.1 Single Image Restoration: The ill-posed nature of real low-quality face restoration and limited single-image or structural priors can prevent plausible realistic details.
  • 2.2 Reference-Based Image Restoration: Reference-based restoration adds an image to guide recovery with structural or texture information from related content.
  • 2.2 Reference-Based Image Restoration: Face reference methods use identity-aware features from fixed frontal high-quality images, but unconstrained poses, expressions, backgrounds, and illumination complicate this setup.
  • 2.2 Reference-Based Image Restoration: DFDNet uses multi-scale component dictionaries and progressive DFT blocks, allowing more reference candidates without requiring an identity-belonging reference.

3 Proposed Method

DFDNet restores degraded faces by matching their component features to multi-scale dictionaries built offline from high-quality images, then transferring selected details progressively from coarse to fine. The network normalizes feature distributions, scores confidence, and fuses dictionary features to accommodate varied degradation.

  • 3.2 Deep Face Dictionary Network: Given a degraded face, DFDNet selects dictionary features with the most similar component structure and transfers them through a dictionary feature transformation pipeline.
  • 3.2 Deep Face Dictionary Network: Component AdaIN re-normalizes dictionary features to the input component's distribution or style before feature matching.
  • 3.2 Deep Face Dictionary Network: A confidence score based on the residual between selected dictionary and input features supports weighted fusion across degradation levels, while progressive transfer moves from coarse to fine.
  • 3.1 Off-line Generation of Component Dictionaries: DFDNet first generates multi-scale deep component dictionaries from high-quality faces using VggFace features, facial landmarks, RoIAlign, and K-means clustering.
  • 3.1 Off-line Generation of Component Dictionaries: The dictionaries are built from 10,000 selected FFHQ images and contain clustered left/right eye, nose, and mouth features across scales.
  • 3.2 Deep Face Dictionary Network: The DFDNet encoder uses the same fixed VggFace architecture and parameters as dictionary generation to keep input and dictionary features in the same feature space.

CAdaIN.

CAdaIN re-normalizes component dictionary clusters to reduce style differences from unconstrained inputs and facilitate accurate feature matching.

  • CAdaIN.: CAdaIN re-normalizes dictionary clusters to reduce differences such as illumination and skin color between inputs and dictionary features.
  • CAdaIN.: AdaIN preserves structure while translating content to the desired style, giving re-normalized dictionaries a distribution similar to the input components.
  • CAdaIN.: The input component feature is matched against all re-normalized clusters using inner-product similarity.
  • CAdaIN.: The inner-product operation can be implemented as a zero-bias convolution, efficiently producing similarity scores across dictionary clusters.
  • CAdaIN.: DFDNet selects the highest-scoring re-normalized cluster and uses it to provide high-quality details for restoring the input component.
  • CAdaIN.: The method accounts for degradation level because lightly degraded inputs rely less on dictionaries than heavily degraded inputs.

Confidence Score.

A confidence score controls how the selected dictionary feature is fused with the degraded input, aiming to restore details absent from the input.

  • Confidence Score.: DFDNet predicts a confidence score from the input components and the selected dictionary feature.
  • Confidence Score.: The fused result is intended to add absent high-quality details back to the input component feature.
  • Confidence Score.: The confidence score block uses learnable parameters ΘC.

Reverse RoIAlign.

Reverse RoIAlign places processed facial components back at their original locations, while preserving other features for decoder-based restoration.

  • Reverse RoIAlign.: Reverse RoIAlign maps processed eye, nose, and mouth features back to their original locations in the decoder feature.
  • Reverse RoIAlign.: This operation translates non-component features, such as background information, to the decoder for improved restoration.
  • Reverse RoIAlign.: The scale-s SFT layer modulates decoder features using learned affine parameters derived from restored features.
  • Reverse RoIAlign.: After progressive DFT processing, DFDNet gradually learns fine details for the final restored image.

3.3 Model Objective

DFDNet is trained with reconstruction and adversarial objectives to keep outputs close to ground truth while recovering realistic details.

  • 3.3 Model Objective: The reconstruction loss constrains the restored result ˆI to remain close to the ground-truth image Ih.
  • 3.3 Model Objective: The adversarial loss targets realistic details in the restored output.

Reconstruction Loss.

DFDNet’s reconstruction loss combines pixel-space and feature-space mean square error to balance stable reconstruction with perceptual visual quality.

  • The reconstruction loss applies mean square error in both pixel space and feature space through a perceptual loss.The feature-space term uses VggFace activations.
  • The pixel-space term tends to produce blurry results, whereas the perceptual term improves the visual quality of reconstructions.
  • The two loss terms jointly support stable neural-network training, with four VggFace convolutional layers used experimentally.The number of layers is M = 4.

Adversarial Loss.

DFDNet uses adversarial training with discriminators operating at multiple restoration-result scales, alongside reconstruction loss.

  • Multi-scale discriminators are applied at different sizes of the restoration results to generate realistic details.
  • Spectral normalization is added after each discriminator convolution layer to support stable training.
  • The discriminator losses use down-sampling scales r ∈ {1, 2, 4, 8}.
  • The full DFDNet objective combines reconstruction loss with adversarial loss.

4 Experiments

Experiments compare DFDNet with reference-based, face-prior, and single-image restoration methods across synthetic and real low-quality face restoration settings. DFDNet achieves strong quantitative results and produces plausible qualitative restorations without identity-matched references.

  • Experimental Setup: Experiments compare DFDNet with reference-based, face-prior, and single-image methods on two datasets and ×4/×8 super-resolution tasks.The evaluation reports PSNR, SSIM, and LPIPS, using retrained baselines where needed for fair comparison.
  • Quantitative Results: At least 0.4 dB in ×4 and 0.3 dB in ×8 separate DFDNet from the second-best method across both datasets.These comparisons are reported for the quantitative super-resolution evaluation.
  • Quantitative Results: Reference-based methods outperform other competing methods, but DFDNet surpasses them using component dictionaries and progressive dictionary feature transfer.The authors attribute DFDNet’s advantage to having more component-reference candidates available for degraded inputs.
  • Quantitative Results: DFDNet generalizes across datasets: dictionaries built on FFHQ still support performance above competing methods on CelebA.DFDNet is trained and validated on VggFace2, while the test sets include non-overlapping VggFace2 images and CelebA images.
  • Visual Comparisons: On ×4 and ×8 visual comparisons, DFDNet reconstructs promising facial structure with richer eye and mouth details than competing methods.The comparisons also report plausible performance when the degraded input is non-frontal.
  • Real-world Evaluation: On real low-quality images, DFDNet produces plausible and realistic results, while GFRNet can retain artifacts from an inconsistent single reference.DFDNet does not require an identity-belonging high-quality reference in this evaluation.
  • Ablation Study: Adding progressive DFT blocks yields at least 2.2 dB in ×4 and 0.6 dB in ×8 gains, while removing CAdaIN severely degrades performance.Confidence scores improve fusion of input and matched dictionary features; without them, slight degradation can cause unwanted component changes.

5 Conclusion

DFDNet addresses the dependence on identity-specific high-quality references by using multi-scale component dictionaries and progressive dictionary feature transfer for blind face restoration.

  • DFDNet clusters perceptually significant facial components from high-quality images to avoid requiring identity-belonging references.The dictionaries cover the left and right eyes, nose, and mouth.
  • Its dictionary feature transfer uses component AdaIN, fast feature matching, and confidence scores to combine dictionary details with degraded inputs.
  • Multi-scale component dictionaries are integrated progressively so DFDNet learns coarse-to-fine facial details.
  • Experiments validate DFDNet on synthetic and real-world low-quality images without an identity-belonging reference.The authors describe this capability as having practical value across wide real-world application scenes.
Loading 2008.00418v1…