Source-linked AI summary
Learning Warped Guidance for Blind Face Restoration
Xiaoming Li, Ming Liu, Yuting Ye, Wangmeng Zuo, Liang Lin, Ruigang Yang
TL;DR
Blind face restoration must recover detailed faces despite unknown degradations, and single degraded observations can lose fine identity information. The paper introduces GFRNet, which warps a same-identity high-quality guide before reconstruction using landmark and total-variation guidance. It reports favorable quantitative and perceptual performance against restoration methods, including on real degraded images.
Problem
Blind restoration must handle unknown degradation types and parameters, while single degraded observations may over-smooth and fail to retain fine facial details.
Method
GFRNet combines WarpNet and RecNet to align a same-identity high-quality guided image and reconstruct the face, using landmark loss, TV regularization, and synthetic varied degradations.
Results
GFRNet achieves significant gains over state-of-the-art methods in PSNR, SSIM, and perceptual quality, and produces visually pleasing results on real low-quality images.
Takeaways & Limitations
A high-quality same-identity guide can support blind face restoration when it is aligned to the degraded observation before reconstruction.
Takeaways & Limitations
The method assumes the guided image is high-quality, frontal, non-occluded, and has eyes open, although the authors report robustness when this assumption is violated.
Abstract
from arXiv · showhide
This paper studies the problem of blind face restoration from an unconstrained blurry, noisy, low-resolution, or compressed image (i.e., degraded observation). For better recovery of fine facial details, we modify the problem setting by taking both the degraded observation and a high-quality guided image of the same identity as input to our guided face restoration network (GFRNet). However, the degraded observation and guided image generally are different in pose, illumination and expression, thereby making plain CNNs (e.g., U-Net) fail to recover fine and identity-aware facial details. To tackle this issue, our GFRNet model includes both a warping subnetwork (WarpNet) and a reconstruction subnetwork (RecNet). The WarpNet is introduced to predict flow field for warping the guided image to correct pose and expression (i.e., warped guidance), while the RecNet takes the degraded observation and warped guidance as input to produce the restoration result. Due to that the ground-truth flow field is unavailable, landmark loss together with total variation regularization are incorporated to guide the learning of WarpNet. Furthermore, to make the model applicable to blind restoration, our GFRNet is trained on the synthetic data with versatile settings on blur kernel, noise level, downsampling scale factor, and JPEG quality factor. Experiments show that our GFRNet not only performs favorably against the state-of-the-art image and face restoration methods, but also generates visually photo-realistic results on real degraded facial images.
1 Introduction
The paper frames guided blind face restoration as a way to recover fine, identity-aware facial details from unknown degradations by combining a degraded observation with a same-identity high-quality guide. GFRNet aligns the guide before reconstruction and reports gains over competing restoration methods on synthetic and real images.
- Motivation: Blind face restoration must handle unknown degradation types and parameters, while single-observation mappings can over-smooth fine facial details.The motivating degradations include low resolution, defocus, noise, and compression.
- Guided restoration: A same-identity high-quality guided image can ease blind restoration and help recover fine, identity-aware facial details.The setting is described as feasible for grouped smartphone images and film restoration.
- Challenge: Plain CNNs provide limited improvement when degraded and guided images differ in pose, expression, or lighting.Directly supplying both images to U-Net does not adequately exploit the guide.
- Method: WarpNet trained only with reconstruction and adversarial learning may fail to align the guide, especially when the two images have distinctly different poses.The paper motivates landmark supervision because the ground-truth flow field is unavailable.
- Method: GFRNet combines a warping subnetwork and reconstruction subnetwork, with landmark loss and total-variation regularization used to train WarpNet.WarpNet aligns the guided image to the desired pose and expression before reconstruction.
2 Related Work
Prior restoration research largely targets specialized, non-blind tasks, while guided image methods generally assume spatial alignment. GFRNet addresses the alignment challenge by learning dense warping for guided face restoration.
- Existing restoration: Most face restoration methods target specialized tasks such as hallucination, denoising, or deblurring and are non-blind.Blind restoration instead requires one model to handle varied instances of one or more degradation types.
- Blind restoration: Deep blind restoration research has focused mainly on blind deblurring, often estimating blur kernels or learning to bypass explicit kernel estimation.These approaches address a narrower degradation setting than guided blind face restoration.
- Guided restoration: CNN-based guided depth enhancement and filtering methods exploit structural detail but assume the guided image is spatially aligned with the degraded image.That alignment assumption limits direct transfer to differently posed face images.
- Learned warping: Spatial transformer and appearance-flow networks learn image warping or dense flow, but related training settings generally require similar lighting between target and warped input.These techniques have been applied to view synthesis, gaze manipulation, expression editing, and video frame synthesis.
3 Proposed Method
GFRNet restores degraded faces by first warping a same-identity guided image into alignment, then reconstructing the face from the degraded observation and warped guidance. It is trained with landmark- and regularization-based warping supervision and synthetic degradations spanning blur, downsampling, noise, and JPEG compression.
- Guided Face Restoration Network: GFRNet combines WarpNet, which predicts a dense flow field to deform the guided image, with RecNet, which reconstructs the face from warped guidance and the degraded observation.The warped guidance is intended to match the target’s pose and expression before reconstruction.
- Warping Subnetwork (WarpNet): WarpNet predicts a dense flow field Φ from the degraded observation and guided image, producing warped guidance Iw through differentiable image warping.The flow field is used to deform the guided image rather than directly predicting the warped image.
- Warping Subnetwork (WarpNet): Landmark loss, total variation regularization, and gradients from RecNet facilitate WarpNet learning when the ground-truth flow field is unavailable.These signals guide alignment of the warped guidance with the target face.
- Reconstruction Subnetwork (RecNet): RecNet uses a U-Net with skip connections because its inputs and output share pose and expression, supporting reconstruction of rich and fine details.The RecNet receives Id and Iw and produces the restoration result.
- Degradation Model and Synthetic Training Data: The synthetic degradation model combines Gaussian blur, bicubic downsampling, additive white Gaussian noise, and JPEG compression, with parameters sampled across versatile settings.The model can represent individual degradations or their combinations, and bicubic interpolation restores the degraded observation to the guided image size when needed.
Losses on Restoration Result ˆI.
The restoration objective combines reconstruction, adversarial, and flow losses to constrain fidelity, perceptual quality, and guided-image alignment. Reconstruction uses ℓ2 and VGG-Face perceptual terms, while adversarial training applies global and local discriminators; landmark and TV terms guide WarpNet.
- Adversarial loss: Global and local adversarial losses are incorporated to improve the perceptual quality of the restoration result.The local discriminator operates on the minimal bounding box enclosing the facial landmarks, while the global discriminator evaluates the full image.
- Flow loss: Landmark loss aligns warped-guidance landmarks with those of the degraded observation, while total variation regularization requires the dense flow field to be spatially smooth.The landmark coordinates are normalized to [−1, 1] in implementation.
- Overall objective: The overall objective combines reconstruction loss, adversarial loss, and flow loss as L = Lr + La + Lflow.Flow loss combines landmark supervision with total variation regularization to guide WarpNet toward spatially aligned, smooth flow fields.
4 Experimental Results
Experiments evaluate GFRNet on synthetic and real low-quality faces, compare it with restoration methods, and test its components and degradation settings. GFRNet achieves strong quantitative and qualitative results, while ablations support the roles of warping and flow loss.
- Quantitative Results: GFRNet achieves significant PSNR and SSIM gains over competing restoration methods on two test subsets.The evaluation covers super-resolution, deblurring, denoising, compression artifact removal, and face hallucination methods.
- Quantitative Results: More than 4 dB PSNR improvement is reported over SR and blind deblurring methods for 4× SR on WebFace.The same comparison reports gaps exceeding 3.5 dB over denoising, 8 dB over artifact removal, and 5 dB over face hallucination methods.
- Quantitative Results: GFRNet remains substantially better than retrained competing models using the same training data and guided inputs.This comparison attributes the remaining performance gains to GFRNet’s architecture and model objective rather than only its data or guidance input.
- Synthetic and Visual Results: GFRNet restores finer details and suppresses visual artifacts more effectively than competing methods at 4× and 8× SR.The reported improvements are especially visible around the eyes, nose, and mouth, with visually photo-realistic outputs.
- Real Low-Quality Images: On real images below 60 × 60 resolution with unknown degradation, GFRNet produces realistic facial details and robust results across poses.Competing methods provide only moderate visual-quality improvements in this setting.
- Ablation Studies: Removing flow loss prevents stable alignment, whereas the full model correctly aligns guided images to the desired pose and expression.The full model also produces sharper and richer details than its variants.
- Ablation Studies: The full model outperforms parameter-matched wider variants, indicating gains mainly from combining WarpNet with flow loss rather than increasing parameter count.The comparison uses Ours(−W2) and Ours(−WG2), whose channels are doubled relative to corresponding ablations.
- Degradation Settings: Training with only blurring or blurring plus downsampling leaves real-image results blurry, while richer degradation settings improve restoration.The experiment compares four degradation settings on a real low-quality image because synthetic comparisons would be unfair across differently trained models.
5 Conclusion
The paper concludes that GFRNet combines guided restoration with learned warping and flow supervision for blind face restoration. Quantitative and qualitative experiments show favorable performance against related methods and pleasing results on real low-quality faces.
- Conclusion: GFRNet uses a same-identity high-quality guide, WarpNet, landmark loss, and TV regularization to align guidance for blind restoration.A general degradation model is used to synthesize realistic low-quality face images for training.
- Conclusion: GFRNet performs favorably against relevant state-of-the-art methods and generates visually pleasing results on real low-quality face images.The conclusion summarizes both quantitative and qualitative evidence.