Source-linked AI summary
Towards Real-World Blind Face Restoration with Generative Facial Prior
Xintao Wang, Yu Li, Honglun Zhang, Ying Shan
TL;DR
Blind face restoration lacks reliable geometry priors for very low-quality inputs and accessible high-quality references. GFP-GAN incorporates a pretrained face GAN through CS-SFT layers and complementary losses, achieving a balance of realism and fidelity in one forward pass. Experiments report superior performance to prior art on synthetic and real-world datasets.
Problem
Very low-quality inputs cannot provide accurate geometric priors, while high-quality references are inaccessible for real-world blind face restoration.
Method
GFP-GAN combines a degradation-removal module with a pretrained face GAN, connecting them through latent mapping and CS-SFT layers with facial-detail and identity-preserving losses.
Results
GFP-GAN achieves a balance of realness and fidelity and reports superior performance to prior art on synthetic and real-world datasets.
Takeaways & Limitations
The generative facial prior supports joint face restoration and color enhancement in a single forward pass.
Takeaways & Limitations
Severe real-image degradation can produce twisted facial details with artifacts, and very large poses can yield unnatural results.
Abstract
from arXiv · showhide
Blind face restoration usually relies on facial priors, such as facial geometry prior or reference prior, to restore realistic and faithful details. However, very low-quality inputs cannot offer accurate geometric prior while high-quality references are inaccessible, limiting the applicability in real-world scenarios. In this work, we propose GFP-GAN that leverages rich and diverse priors encapsulated in a pretrained face GAN for blind face restoration. This Generative Facial Prior (GFP) is incorporated into the face restoration process via novel channel-split spatial feature transform layers, which allow our method to achieve a good balance of realness and fidelity. Thanks to the powerful generative facial prior and delicate designs, our GFP-GAN could jointly restore facial details and enhance colors with just a single forward pass, while GAN inversion methods require expensive image-specific optimization at inference. Extensive experiments show that our method achieves superior performance to prior art on both synthetic and real-world datasets.
1. Introduction
Blind face restoration is difficult because real-world degradation weakens geometry priors and high-quality references are often unavailable. GFP-GAN instead uses a pretrained face GAN to provide diverse facial priors while balancing realism and fidelity.
- Very low-quality inputs degrade estimated landmarks, parsing maps, and facial component heatmaps used as geometry priors.
- High-resolution references are inaccessible in practice, while component dictionaries limit the diversity and richness of recoverable facial details.
- Pretrained face GANs encapsulate geometry, facial textures, and colors, enabling joint facial-detail restoration and color enhancement.
- GFP-GAN connects a degradation-removal module with a pretrained face GAN through latent code mapping and coarse-to-fine CS-SFT layers.CS-SFT spatially modulates split features while allowing the remaining features to pass through for information preservation.
- GFP-GAN combines facial-component loss with local discriminators and identity-preserving loss to enhance perceptual details while retaining fidelity.
- Extensive experiments report superior performance to prior art on both synthetic and real-world datasets.
2. Related Work
Face restoration methods use general restoration techniques, facial geometry priors, reference priors, and component dictionaries to improve realism and fidelity. GFP-GAN differs by using a pretrained face GAN as a rich, whole-face generative prior.
- Image restoration commonly includes super-resolution, denoising, deblurring, and compression removal.
- Generative adversarial networks are commonly used as loss supervision to push restoration solutions toward the natural manifold.
- Face restoration incorporates geometry priors such as landmarks, parsing maps, and facial component heatmaps to improve performance.
- Reference-prior methods often require same-identity reference images, while DFDNet uses component dictionaries to guide restoration.
- DFDNet degrades beyond dictionary coverage, whereas GFP-GAN treats faces as a whole and uses richer, more diverse priors.
3. Methodology
GFP-GAN combines a degradation-removal U-Net with a pretrained face GAN, using latent mapping and CS-SFT layers to balance realistic facial details with input fidelity. Its training further uses multi-scale reconstruction, adversarial, facial-component, and identity-preserving objectives.
- Framework: GFP-GAN combines a U-Net degradation-removal module with a pretrained face GAN, connected by latent code mapping and multi-resolution CS-SFT layers.The U-Net extracts latent and spatial features for conditioning the pretrained GAN.
- Degradation Removal: The degradation-removal module targets mixtures of low resolution, blur, noise, and JPEG artifacts while extracting clean latent and spatial features.Its U-Net structure increases receptive field for large blur elimination and supports multi-resolution feature extraction.
- Generative Facial Prior: The latent features are mapped to intermediate latent codes W, which pass through the pretrained GAN to generate multi-resolution facial features.These GAN features provide learned facial details, while spatial features condition them during restoration.
- Channel-Split Spatial Feature Transform: CS-SFT modulates only part of the GAN features with spatial input features and directly passes the remaining channels, balancing fidelity with realness.The split preserves input information while retaining the GAN’s facial prior; it also reduces modulation complexity.
- Restoration and Color Enhancement: The model applies CS-SFT at each resolution scale to produce a restored face and jointly supports facial-detail restoration and color enhancement.The generative facial prior contains facial textures and color information useful for these tasks.
- Training Objectives: Training combines reconstruction and adversarial losses with facial-component and identity-preserving losses to improve realism, local details, and identity retention.Local discriminators focus on the left eye, right eye, and mouth, while ArcFace features provide identity guidance.
4. Experiments
Experiments evaluate GFP-GAN on synthetic and real-world datasets against face restoration, GAN inversion, and general image restoration methods. Results show strong perceptual quality, identity preservation, qualitative fidelity, generalization, and a realness–fidelity trade-off enabled by CS-SFT layers, while severe degradation and large poses remain challenging.
- Datasets and Implementation: GFP-GAN is trained on FFHQ images using synthetic degradations combining blur, downsampling, noise, JPEG compression, and color jittering.The model uses 70,000 high-quality FFHQ images and is evaluated on one synthetic and three real-world datasets.
- Datasets and Implementation: The evaluation covers CelebA-Test, LFW-Test, CelebChild-Test, and WebPhoto-Test, spanning synthetic corruption and diverse real-world degradation.WebPhoto-Test includes 407 faces from 188 low-quality photographs, including severe detail and color degradation.
- Comparisons with State-of-the-art Methods: GFP-GAN achieves the lowest LPIPS, FID, and NIQE on both CelebA-Test settings while retaining better identity than competing methods.The authors caution that lower FID or NIQE than ground truth does not imply superior performance, and PSNR and SSIM correlate poorly with human judgments here.
- Comparisons with State-of-the-art Methods: Qualitatively, GFP-GAN restores faithful eyes, teeth, mouth, hair, ears, and eye-gaze direction while avoiding forced teeth and inaccurate hair textures.Its whole-face treatment contrasts with component dictionaries and parsing-map methods that struggle beyond their modeled regions.
- Comparisons with State-of-the-art Methods: On LFW, CelebChild, and WebPhoto-Test, GFP-GAN achieves superior performance, demonstrating generalization to real-world images with distinct sources and degradations.PULSE can obtain high perceptual quality but does not retain face identity as well, according to the authors.
- Ablation Studies: Ablations show that spatial features in CS-SFT preserve identity, while CS-SFT balances realness and fidelity better than simple SFT or omitting the generative prior.Removing spatial modulation harms identity; replacing CS-SFT degrades perceptual metrics but strengthens identity; removing GFP causes a performance drop.
- Discussion and Limitations: For grayscale inputs, face color may be biased, and severe real-image degradation or very large poses can produce artifacts or unnatural results.The authors attribute these limitations to differences between synthetic training degradation and real-world data distributions.
5. Conclusion
GFP-GAN uses rich generative facial priors with channel-split spatial feature transform layers for blind face restoration. The framework balances realness and fidelity while jointly restoring faces and enhancing colors in real-world images.
- GFP-GAN leverages rich and diverse generative facial priors for blind face restoration.
- Channel-split spatial feature transform layers incorporate the generative facial prior into the restoration process.
- The framework achieves a good balance of realness and fidelity.
- GFP-GAN jointly performs face restoration and color enhancement for real-world images, outperforming prior art.