Source-linked AI summary
GAN Prior Embedded Network for Blind Face Restoration in the Wild
Tao Yang, Peiran Ren, Xuansong Xie, Lei Zhang
TL;DR
Blind face restoration from severely degraded real-world images is difficult because degradation is unknown and complex, and direct DNN training can yield unacceptable or over-smoothed results. GPEN embeds a pre-trained face-generation GAN into a U-shaped DNN and fine-tunes the combined network, using deep and shallow features for complementary reconstruction controls. Experiments report significantly better quantitative and qualitative BFR performance than state-of-the-art methods, especially on severe real-world degradation.
Problem
Blind face restoration seeks photo-realistic high-quality faces from severely degraded real-world images despite unknown and complex degradation.
Method
GPEN embeds a pre-trained GAN as a decoder prior in a U-shaped DNN and fine-tunes the whole network using synthesized low-quality/high-quality face pairs.
Results
GPEN significantly outperforms state-of-the-art BFR methods quantitatively and qualitatively, restoring clear facial details while retaining image backgrounds.
Takeaways & Limitations
The method produces photo-realistic restorations for severely degraded faces in the wild and can also be applied to face inpainting and colorization.
Takeaways & Limitations
GPEN currently produces only one high-quality output for each low-quality input because style mixing may create inconsistent backgrounds.
Abstract
from arXiv · showhide
Blind face restoration (BFR) from severely degraded face images in the wild is a very challenging problem. Due to the high illness of the problem and the complex unknown degradation, directly training a deep neural network (DNN) usually cannot lead to acceptable results. Existing generative adversarial network (GAN) based methods can produce better results but tend to generate over-smoothed restorations. In this work, we propose a new method by first learning a GAN for high-quality face image generation and embedding it into a U-shaped DNN as a prior decoder, then fine-tuning the GAN prior embedded DNN with a set of synthesized low-quality face images. The GAN blocks are designed to ensure that the latent code and noise input to the GAN can be respectively generated from the deep and shallow features of the DNN, controlling the global face structure, local face details and background of the reconstructed image. The proposed GAN prior embedded network (GPEN) is easy-to-implement, and it can generate visually photo-realistic results. Our experiments demonstrated that the proposed GPEN achieves significantly superior results to state-of-the-art BFR methods both quantitatively and qualitatively, especially for the restoration of severely degraded face images in the wild. The source code and models can be found at https://github.com/yangxy/GPEN.
1. Introduction
Blind face restoration in the wild remains difficult because real low-quality faces have unknown, complex degradation, while existing approaches can fail or produce over-smoothed results. GPEN embeds and fine-tunes a GAN prior within a U-shaped DNN to restore global structure, local details, and background.
- Blind face restoration must recover photo-realistic high-quality faces from low-quality images with unknown and complex real-world degradation.
- Existing methods perform impressively on artificially degraded faces but fail to handle real-world low-quality face images in the wild.
- GPEN pre-trains a GAN for high-quality face generation, embeds it as a decoder prior in a DNN, and fine-tunes the whole network on synthesized low-quality/high-quality pairs.
- Deep features generate the GAN latent code for global structure, while shallow features generate noise for local details and background preservation.
- The authors report that GPEN establishes state-of-the-art blind face restoration and handles severely degraded real-world face images.
2. Related Work
Related work spans traditional and DNN-based face restoration, GAN-based image synthesis, GAN inversion, and GAN-prior transfer. GPEN differs by jointly fine-tuning a face-generation prior with a restoration DNN in the same domain.
- Face Image Restoration: Traditional face restoration uses degradation models and handcrafted priors, whereas DNN-based methods achieve better performance across face restoration tasks.
- Generative Adversarial Network (GAN): GANs are widely used for photo-realistic image generation in applications including inpainting, super-resolution, colorization, and texture synthesis.
- GAN Prior for Image Generation: GAN inversion reverses images into latent codes; optimization-based methods are slow, while pSp performs projection without additional optimization.
- GAN Prior Transfer: Prior GAN-transfer methods target different source and target domains, whereas GPEN embeds a face-generation GAN into a restoration DNN within the same domain.
- GAN Prior Transfer: GPEN jointly fine-tunes the GAN prior and DNN so degraded inputs generate latent codes and noise at different network layers.
3. Proposed Method
GPEN embeds a pretrained GAN prior into a U-shaped restoration network and fine-tunes the combined model for blind face restoration. Its design maps deep and shallow encoder features to GAN inputs that control global structure, local details, and background.
- Motivation: BFR is ill-posed because one degraded input can correspond to multiple high-quality faces, causing pixel-wise DNN solutions to become over-smoothed.Adversarial training only partially reduces this uncertainty when degradation is severe.
- Framework: GPEN first trains a GAN prior for high-quality face generation, embeds it as a decoder in a DNN, and fine-tunes the whole network.Fine-tuning uses synthesized low-quality/high-quality face pairs.
- Framework: The GAN generator converts a learned latent code into a high-quality face, making the generation stage essentially one-to-one and reducing prior mapping uncertainty.The encoder maps the degraded input to the GAN latent space before generation.
- GAN Prior Network: The GAN prior follows a U-shaped encoder-decoder design and uses StyleGAN v2 blocks for their high-quality image-generation capability.Other GAN architectures, including StyleGAN v1, PGGAN, and BigGAN, can also be adopted.
- GAN Prior Network: Deep encoder features generate the latent code, while shallow features generate noise inputs that control global structure, local details, and background.Noise inputs are reused at the same spatial resolution across GAN blocks and concatenated into convolutions.
- Training Strategy: GPEN fine-tuning combines adversarial, content, and discriminator-based feature-matching losses.The content loss preserves fine features and original color information; the feature-matching loss uses discriminator features, with α = 1 and β = 0.02.
4. Experiments
Experiments evaluate GPEN on synthetic and real-world degraded faces, including ablations, blind face restoration, and face super-resolution. GPEN generally improves perceptual quality, especially under severe or unknown degradation.
- Experimental setup: GPEN is trained using FFHQ faces, synthesized degraded inputs, and a jointly fine-tuned embedded GAN prior network.The degradation model combines blur, Gaussian noise, downsampling, and JPEG compression.
- Ablation study: The ablation variants test whether fine-tuning, noise inputs, and concatenating noise into convolutions contribute to GPEN performance.GPEN-w/o-ft freezes the embedded GAN, GPEN-w/o-noise removes noise inputs, and GPEN-noise-add adds rather than concatenates noise.
- Ablation study: GPEN achieves better PSNR, FID, and LPIPS measures than its ablation variants, while producing more faithful facial details and backgrounds.The frozen-prior variant changes facial appearance and background, the no-noise variant is blurrier, and additive noise introduces false background details.
- Synthetic BFR: On synthetic BFR images, GPEN has comparable PSNR to competing methods but significantly better FID and LPIPS results.The comparison includes Pix2PixHD, Super-FAN, GFRNet, GWAInet, DFDNet, and HiFaceGAN; DFDNet fails to recover many images in this experiment.
- Face super-resolution: On synthetic FSR, GPEN obtains the best FID and LPIPS scores under almost all zooming factors, although bilinear interpolation has the best PSNR.The zooming factor ranges from 8× to 256×, and mGANprior and PULSE are evaluated on only the first 1,000 CelebA-HQ images.
5. Conclusion and Discussion
GPEN embeds a pre-trained GAN into a U-shaped DNN and fine-tunes the whole network on artificially degraded face images. It restores high-quality faces from severe degradation, but currently produces only one HQ output per LQ input.
- Conclusion: GPEN embeds a pre-trained GAN as a decoder in a U-shaped DNN and fine-tunes the whole network with artificially degraded face images.The approach learns to generate high-quality face images from severely degraded inputs.
- Conclusion: Experiments on synthetic and real-world images show that GPEN significantly outperforms recent state-of-the-art methods while restoring facial details and retaining image backgrounds.
- Discussion: GPEN currently cannot generate multiple HQ images from one LQ image because style mixing may produce inconsistent backgrounds.The authors identify generating multiple outputs as future work, potentially using an extra HQ reference face.