Source-linked AI summary
Progressive Semantic-Aware Style Transformation for Blind Face Restoration
Chaofeng Chen, Xiaoming Li, Lingbo Yang, Xianhui Lin, Lei Zhang, Kwan-Yee K. Wong
TL;DR
Blind face restoration seeks plausible HQ images from severely degraded LQ inputs, where existing approaches often struggle on real-world images. PSFR-GAN uses progressive semantic-aware style transformation with multi-scale inputs, region-specific style loss, and an LQ parsing network. Experiments report realistic outputs and better generalization from synthetic training data to natural LQ faces than state-of-the-art methods.
Problem
Blind face restoration must recover HQ images from LQ faces with unknown degradations, while few existing methods generalize well to real LQ images.
Method
PSFR-GAN progressively modulates multi-scale features using LQ images and parsing maps, adds semantic-aware style loss, and uses a pretrained LQ face parsing network.
Results
Experiments show that synthetic-data-trained PSFR-GAN produces realistic high-resolution results and generalizes better to natural LQ faces than state-of-the-art methods.
Takeaways & Limitations
The framework provides a robust, easy-to-use face restoration solution that generates HQ outputs from LQ inputs without extra inputs.
Takeaways & Limitations
The degradation model assumes randomly selected blur, downsampling, Gaussian-noise, and JPEG-compression parameters during synthetic training.
Abstract
from arXiv · showhide
Face restoration is important in face image processing, and has been widely studied in recent years. However, previous works often fail to generate plausible high quality (HQ) results for real-world low quality (LQ) face images. In this paper, we propose a new progressive semantic-aware style transformation framework, named PSFR-GAN, for face restoration. Specifically, instead of using an encoder-decoder framework as previous methods, we formulate the restoration of LQ face images as a multi-scale progressive restoration procedure through semantic-aware style transformation. Given a pair of LQ face image and its corresponding parsing map, we first generate a multi-scale pyramid of the inputs, and then progressively modulate different scale features from coarse-to-fine in a semantic-aware style transfer way. Compared with previous networks, the proposed PSFR-GAN makes full use of the semantic (parsing maps) and pixel (LQ images) space information from different scales of input pairs. In addition, we further introduce a semantic aware style loss which calculates the feature style loss for each semantic region individually to improve the details of face textures. Finally, we pretrain a face parsing network which can generate decent parsing maps from real-world LQ face images. Experiment results show that our model trained with synthetic data can not only produce more realistic high-resolution results for synthetic LQ inputs and but also generalize better to natural LQ face images compared with state-of-the-art methods. Codes are available at https://github.com/chaofengc/PSFRGAN.
1. Introduction
PSFR-GAN addresses blind face restoration by progressively transforming multi-scale pixel and semantic inputs, adding region-specific texture supervision and an LQ parsing network for practical real-world use.
- Blind face restoration recovers HQ faces from LQ inputs with unknown degradations, but existing methods generalize poorly to natural LQ images.
- PSFR-GAN formulates restoration as a coarse-to-fine, multi-scale semantic-aware style transformation rather than an encoder-decoder mapping.It progressively modulates features using multi-scale LQ images and parsing maps.
- The framework combines LQ color information with parsing-map shape and semantic information to add details progressively across feature scales.
- Semantic-aware style loss computes Gram-matrix loss separately for semantic regions to improve texture restoration and reduce artifacts.
- A pretrained LQ face parsing network generates parsing maps before PSFR-GAN inference, allowing HQ outputs from only LQ inputs.
- Experiments report better generalization from synthetic training data to natural LQ faces than current state-of-the-art methods.
2. Related Works
Related work spans face super-resolution, blind restoration, and GAN-based HQ face generation, with StyleGAN-style modulation and parsing-map generation informing this framework.
- Face super-resolution methods use cascaded networks, GANs, and additional mechanisms to hallucinate or restore high-resolution facial details.
- StyleGAN generates HQ faces through latent-conditioned style parameters and adaptive instance normalization, while later models extend this paradigm to parsing maps.
3. Proposed Method
PSFR-GAN progressively upsamples learned features while applying spatially adaptive style transformations from multi-scale LQ images and parsing maps, with semantic-aware texture and standard restoration objectives.
- Progressive Semantic-Aware Style Transformation: The network starts from a learned C × 16 × 16 constant and progressively upsamples residual-block features to the HQ output resolution.The final feature is converted to the output image by a ToRGB convolution.
- Progressive Semantic-Aware Style Transformation: Each style-transformation block derives scale-specific parameters from resized LQ images and parsing maps corresponding to the current feature scale.
- Progressive Semantic-Aware Style Transformation: Spatially adaptive style parameters use LQ color and texture information together with parsing-map shape and semantic guidance.The parsing maps are generated from LQ inputs by the pretrained face parsing network.
- Semantic-Aware Style Loss: Semantic-aware style loss computes Gram-matrix differences separately for each semantic region using VGG19 features.The loss uses relu3_1, relu4_1, and relu5_1 features, with the background labeled M0.
- Model Objectives: Training additionally uses reconstruction loss combining pixel and feature-space MSE, plus multi-scale feature matching and adversarial objectives.
- Model Objectives: Spectral normalization is applied to generator and discriminators, and generator and discriminator losses are minimized alternately.
4. Implementation Details
The implementation uses synthetic degradation and curated datasets to train and evaluate PSFR-GAN across synthetic and real low-quality face inputs.
- Degradation Model: The degradation model applies blur, downsampling, additive white Gaussian noise, and JPEG compression to high-resolution images, with parameters randomly selected per image.
- Degradation Model: The degradation-generated low-quality images are upsampled for parsing-map prediction and face restoration.
- Training Data: Training uses FFHQ images resized to 512 × 512 as ground-truth high-resolution images, with low-quality inputs synthesized online.
- Testing Data: The synthetic test set contains 2,800 CelebAHQ images without identity overlap with FFHQ, degraded using the training procedure.
- Testing Data: The real test set combines 1,020 CelebA faces, 106 GFRNet-provided images, and old internet photos into 1,157 real low-quality faces.
5. Experiments
Experiments compare PSFR-GAN with restoration methods on synthetic and real low-quality face datasets using distortion, perceptual, and FID-based evaluation. The results report stronger quantitative and visual performance, including better real-world generalization and faster processing than PULSE.
- Experimental setup: Experiments evaluate PSFR-GAN on synthetic CelebAHQ-Test and real PSFR-RealTest datasets, alongside ablations of multi-scale parsing guidance and semantic-aware style loss.The evaluation includes both quantitative comparisons and visual analyses.
- Synthetic datasets: Visual comparisons show PSFR-GAN producing realistic results across light and severe degradation, large pose, and differing skin colors, while task-specific methods are oversmoothed.HiFaceGAN and DFDNet exhibit artifacts or reference-matching failures in challenging cases, whereas PSFR-GAN remains more realistic and robust.
- Evaluation metrics: CelebAHQ-Test uses PSNR, SSIM, MSSIM, LPIPS, and FID to assess pixel fidelity, perceptual realism, and statistical similarity to HQ faces.FID is also reported for reference on CelebAHQ-Test, while the reference HQ dataset is used to evaluate PSFR-RealTest.
- Synthetic datasets: On CelebAHQ-Test, PSFR-GAN outperforms DFDNet by a large margin on most metrics, including a 50% improvement in FID.Methods designed for specific restoration tasks generally obtain lower perceptual scores than blind face restoration methods.
- Real-world images: On PSFR-RealTest, PSFR-GAN's FID score is 20% higher than DFDNet's second-best result, with clearer outputs and fewer conspicuous artifacts.The reported visual examples include improved teeth and eye textures, supported by multi-scale parsing guidance.
- Comparison with PULSE: PSFR-GAN produces better face-restoration results than PULSE in the visual comparison and processes each GPU image in 0.1s versus PULSE's 4s.PULSE is carefully fine-tuned separately for each input, so quantitative comparison on PSFR-RealTest is treated as unfair.
6. Discussions and Ablation Study
PSFR-GAN restores faces progressively from coarse semantic structure to fine color and texture, while parsing maps and semantic-aware style loss improve structure and reduce artifacts. It remains robust across severe upscale factors.
- Analysis of PSFR-GAN: PSFR-GAN adds high-level semantics, mid-level shape and edges, then low-level color and details during coarse-to-fine restoration.Multi-scale parsing inputs contribute semantic and shape information, while LQ images contribute color and texture information.
- Analysis of PSFR-GAN: Parsing maps clarify semantic boundaries and progressively remove artifacts, while LQ inputs add color and texture details.Without parsing maps, nose and mouth borders become unclear and cheek artifacts appear; adding LQ inputs restores color and texture.
- Robustness to degradation: PSFR-GAN works well for upscale factors ≤12 and produces reasonable results for ×16 while other degradation parameters remain fixed.This experiment demonstrates robustness to different upscale factors.
- Ablation Study: The ablation study finds that parsing maps provide the largest improvement, while semantic-aware style loss also benefits restoration.Parsing maps regularize face structure, whereas the style loss synthesizes realistic textures for each semantic region.
- Ablation Study: Models without parsing maps fail to generate clear face shapes when the LQ face border is unclear, while models without style loss produce eye artifacts.The full model with both components avoids these observed flaws.
7. Conclusion
The paper concludes that PSFR-GAN performs progressive semantic-aware face restoration and uses a pretrained parsing network to support high-resolution outputs from LQ inputs. The parsing network is trained with joint parsing and pixel supervision.
- Conclusion: PSFR-GAN restores LQ faces from coarse to fine through multi-scale semantic-aware style transformation.The framework uses multi-scale LQ images and parsing maps as inputs.
- Conclusion: A semantic-aware style loss based on the original Gram matrix loss is introduced for face restoration.
- Conclusion: Experiments on synthetic and real LQ test datasets demonstrate the superiority and robustness of PSFR-GAN.
- Conclusion: The pretrained FPN produces parsing maps and HR face estimates using parsing and pixel-space L2 losses.Its architecture is an encoder-ResNet-decoder with four downsampling blocks, ten ResNet blocks, and four upsampling blocks.
- Conclusion: Multi-task learning improves FPN parsing results, especially for eyes and eyebrows, by adding HR-face supervision.
A.2. Datasets and Implementation
The implementation trains FPN on manually annotated CelebA-Mask-HQ faces and synthesizes degraded inputs using blur, downsampling, AWGN, and JPEG compression. These details define the training data and degradation setup.
- Datasets: CelebA-Mask-HQ provides 30,000 1024 × 1024 HR faces with manually annotated 512 × 512 masks covering 19 facial classes.
- Implementation: FPN training uses Adam with β1 = 0.9, β2 = 0.999, learning rate 0.0002, and batch size 8.
- Degradation Model: The degradation model applies blur, downsampling, additive white Gaussian noise, and JPEG compression to generate LQ inputs.
- Degradation Model: Blur kernels include Gaussian, average, median, and motion blur with randomly selected kernel-size ranges.
- Degradation Model: AWGN uses 0 <= δ <= 0.1 × 255, while JPEG compression quality is randomly selected from [10, 65].Higher JPEG compression values in the specified operation correspond to stronger compression and lower image quality.
- Implementation: The degradation pipeline is implemented with the imgaug library.
C. More Results
The paper reports additional results on PSFR-RealTest and the Solvay conference test, primarily comparing PSFR-GAN with DFDNet and also providing finetuned PULSE results.
- More Results: Additional experiments evaluate PSFR-GAN on PSFR-RealTest and Solvay conference test datasets.
- More Results: The main comparison uses DFDNet because public code and test models are available and its results are described as current state-of-the-art.
- More Results: The paper also reports carefully finetuned PULSE results on the Solvay test.
C.1. Results of PSFR-RealTest
Figures 12–14 provide additional examples from the PSFR-RealTest dataset.
- Figures 12–14 show more PSFR-RealTest examples.
C.2. Results of Solvay Conference Test
The Solvay Conference test enhances aligned, cropped faces with PSFR-GAN and pastes them back into the original photograph. Figures 16–20 provide complete results and detailed comparisons with other methods.
- Faces are cropped and aligned, enhanced by PSFR-GAN, and pasted back into the original photograph.
- Figures 16–20 show complete results and detailed comparisons with other methods for the Solvay Conference test.
- Additional figures document parsing results, degradation-model code, and further PSFR-RealTest Dataset results.Figure 9 shows the face parsing network architecture; Figure 10 compares parsing with and without IH supervision.
- Figure 15 presents the overall result of the fifth Solvay conference test.The caption directs readers to zoom in for details.