Source-linked AI summary
FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors
Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, Jian Yang
TL;DR
Very low-resolution face super-resolution needs methods that exploit facial geometry without requiring well-aligned inputs and metrics that better reflect visual quality. FSRNet estimates landmark heatmaps and parsing maps within an end-to-end coarse-to-fine network, while FSRGAN adds adversarial loss; experiments report superior results, with FSRNet handling unaligned 16×16 faces at 8× upscaling and FSRGAN producing more realistic images.
Problem
Very low-resolution face super-resolution requires exploiting facial priors without well-aligned inputs, while classic metrics can be inconsistent with visual perception.
Method
FSRNet uses a coarse-to-fine end-to-end network that estimates landmark heatmaps and parsing maps, and FSRGAN adds adversarial loss for realistic faces.
Results
FSRNet achieves state-of-the-art performance on unaligned 16×16 faces with 8× upscaling, while FSRGAN generates more realistic face images.
Takeaways & Limitations
Face alignment and parsing complement PSNR and SSIM as evaluation metrics for face super-resolution, covering geometry and perceptual quality more directly.
Abstract
from arXiv · showhide
Face Super-Resolution (SR) is a domain-specific super-resolution problem. The specific facial prior knowledge could be leveraged for better super-resolving face images. We present a novel deep end-to-end trainable Face Super-Resolution Network (FSRNet), which makes full use of the geometry prior, i.e., facial landmark heatmaps and parsing maps, to super-resolve very low-resolution (LR) face images without well-aligned requirement. Specifically, we first construct a coarse SR network to recover a coarse high-resolution (HR) image. Then, the coarse HR image is sent to two branches: a fine SR encoder and a prior information estimation network, which extracts the image features, and estimates landmark heatmaps/parsing maps respectively. Both image features and prior information are sent to a fine SR decoder to recover the HR image. To further generate realistic faces, we propose the Face Super-Resolution Generative Adversarial Network (FSRGAN) to incorporate the adversarial loss into FSRNet. Moreover, we introduce two related tasks, face alignment and parsing, as the new evaluation metrics for face SR, which address the inconsistency of classic metrics w.r.t. visual perception. Extensive benchmark experiments show that FSRNet and FSRGAN significantly outperforms state of the arts for very LR face SR, both quantitatively and qualitatively. Code will be made available upon publication.
1. Introduction
Face super-resolution targets high-resolution faces from very low-resolution inputs, where facial priors can improve recovery but prior methods used inconvenient multi-stage training. FSRNet introduces end-to-end geometry-prior learning, while FSRGAN adds adversarial training and face alignment/parsing provide complementary evaluation metrics.
- Motivation: Face super-resolution generates high-resolution face images from low-resolution inputs and supports downstream alignment, parsing, and recognition tasks.Existing face-analysis techniques degrade substantially on very low-resolution faces.
- Motivation: Facial correspondence and component information provide useful shape and detail priors, but prior prior-based methods relied on complicated multi-stage training.
- Proposed approach: FSRNet jointly estimates landmark heatmaps and parsing maps during end-to-end training to improve super-resolution of very low-resolution, unaligned faces.It first recovers a coarse high-resolution image, then combines image features and estimated priors in a fine decoder.
- Proposed approach: FSRGAN incorporates adversarial loss into FSRNet to produce more realistic high-resolution face textures.
- Evaluation: Face alignment and parsing are introduced as evaluation metrics intended to address inconsistencies between classic metrics and visual perception.
- Contributions: FSRNet achieves state-of-the-art performance on unaligned 16×16 face images upscaled by 8, while FSRGAN generates more realistic faces.
2. Related Work
Related work includes facial-prior methods and end-to-end face hallucination approaches. FSRNet differs by combining facial geometry priors with end-to-end training.
- Facial Prior Knowledge: Early facial-prior methods modeled gradient distributions, eigen transformations, or nonlinear latent structures, often under controlled face settings.
- Facial Prior Knowledge: Deep prior-based methods alternated face hallucination with correspondence estimation or used separate component generation and enhancement stages.
- FSRNet’s distinction: FSRNet is presented as fully leveraging facial priors within an end-to-end framework, unlike prior multi-step prior-based approaches.
- End-to-End Face SR: The related-work discussion identifies end-to-end face SR methods that use GANs, transformative discriminative autoencoders, or attention-aware enhancement.
3. Face Super-Resolution Network
FSRNet combines coarse and fine super-resolution with jointly estimated facial landmark and parsing priors. FSRGAN extends it with adversarial and perceptual objectives to generate more realistic high-resolution faces.
- Overview: FSRNet comprises coarse SR, fine SR encoder, prior estimation, and fine SR decoder components.The coarse and fine networks form the basic FSRNet.
- Fine SR Network: The decoder concatenates image features with estimated prior information to recover the final HR face.The upper-bound model instead directly concatenates ground-truth priors with image features; removing priors yields the baseline configuration.
- Training Objective: FSRNet training minimizes reconstruction losses for coarse and final HR outputs together with a weighted prior-estimation loss.The training set provides LR images, ground-truth HR images, and corresponding ground-truth prior information.
- Coarse SR Network: A coarse SR network first produces a coarse HR image because facial priors are difficult to estimate directly from very LR inputs.The coarse image is then supplied to both the prior estimation and fine encoder branches.
- Prior Estimation Network: The prior estimation network jointly predicts facial landmark heatmaps and parsing maps with shared features, while the fine encoder extracts image features.An HourGlass structure supports multi-task prior estimation, with task-specific final layers.
- FSRGAN: FSRGAN adds conditional adversarial training and a VGG-16 feature-space perceptual loss to FSRNet for realistic HR face generation.The discriminator distinguishes generated from real HR faces, while the perceptual term uses high-level ‘relu5 3’ features.
4. Prior Knowledge for Face Super-Resolution
The section tests whether facial priors improve face super-resolution and compares landmark and parsing prior designs. Results show that richer, especially combined, priors improve recovery, while imperfect estimated priors remain below ground-truth-prior performance.
- The experiments use 2,330 Helen images, with 194 landmark labels and 11 parsing maps per image.
- Ground-truth landmark heatmaps and parsing maps provide upper-bound comparisons for measuring the value of facial prior knowledge.
- Global parsing maps improve face super-resolution more than local parsing maps, while additional landmark heatmaps yield only minor gains beyond 49 landmarks.
- Using both facial priors produces the strongest improvement, supporting the value of combining complementary facial geometry information.
5. Experiments
Experiments evaluate FSRNet and FSRGAN on Helen and CelebA using reconstruction, alignment, parsing, visual, and robustness comparisons. The results show improved geometry and visual quality over competing methods, with estimated-prior performance limited by prior prediction accuracy.
- Ablation Study: 25.85 dB is achieved with estimated priors, compared with 26.55 dB using ground-truth priors, indicating room for better prior estimation.
- Ablation Study: The best PSNR rises from 25.69 to 25.95 dB as the prior-estimation network grows from 1 to 4 hourglasses.
- Comparisons with State-of-the-Art Methods: FSRNet produces sharper edges and shapes than competing methods, while FSRGAN recovers sharper facial textures than FSRNet.
- Comparisons with State-of-the-Art Methods: FSRGAN achieves relatively sharper facial shapes than Wavelet-SRNet and CBN, including sharper noses in all compared cases.
- Comparisons with State-of-the-Art Methods: Face alignment and parsing provide geometry-focused metrics, and FSRGAN outperforms FSRNet on both consistently with superior visual quality.
6. Conclusions
The paper concludes that FSRNet uses facial priors in an end-to-end network to improve both photometric recovery and geometry estimation for unaligned, very low-resolution faces. FSRGAN further targets realistic visual results, while future work includes better prior estimation and additional priors.
- FSRNet’s prior estimation network improves PSNR/SSIM recovery and provides facial landmark and parsing estimates directly from very low-resolution inputs.
- FSRNet achieves superior quantitative and qualitative performance on unaligned face images compared with state-of-the-art methods.
- Future directions include iterative fine-SR learning, improved prior estimation, and additional facial priors such as texture.
7. Appendix
The appendix documents FSRGAN’s patch-based discriminator, additional comparisons with CBN and other methods, and robustness tests across facial variations. These examples report sharper recovery and robustness under misalignment, pose, expression, occlusion, and scale changes.
- Discriminator: The Patch-GAN discriminator downsamples 128×128 inputs to 8×8 feature maps, classifying each corresponding 16×16 image patch as real or fake.
- Qualitative Comparisons: Across PubFig83 comparisons, the appendix evaluates models trained on CelebA at 4× with additional unknown Gaussian blur.
- Qualitative Comparisons: On CBN’s three representative failure cases, FSRNet and FSRGAN recover sharper, more accurate results, including correct gaze direction in the second case.
- Robustness to Facial Variations: Examples test robustness to misalignment, pose, expression, and occlusion at 4× and 8× scale factors.
- Robustness to Facial Variations: At 4× scaling, the models recover results described as extremely similar to the ground-truth HR images.