Source-linked AI summary
Progressive Face Super-Resolution via Attention to Facial Landmark
Deokyun Kim, Minseon Kim, Gihyun Kwon, Dae-Shik Kim
TL;DR
Face SR must reconstruct facial details from low-resolution images without distortion, while prior attribute information can be difficult to obtain. The paper combines progressive training, facial attention loss, and a distilled FAN to generate photo-realistic 8× face SR images with facial details. The reported experiments show improvements in perceptual and quantitative quality, while the method remains dependent on obtaining suitable landmark heatmaps.
Problem
Face SR seeks to reconstruct high-resolution faces from low-resolution inputs while preserving facial details without distortion, but useful facial attribute information is difficult to obtain in the wild.
Method
The method progressively trains generator and discriminator stages, applies facial attention loss using landmark heatmaps, and uses a distilled FAN to obtain SR-oriented heatmaps.
Results
The method generates photo-realistic 8× face images with retained facial details and reportedly outperforms prior methods qualitatively and quantitatively, especially in perceptual quality.
Takeaways & Limitations
Facial attention and distilled alignment heatmaps provide a way to focus face SR restoration on landmark-adjacent facial details while reducing overall training time.
Takeaways & Limitations
Performance could improve with better methods for obtaining heatmaps that more accurately represent facial landmark areas.
Abstract
from arXiv · showhide
Face Super-Resolution (SR) is a subfield of the SR domain that specifically targets the reconstruction of face images. The main challenge of face SR is to restore essential facial features without distortion. We propose a novel face SR method that generates photo-realistic 8x super-resolved face images with fully retained facial details. To that end, we adopt a progressive training method, which allows stable training by splitting the network into successive steps, each producing output with a progressively higher resolution. We also propose a novel facial attention loss and apply it at each step to focus on restoring facial attributes in greater details by multiplying the pixel difference and heatmap values. Lastly, we propose a compressed version of the state-of-the-art face alignment network (FAN) for landmark heatmap extraction. With the proposed FAN, we can extract the heatmaps suitable for face SR and also reduce the overall training time. Experimental results verify that our method outperforms state-of-the-art methods in both qualitative and quantitative measurements, especially in perceptual quality.
1 Introduction
The paper proposes a progressively trained face SR method that constrains landmark regions with facial attention loss and distilled FAN heatmaps. It targets photo-realistic 8× upscaling while preserving facial details and reducing training time.
- Motivation: Face SR reconstructs high-resolution faces from low-resolution inputs while restoring facial details, but enlargement can distort faces and obscure facial attributes.Prior methods use attribute vectors or auxiliary networks, but attribute information is difficult to obtain in the wild and landmark-focused methods may neglect surrounding facial attributes.
- Method: Progressive training grows the generator and discriminator through successive steps to stably produce photo-realistic 8× upscaled face images.The method constrains each step and applies facial attention loss at intermediate and final stages.
- Method: Facial attention loss focuses restoration on areas adjacent to facial landmarks by weighting pixel differences with landmark heatmaps.The loss is intended to restore facial attributes around landmarks rather than only localize landmark points.
- Method: A distilled FAN produces SR-oriented landmark heatmaps with comparable performance to the original FAN while reducing overall training time.The compressed network is trained using a hint-based method and is designed to avoid inappropriate attention in occluded landmark areas.
- Evaluation: Experiments evaluate aligned and unaligned CelebA and AFLW faces using PSNR, SSIM, and MS-SSIM, alongside ablations and comparisons with prior studies.The paper reports that the proposed loss benefits super-resolution and that the method is superior to previous studies in the stated evaluations.
2 Related work
Prior face SR methods use facial attributes, geometry, or landmark heatmaps to preserve information during upscaling. Their limitations include difficult-to-obtain attribute vectors and incomplete reflection of facial attributes around landmarks.
- Facial information priors: Facial attributes and spatial configuration of facial components are used as priors to reduce ambiguity and preserve structure in face SR.These approaches include attribute vectors, spatial transformers, and image–attribute domain fusion.
- Facial information priors: Attribute-vector methods can preserve indicated facial attributes, but the vectors are difficult to acquire in the wild and describe only partial attributes.This limits their practicality for unconstrained face SR settings.
- Landmark and geometry methods: Geometry-based methods estimate facial shape or component positions, while heatmap loss localizes landmarks for 4× face upscaling.These methods preserve facial-component spatial configuration but are described as failing to fully reflect accurate facial attributes.
3 Approach
The approach combines progressive face SR training, facial attention loss, and a distilled FAN to produce higher-fidelity face images while stabilizing multi-stage upscaling.
- Progressive training: The generator and discriminator are trained progressively across 2×, 4×, and 8× upscaling steps.Each step combines the previous output, interpolated to the next resolution, with the current step's output using a linearly increasing weight α.
- Facial attention loss: Facial attention loss weights pixel differences using landmark heatmaps to restore details around facial landmarks.The heatmaps are normalized and channel-wise max-pooled across landmarks before weighting the reconstruction error.
- Distilled FAN: The distilled FAN is designed to provide broader landmark-area attention than the original FAN, which concentrates heatmap values near landmark points.The paper reports that original-FAN attention can create artifacts by focusing only on distinct landmark points.
- Distilled FAN: The distilled FAN uses single-scale feature maps and hint-based training to produce face-SR-oriented heatmaps without the original FAN's encoder-decoder and skip-layer design.It minimizes the squared feature difference between distilled and original FAN outputs.
- Overall training loss: The training objective combines pixel, perceptual, WGAN, heatmap, and attention losses at Steps 2 and 3.Landmark-based losses are omitted from Step 1, while α, β, γ, λ, and η weight the overall terms.
4 Experiments
Experiments evaluate the distilled FAN, facial attention loss, progressive training, and comparisons with existing face SR methods on aligned and unaligned datasets. The proposed components improve landmark heatmap suitability, standard image metrics, facial detail preservation, and perceptual quality.
- 4.2 Distilled FAN Results: The distilled FAN achieves comparable landmark prediction performance to the original FAN with substantially fewer parameters.Its heatmaps also assign relatively low values to occluded landmark areas, making them suitable as facial attention weights.
- 4.2 Distilled FAN Results: Using the distilled FAN as attention weights avoids artifacts associated with concentrating attention only on distinct landmark points.The distilled FAN improves SR performance while using a smaller network.
- 4.3 Ablation Study: Facial attention loss produces the highest PSNR, SSIM, and MS-SSIM values among the ablated loss configurations.The results support its usefulness for generating structurally meaningful face images.
- 4.3 Ablation Study: Progressive training improves measurement values and preserves more facial detail than training the network without progressive training.The non-progressive model shows degradation of facial details in its super-resolved images.
- 4.4 Comparison with State-of-the-Art: Compared with VDSR, FSRNet, FSRGAN, and URDGN, the proposed method achieves especially strong SSIM and MS-SSIM results while restoring accurate facial attributes.VDSR obtains the highest PSNR but produces significantly blurred images; the other baselines show artifacts, blur, or facial distortion.
- 4.4 Comparison with State-of-the-Art: The MOS test indicates that the proposed results are perceptually superior to the compared super-resolved outputs.The test used ratings from 26 raters across aligned and unaligned datasets.
5 Conclusion
The method combines progressive training, facial attention loss, and a distilled FAN to produce face super-resolution images with more precise facial details. Its performance depends on obtaining heatmaps that accurately represent facial landmark areas.
- Progressive training generates photo-realistic face images while learning facial-detail restoration with different guidance at each step.
- Facial attention loss weights facial features near landmarks more strongly, improving their expression in super-resolved images.
- A distilled FAN extracts heatmaps suitable for face SR and reduces overall training time from ∼3 days to ∼1 day.
- The experiments show more accurate facial details and perceptually high-quality face images similar to real images.
- Better heatmap extraction could further improve performance because the method focuses restoration on specific facial areas.