Source-linked AI summary
Super-FAN: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs
Adrian Bulat, Georgios Tzimiropoulos
TL;DR
Very low-resolution faces make both visual enhancement and facial-landmark localization difficult, especially across arbitrary poses. Super-FAN jointly solves these tasks by combining GAN-based super-resolution with FAN heatmap supervision and a new residual architecture. The paper reports large improvements for both tasks across facial poses, plus good qualitative results on real-world low-resolution images, while failures remain mainly in extreme poses, large occlusions, and heavy blur.
Problem
Very low-resolution faces require both improved image quality and accurate facial-landmark localization, but jointly performing these tasks across arbitrary poses remains open.
Method
Super-FAN integrates a GAN-based super-resolution network, a face-alignment sub-network with heatmap regression, and a novel heatmap loss for joint optimization.
Results
Super-FAN shows large improvement over the state of the art for both face super-resolution and alignment across the whole spectrum of facial poses, with good real-world low-resolution results.
Takeaways & Limitations
Joint training supports integrated enhancement and alignment across synthetic and real-world low-resolution faces, including arbitrary poses.
Takeaways & Limitations
The method has failure cases involving extreme poses, large occlusions, and heavy blurring, and its blur-augmentation strategy is acknowledged as non-optimal.
Abstract
from arXiv · showhide
This paper addresses 2 challenging tasks: improving the quality of low resolution facial images and accurately locating the facial landmarks on such poor resolution images. To this end, we make the following 5 contributions: (a) we propose Super-FAN: the very first end-to-end system that addresses both tasks simultaneously, i.e. both improves face resolution and detects the facial landmarks. The novelty or Super-FAN lies in incorporating structural information in a GAN-based super-resolution algorithm via integrating a sub-network for face alignment through heatmap regression and optimizing a novel heatmap loss. (b) We illustrate the benefit of training the two networks jointly by reporting good results not only on frontal images (as in prior work) but on the whole spectrum of facial poses, and not only on synthetic low resolution images (as in prior work) but also on real-world images. (c) We improve upon the state-of-the-art in face super-resolution by proposing a new residual-based architecture. (d) Quantitatively, we show large improvement over the state-of-the-art for both face super-resolution and alignment. (e) Qualitatively, we show for the first time good results on real-world low resolution images.
1. Introduction
Super-FAN targets the coupled problems of enhancing very low-resolution facial images and localizing their semantic landmarks. It jointly addresses the chicken-and-egg difficulty of alignment and super-resolution across arbitrary poses.
- Motivation: Very low-resolution faces challenge both visual-quality improvement and localization of semantic landmarks such as noses and eye corners.The paper frames these as face super-resolution and face alignment, respectively.
- Motivation: Landmark detection can benefit super-resolution, while effective super-resolution could improve landmark localization, creating a coupled open problem for arbitrary poses.Prior methods can produce blurry images with artifacts when landmarks are poorly localized.
- Contributions: Super-FAN integrates landmark localization through heatmap regression into a GAN-based super-resolution network and introduces a heatmap loss.This is presented as the first end-to-end system addressing both tasks simultaneously.
- Contributions: The system jointly trains alignment and super-resolution on synthetic and real-world low-resolution faces spanning arbitrary facial poses.The paper reports quantitative improvements for both tasks and qualitative results on real-world images.
2. Closely related work
Prior work largely emphasized generic or face-specific super-resolution and landmark methods on frontal, aligned, or synthetic data. Super-FAN extends evaluation and modeling to arbitrary poses and real-world low-resolution faces.
- Prior limitations: Generic super-resolution methods often target already-good-resolution images and do not incorporate face-specific information, unlike face super-resolution for 16 × 16 or 32 × 32 inputs.The paper distinguishes these settings from its very low-resolution facial-image focus.
- Positioning: The paper proposes an improved GAN-based architecture as the strong baseline for its integrated face super-resolution and alignment network.Its architecture is compared directly with the prior design in Figure 3.
- Prior limitations: Earlier face super-resolution studies primarily evaluated frontal, pre-aligned, or synthetically generated images, including CelebA-based settings.The cited methods include GAN, decoder-encoder-decoder, and spatial-transformer approaches.
- Closest prior work: The closest prior method alternated super-resolution with dense correspondence using a separately learned, fixed cascaded-regression model.That separation leaves landmark localization vulnerable when the low-resolution input is difficult.
- Evaluation scope: The evaluation uses balanced pose coverage in LS3D-W and qualitative assessment on more than 200 real-world low-resolution faces from WiderFace.This goes beyond the predominantly frontal-dataset evaluation of prior work.
3. Datasets
The dataset design combines multiple sources for training and reserves a balanced LS3D-W test set across three pose ranges. WiderFace supplies heavily blurred, real-world faces for qualitative evaluation.
- Dataset construction: Training data combines 300W-LP, AFLW, Celeb-A, and part of balanced LS3D-W, while testing uses the remaining LS3D-W images.The LS3D-W test split represents each pose range equally.
- Training datasets: 300W-LP contributes synthetically rendered large-pose faces but contains only about 3,000 unique faces among 61,225 images and warping artifacts.The entire dataset was included in training.
- Training datasets: AFLW contributes 25,993 faces collected from Flickr with varied poses and expressions.All faces were included in training.
- Training datasets: A randomly selected subset of approximately 20,000 Celeb-A faces was used to reduce frontal-pose bias.Celeb-A contains mostly occlusion-free frontal or near-frontal images.
- Evaluation dataset: Balanced LS3D-W contains 7,200 in-the-wild images distributed equally across three pose ranges, with 4,200 for training and 3,000 for testing.Each pose range contributes 2,400 images overall.
- Real-world evaluation: WiderFace provides 200 randomly selected very low-resolution, heavily blurred faces for qualitative real-world assessment.The source dataset varies substantially in pose, occlusion, and quality.
4. Method
Super-FAN connects super-resolution, adversarial discrimination, and face alignment, using structural heatmap supervision to improve reconstructed facial images. Its residual super-resolution design redistributes processing across resolutions and combines pixel and feature losses.
- Architecture: Super-FAN comprises a super-resolution network, a discriminator distinguishing super-resolved from original high-resolution images, and FAN for landmark localization.The discriminator is omitted at test time.
- Super-resolution network: The proposed residual architecture uses a 12-3-2 block distribution across 16 × 16, 32 × 32, and 64 × 64 resolutions instead of the prior 16-1-1 design.More blocks at higher resolutions are intended to support sharper detail generation.
- Losses: Pixel-wise MSE compares the 64 × 64 super-resolved output with the corresponding high-resolution image, while perceptual loss compares feature representations.The upsampling factor is 4, and ResNet-50 features are used at multiple levels.
- Architecture comparison: Figure 3 contrasts the proposed super-resolution architecture with the architecture described in [20].The detailed architectural differences are discussed in the super-resolution subsection.
4.2. Adversarial network
The adversarial network uses WGAN to encourage realistic super-resolved faces, while stabilizing training and enabling earlier introduction of the GAN loss.
- The generator is the super-resolution network, and the discriminator distinguishes super-resolved images from original high-resolution faces.
- Super-FAN replaces the prior GAN formulation with WGAN to improve training stability and ease.
- WGAN stabilization allows the GAN loss to be introduced earlier during training, reducing overall training time.
- The WGAN loss samples generated faces from Pg, real high-resolution faces from Pr, and interpolation points along straight lines between them.
4.3. Face Alignment Network
The Face Alignment Network adds facial structural information to super-resolution by matching landmark heatmaps between super-resolved and high-resolution images, without requiring landmark annotations.
- Using pixel, perceptual, and adversarial losses alone can omit pose or expression details and misplace facial parts.
- The method integrates facial landmark localization through heatmap regression to enforce structural consistency between low- and high-resolution faces.
- A FAN processes the super-resolved image and is trained to match the heatmaps produced by another FAN on the original high-resolution image.
- Each landmark is represented by a heatmap containing a 2D Gaussian centered at its location, capturing shape, spatial context, and structural relationships.
- The heatmap loss requires only a pre-trained FAN, enabling weakly supervised training when landmark annotations are unavailable.
4.4. Overall training loss
Super-FAN is trained with an overall loss that combines its component objectives.
- The overall training loss used for Super-FAN combines pixel, feature, heatmap, and WGAN loss terms.
4.5. Training
Training uses 16 × 16 pixel inputs and 64 × 64 pixel outputs with augmentation, staged optimization, and a final end-to-end training phase; visual results compare successive model variants.
- Faces were cropped to 50 px height, with 16 × 16 px inputs and 64 × 64 px outputs.
- The visual comparison shows better structure and realism after adding heatmap loss, with Super-FAN additionally incorporating GAN loss and end-to-end training.
- Super-FAN jointly trained the super-resolution network, discriminator, and FAN for five epochs at a learning rate of 2.5e-4.
5. Experiments
Experiments evaluate Super-FAN for super-resolution and landmark localization across facial poses, using quantitative benchmarks and real-world WiderFace images. Results show benefits from joint training and strong qualitative performance, alongside failures under extreme conditions.
- Super-resolution variants: The tested super-resolution variants progressively add pixel, perceptual, heatmap, GAN, and end-to-end joint-training losses.Ours-Super-FAN extends Ours-pixel-feature-heatmap with GAN loss and end-to-end training.
- Experimental setup: Super-FAN is evaluated against SR-GAN, SR-ResNet, bilinear interpolation, and CBN using LS3D-W and real-world WiderFace faces.The evaluation includes both super-resolution and landmark localization, with quantitative and qualitative comparisons.
- Super-resolution results: PSNR and SSIM do not reliably identify visual quality: Ours-pixel-feature-heatmap leads PSNR, Ours-pixel leads SSIM, while sharper images come from Ours-pixel-feature-heatmap and Ours-Super-FAN.The authors note that Ours-pixel has top SSIM despite producing blurry and unrealistic images.
- Facial landmark localization results: Super-FAN is the only landmark-localization method approaching the FAN-HR upper bound and largely outperforms the non-joint equivalent.The results support the importance of jointly training face alignment and super-resolution networks.
- Real-world images: On real-world low-resolution faces, augmentation with blur, JPEG artifacts, and color distortion alleviates initially sub-optimal sharpness, while Super-FAN produces sharper, more detailed results across poses.The comparison uses more than 200 WiderFace images and includes SR-GAN and CBN.
- Limitations: Failures mainly involve extreme poses, large occlusions, and heavy blurring, with the authors describing the blur augmentation strategy as effective but far from optimal.Improving the augmentation strategy is left for future work.
6. Conclusions
Super-FAN integrates facial super-resolution and landmark localization in an end-to-end system, incorporating facial structure through alignment and a novel heatmap loss. It reports improvements across facial poses and good results on real-world low-resolution images.
- The architecture incorporates facial structural information by integrating a face-alignment sub-network and optimizing a novel heatmap loss.
- The method shows large improvements over the state of the art for both super-resolution and alignment across the whole spectrum of facial poses.
- Super-FAN produces good results on real-world low-resolution facial images.
A1. Ablation studies
The ablations examine loss functions, skip connections, speed, and qualitative behavior across real-world and varied-pose datasets. Results show limited benefit from pixel L1 loss and long-skip connections, while the system handles many challenging inputs with identifiable failure cases.
- Loss ablations: Replacing the generator's L2 pixel loss with L1 produced no improvement.The comparison uses PSNR and SSIM.
- Loss ablations: Replacing the L2 heatmap loss with L1 showed a decent improvement for large poses.The comparison reports AUC across pose on the LS3D-W balanced test set.
- Architecture ablations: Adding two long skip connections produced no improvement over the no-skip version.The comparison uses PSNR and SSIM.
- Speed: 10% slower than SR-GAN, Super-FAN processed 1,000 images in 4.6s versus 4.3s on an NVIDIA Titan-X GPU.
- Additional qualitative results: On 200 randomly selected low-resolution WiderFace images, Super-FAN generally handled pose variation and challenging illumination conditions, with a few failure cases.In most failures, it was impossible to tell whether the input was actually a face.
- Additional qualitative results: On LS3D-W Balanced images, Super-FAN produced accurate landmark-fitting results for arbitrary poses and varied facial expressions.The predictions were plotted over the low-resolution input images.
- Additional qualitative results: The system was also tested qualitatively on SCface, which contains 4,160 images of 130 subjects captured at different distances and with different cameras.