PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Spatial Attention for Face Super-Resolution

Chaofeng Chen, Dihong Gong, Hao Wang, Zhifeng Li, Kwan-Yee K. Wong

arXiv:2012.01211v2cs.CV

TL;DR

Face super-resolution struggles to recover sparse, detailed face structures, while prior auxiliary-task methods require extra labels and often produce limited resolutions. The paper introduces SPARNet's spatially attentive FAUs and extends it to SPARNetHD for 512×512 outputs. Experiments report stronger multi-metric performance and good generalization from synthetic training to real low-quality faces.

  • Problem

    Face super-resolution must recover difficult sparse structures, but auxiliary parsing or landmark supervision requires extra labels and existing methods often target only relatively low resolutions.

  • Method

    SPARNet stacks Face Attention Units that use differentiable spatial attention to emphasize key face structures, while SPARNetHD adds 512×512 output generation with multi-scale discriminators.

  • Results

    SPARNet outperforms generic and face super-resolution methods on PSNR and SSIM, achieves the best landmark-detection performance, and SPARNetHD produces realistic 512×512 outputs that generalize to natural low-resolution faces.

  • Takeaways & Limitations

    Spatial attention can focus face super-resolution on key structures without extra supervision, and the extended model supports high-resolution outputs with reported real-world generalization.

Abstract

from arXiv · show

General image super-resolution techniques have difficulties in recovering detailed face structures when applying to low resolution face images. Recent deep learning based methods tailored for face images have achieved improved performance by jointly trained with additional task such as face parsing and landmark prediction. However, multi-task learning requires extra manually labeled data. Besides, most of the existing works can only generate relatively low resolution face images (e.g., $128\times128$), and their applications are therefore limited. In this paper, we introduce a novel SPatial Attention Residual Network (SPARNet) built on our newly proposed Face Attention Units (FAUs) for face super-resolution. Specifically, we introduce a spatial attention mechanism to the vanilla residual blocks. This enables the convolutional layers to adaptively bootstrap features related to the key face structures and pay less attention to those less feature-rich regions. This makes the training more effective and efficient as the key face structures only account for a very small portion of the face image. Visualization of the attention maps shows that our spatial attention network can capture the key face structures well even for very low resolution faces (e.g., $16\times16$). Quantitative comparisons on various kinds of metrics (including PSNR, SSIM, identity similarity, and landmark detection) demonstrate the superiority of our method over current state-of-the-arts. We further extend SPARNet with multi-scale discriminators, named as SPARNetHD, to produce high resolution results (i.e., $512\times512$). We show that SPARNetHD trained with synthetic data cannot only produce high quality and high resolution outputs for synthetically degraded face images, but also show good generalization ability to real world low quality face images.

I. INTRODUCTION

Face super-resolution must recover sparse, difficult face structures without relying on extra labeled supervision. SPARNet addresses this with spatially attentive FAUs and extends the approach to 512×512 outputs through SPARNetHD.

  • Motivation: Face super-resolution generates high-resolution face images from low-resolution inputs, supporting face analysis on images such as surveillance footage.Face recognition often performs poorly on low-resolution images.
  • Motivation: Key face structures occupy little image area yet contain large pixel variations, making them difficult to recover with general image super-resolution.These structures include face-component shapes and the face outline.
  • Motivation: Equal-pixel-weighted MSE is ineffective for sparse structures, while auxiliary parsing and landmark tasks require extra manually labeled data.Prior methods also use predicted face priors to guide super-resolution.
  • Motivation: The paper frames face super-resolution as an imbalance between hard regions containing key structures and easy regions that do not, motivating bootstrapping or OHEM-like strategies.The analogy is drawn to foreground-background imbalance in object detection.
  • Approach: SPARNet uses Face Attention Units with differentiable 2D spatial attention to bootstrap features related to key face structures without extra supervision.Different FAUs can focus on different structures, with deeper layers emphasizing coarse features and shallower layers emphasizing textures.
  • Approach: SPARNetHD extends SPARNet to 512×512 face outputs using increased output resolution and multi-scale discriminators, while synthetic-data training generalizes to natural low-resolution inputs.The paper reports restoration of key face components and realistic high-resolution textures.

II. RELATED WORKS

Face super-resolution research spans subspace, component, CNN, attention, and GAN-based methods, but high magnification and supervision requirements remain challenges. SPARNet differs by using multi-scale spatial attention designed to preserve face edges and shapes.

  • Face Super-Resolution: Early face hallucination methods used subspaces or facial components, but required aligned faces or landmark detection and performed poorly at high upscale factors.Subspace methods typically rely on PCA, while component methods depend on detecting landmarks in low-resolution faces.
  • Face Super-Resolution: Deep CNN methods improved face super-resolution through cascaded estimation, GANs, wavelet prediction, and handling of unaligned, noisy, or attribute-varied faces.Recent methods also incorporate parsing maps, landmark heatmaps, identity information, or facial attention loss.
  • Face Super-Resolution: Auxiliary face-prior supervision can improve performance but requires extra data labeling and directs the super-resolution network indirectly toward key face features.The cited approaches include parsing, landmark, and identity supervision.
  • Attention Neural Networks: Attention methods reweight features to emphasize important information and have been applied across classification, captioning, visual question answering, image super-resolution, and face super-resolution.Prior image-generation attention methods include channel attention and reinforcement-learning-based patch attention.
  • Attention Neural Networks: SPARNet uses spatial rather than channel attention and exploits multi-scale features because pooling-based semantic attention may lose low- and middle-level edges and shapes needed for face super-resolution.This distinguishes the method from related attention approaches for image SR and face SR.
  • Generative Adversarial Networks: GAN-based high-resolution generation commonly uses multi-scale generators and discriminators, and SPARNet follows Pix2PixHD's multi-scale discriminator idea.Related work also includes progressive GAN training and semantic-layout-conditioned generation.

A. Overview

SPARNet upsamples an LR face, processes it through stacked Face Attention Units, and reconstructs an SR image with pixel-level L2 optimization. Its spatial attention branch emphasizes informative face structures within residual processing.

  • SPARNet first bicubically upsamples the LR input, then applies downscale, feature extraction, and upscale modules to produce the SR image.
  • SPARNet is optimized by minimizing a pixel-level L2 loss over LR-HR training pairs.
  • The Face Attention Unit extends a residual block with a spatial attention branch that continuously enhances important face features.
  • The attention map assigns spatial scores through a sigmoid, while the feature branch supplies features that are reweighted before residual fusion.

C. Multi-scale Discriminator Network

SPARNetHD extends SPARNet for 512×512 face super-resolution using multiscale discrimination and four complementary losses. The design balances pixel fidelity with sharper, more realistic textures and high-level semantics.

  • SPARNetHD increases SPARNet’s channels and uses discriminators at 512×512, 256×256, and 128×128 scales.Each discriminator compares SR images with ground-truth images downsampled to its scale.
  • Its training objective combines pixel, adversarial, feature matching, and perceptual losses.The loss weights are λpix, λadv, λfm, and λpcp.
  • SPARNetHD uses L1 pixel loss to constrain low-level information, especially color, whereas SPARNet uses pixel loss for distortion-metric comparison.
  • The adversarial loss sharpens outputs and promotes realistic textures such as hair, while feature matching stabilizes GAN training.

D. Training Details

SPARNet and SPARNetHD use distinct Adam optimization settings, reflecting their different training configurations.

  • SPARNet uses batch size 64 and learning rate 2 × 10^-4 with Adam parameters β1 = 0.9 and β2 = 0.99.
  • SPARNetHD uses batch size 2, learning rates 1 × 10^-4 for G and 4 × 10^-4 for D, and Adam parameters β1 = 0.5 and β2 = 0.99.
  • SPARNetHD sets λpix = 100, λadv = 1, λfm = 10, and λpcp = 1, and both models run in PyTorch on a Tesla K40 GPU.

IV. EXPERIMENTS

The experiments analyze SPARNet’s spatial attention, compare it with prior methods using distortion, identity, and landmark metrics, and evaluate SPARNetHD on real low-resolution faces trained from synthetic data.

  • The experiments separately evaluate SPARNet and SPARNetHD, including spatial-attention effectiveness, face-SR comparisons, and real-face generalization.
  • CelebA provides roughly 179K unaligned 16×16-to-128×128 LR-HR training pairs for SPARNet, with augmentation by flipping, rescaling, and rotation.
  • PSNR and SSIM measure luminance fidelity, while landmark detection and identity similarity assess face alignment and identity preservation.
  • The error-map visualization compares average Baseline and SPARNet PSNR/SSIM errors and displays their error reduction.
  • SPARNet produces better PSNR and SSIM image quality than the Baseline and SPARNet-V1 in the Helen test-set comparison.

2) Ablation Study:

The ablation study evaluates how the number and multi-scale design of spatial-attention branches affect SPARNet, comparing variants against a residual baseline. Spatial attention improves reconstruction quality and landmark detection, particularly around key facial structures.

  • The baseline is a residual super-resolution network without spatial-attention branches.
  • SPARNet-VN varies the number of attention branches from 1 to 16 while preserving the feature branch, testing the effect of multiple FAU blocks.The full model uses 16 attention branches in this variant family.
  • SPARNet-SM varies the bottleneck feature-map size from 2 × 2 to 16 × 16 to test whether attention benefits from multi-scale features.Smaller bottleneck sizes correspond to using more feature scales.
  • SPARNet improves PSNR, SSIM, and landmark detection over the baseline, with landmark AUC rising from 56.23% to 58.51%.The largest improvements in error maps occur around key face structures.
  • The ablations compare SPARNet-VN and SPARNet-SM on the Helen test set, using M = 4 for VN and N = 16 for SM.

1) Overall Results:

Across comparisons on Helen and UMD, SPARNet is reported to outperform generic and face-specific super-resolution methods while producing sharper key facial structures. It also offers strong high-upscale performance and efficiency relative to Wavelet-SRNet.

  • Quantitative comparisons: SPARNet outperforms generic and face-specific methods on Helen in both PSNR and SSIM.At 16 × 16 input and 8× upscaling, the comparison includes Helen and UMD test sets.
  • Visual comparisons: SPARNet recovers sharper eyes and nose shapes closer to the ground-truth high-resolution images than competing methods in qualitative comparisons.
  • Quantitative comparisons: With 16 × 16 inputs and 8× upscaling, SPARNet is evaluated against multiple methods on Helen and UMD.Table I reports AUC values under a 10% threshold.
  • Attention comparisons: RCAN’s PSNR is 26.40 versus the baseline’s 26.38, while CBAM’s pooling-based attention can lose shape and edge details.
  • Upscaling and efficiency: SPARNet requires fewer parameters and is more computationally efficient and flexible than Wavelet-SRNet.Wavelet-SRNet’s parameters increase quadratically with the upscale factor and require different input sizes across factors.
  • Landmark-based comparisons: On the Helen dataset supplied by DICNet, SPARNet achieves the best PSNR and SSIM and recovers eyes better than DICNet without landmark information.
  • Upscaling and efficiency: SPARNet achieves state-of-the-art results for both 4× and 16× upscale factors, especially at 16×.

3) Landmark Detection Results:

Landmark detection is used to assess recovery of important facial structures. SPARNet achieves the best landmark-detection performance and preserves identity information better than SICNN under matched training data without explicit identity supervision.

  • Landmark detection: SPARNet achieves the best landmark-detection performance among the compared methods.
  • Cross-dataset evaluation: On UMD-Face, models trained on CelebA without pre-alignment are evaluated, and SPARNet achieves the best performance.
  • Identity similarity: SPARNet* outperforms SICNN in identity similarity when both are trained using the same dataset, without explicit identity supervision.

C. Evaluation of SPARNetHD

SPARNetHD targets real-world low-resolution faces that vary in scale and degradation, extending SPARNet to 512 × 512 outputs. Its synthetic degradation process combines blur, downsampling, noise, and JPEG compression.

  • SPARNetHD is designed for noisy, blurry real-world low-resolution faces without fixed upscale factors, including old photos and low-end-device images.
  • The degradation model generates low-resolution inputs from high-resolution images using blur, downsampling, additive white Gaussian noise, and JPEG compression.The degradation parameters are randomly selected and the low-resolution samples are generated online.
  • The blur kernel is randomly selected from Gaussian, average, median, and motion blur families with specified kernel-size ranges.
  • The degradation process adds AWGN with noise level δ between 0 and 0.1 × 255.
  • JPEG compression quality is randomly selected from 60 to 85, with higher values described as stronger compression.

2) Datasets and Evaluation Metrics:

SPARNetHD is evaluated quantitatively and qualitatively on synthetic and real low-resolution face datasets against several high-resolution restoration methods. Results show superior performance, realistic detail recovery, and strong robustness on real faces.

  • Datasets: Training uses FFHQ images resized to 512×512 as ground-truth high-resolution faces, with synthetic low-resolution inputs generated online.
  • Evaluation: SPARNetHD is compared on CelebAHQ-Test and CelebA-TestN with ESRGAN, SFTGAN, Pix2PixHD, and GFRNet.
  • Quantitative comparison: SPARNetHD outperforms the other methods quantitatively on both CelebAHQ-Test and CelebA-TestN.
  • Qualitative comparison: Qualitatively, SPARNetHD restores key face components and hair and teeth textures more effectively than competing methods on synthetic test images.
  • Qualitative comparison: On natural low-resolution faces, competing methods produce many artifacts, whereas SPARNetHD generates more robust and natural results.
  • Comparison with GFRNet: SPARNetHD generates better texture details than GFRNet without using additional information during training.

V. CONCLUSION

The paper proposes SPARNet for very low-resolution face super-resolution and extends it to SPARNetHD for realistic 512×512 outputs. Experiments support the benefit of spatial attention for texture restoration and generalization to real-world low-resolution faces.

  • SPARNet stacks Face Attention Units that extend vanilla residual blocks with spatial attention branches.
  • Spatial attention directs less attention toward less feature-rich regions, making SPARNet training more effective and efficient.
  • SPARNetHD uses increased channel capacity and multi-scale discriminator networks to generate realistic 512×512 outputs from low-resolution faces.
  • Quantitative and qualitative comparisons indicate that spatial attention benefits restoration of low-resolution face texture details.
  • SPARNetHD trained on synthetic datasets generalizes well to real-world low-resolution faces, supporting practical applicability.
Loading 2012.01211v2…