Learning Spatial Attention for Face Super-Resolution

• SPARNet uses FAUs to assign differentiable 0-to-1 spatial attention scores that bootstrap features for key face structures, with different units learning to focus on different structures.Deeper-layer maps focus more on coarse structures such as eyes and mouth, while shallower-layer maps focus more on detailed textures such as hair.Source: §I · pp. 1–2
• SPARNetHD extends SPARNet from 128×128 to 512×512 outputs using a multi-scale discriminator loss, and synthetic-data training works well on natural low-resolution images.The introduction reports realistic textures and key-component recovery in an example, and says models without spatial attention produce undesirable artifacts.Source: §I · p. 2
• SPARNet achieves state-of-the-art face-SR performance across PSNR, SSIM, identity similarity, and landmark detection without extra supervision such as parsing maps or landmarks.The paper also identifies FAUs and repeated attention units as contributors to improved performance.Source: §I · p. 2

II. RELATED WORKS

• Baker and Kanade pioneered face hallucination; later sub-space methods,,,, and component methods,, both struggle at high upscale factors.Sub-space methods rely on precisely aligned faces, while component methods require facial landmarks that are difficult to detect in low-resolution images.Source: A. Face Super-Resolution · p. 2

A. Face Super-Resolution

• SPARNet learns spatial attention maps to bootstrap key face structures and other feature-rich regions, rather than relying on extra face-prior supervision.Prior approaches use face parsing maps, landmark heatmaps, or identity information, which require additional labeled data and direct the SR network indirectly.Source: A. Face Super-Resolution · p. 2
• Deep CNN face-SR methods include cascaded face-hallucination and correspondence estimation, GAN-based SR, wavelet prediction, and variants for unaligned, noisy, or attribute-diverse faces.Other work adds face parsing, landmark heatmaps,, identity information, facial attention loss, or iterative landmark prediction.Source: A. Face Super-Resolution · p. 2

B. Attention Networks

• Attention methods reweight features to emphasize important information; prior uses span classification,,,, image generation, and face SR.Examples include trunk-and-mask attention, channel-wise Squeeze-and-Excitation, CBAM, channel attention for image SR, and reinforcement-learning patch attention for face SR.Source: B. Attention Networks · pp. 2–3
• For face SR, SPARNet’s attention uses multi-scale features, spatially emphasizes key components, and generates whole-face maps for global structural context.The paper contrasts this with pooling that may lose low- and middle-level edges and shapes, channel attention in, and patch-based attention in.Source: B. Attention Networks · p. 3

C. High Resolution Image Generation with GAN.

• SPARNetHD follows Pix2PixHD in using a multi-scale discriminator for high-resolution generation, but does not require progressive LR-to-HR training and is reported to outperform Pix2PixHD.The related work also reviews progressive GAN training for high-resolution faces [33],, [35] and semantic-layout-conditioned generation.Source: C. High Resolution Image Generation… · p. 3

A. Overview

• SPARNet comprises downscale, feature-extraction, and upscale modules, each built from stacked FAUs; bicubic interpolation upsamples the LR input before it enters the network.The network produces a super-resolved image from the upsampled input.Source: A. Overview · p. 3
• SPARNet is optimized with pixel-level L2 loss on paired low- and high-resolution training images.Source: A. Overview · p. 3

B. Face Attention Unit

• The attention mechanism prioritizes informative components such as eyes, eyebrows, nose, and mouth over less informative regions, and is designed to learn both high-level and local structures.A high-level view helps the network learn face appearance, while a low-level view supports local-detail learning.Source: B. Face Attention Unit · p. 3
• Each Face Attention Unit (FAU) extends a residual block with a spatial-attention branch, and stacking FAUs continuously enhances features important for face super-resolution.The attention map and features are integrated using element-wise multiplication.Source: B. Face Attention Unit · pp. 3–4
• The attention branch uses a multi-scale hourglass followed by a convolution, while the feature branch uses pre-activation residual units with batch normalization and PReLU.The authors report slightly better performance with batch normalization in the pre-activation structure and say PReLU avoids dead features and provides more stable performance.Source: B. Face Attention Unit · p. 4
• In the downscale and upscale modules, the residual branch uses scale convolution; downscaling uses stride-2 convolution, while upscaling uses nearest-neighbor interpolation followed by convolution to help avoid checkerboard artifacts.Source: B. Face Attention Unit · p. 4

C. Multi-scale Discriminator Network

• SPARNetHD widens SPARNet and uses Pix2PixHD-style discriminators at 512 × 512, 256 × 256, and 128 × 128 to improve SR image quality.Each discriminator compares SR images with ground truth downsampled to its corresponding resolution.Source: C. Multi-scale Discriminator Network · p. 4
• Training combines L1 pixel loss for low-level information, adversarial loss for sharper textures, feature matching to stabilize GAN training, and VGG19 perceptual loss for high-level semantics.The perceptual loss uses pretrained VGG19 features [45], while feature matching is based on discriminator features.Source: C. Multi-scale Discriminator Network · p. 4
• Because distortion and perceptual quality can conflict, the paper trains SPARNet with pixel loss for fair PSNR and SSIM comparisons, while SPARNetHD uses more complex losses for perceptual quality.The authors note that pixel-loss-only training can improve PSNR and SSIM but produce over-smoothed outputs with poor perceptual quality.Source: C. Multi-scale Discriminator Network · p. 4

D. Training Details

• SPARNet uses batch size 64, learning rate 2 × 10^-4, and Adam [47] with β1 = 0.9 and β2 = 0.99.The models are implemented in PyTorch and run on a Tesla K40 GPU.Source: D. Training Details · p. 5
• SPARNetHD uses batch size 2 and learning rates of 1 × 10^-4 for G and 4 × 10^-4 for D.Its loss weights are λpix = 100, λadv = 1, λfm = 10, and λpcp = 1; Adam uses β1 = 0.5 and β2 = 0.99 for both G and D.Source: D. Training Details · p. 5

IV. EXPERIMENTS

• Experiments analyze SPARNet’s spatial attention and compare it with face-SR and general-SR methods, then evaluate SPARNetHD on real low-resolution faces using synthetic training data.Source: §IV · p. 5

A. Analysis of SPARNet

• SPARNet is trained on roughly 179K CelebA pairs, with 128 × 128 HR faces downsampled to 16 × 16 and augmented by flipping, rescaling, and rotation.Faces are detected with MTCNN and cropped without pre-alignment.Source: 1) Datasets and Evaluation Metrics · p. 5
• Image quality and landmark detection are evaluated on Helen, while identity similarity uses the test set of 1,000 UMD-Face [52] identities used in.The Helen test set follows prior work,,.Source: 1) Datasets and Evaluation Metrics · p. 5

1) Datasets and Evaluation Metrics:

• PSNR and SSIM on the luminance channel are the primary quantitative metrics, supplemented by landmark detection and identity similarity for face alignment and identification.Source: 1) Datasets and Evaluation Metrics · p. 5
• Landmark accuracy is measured with FAN1, pretrained on HR faces, using the area under the curve of normalized mean error.The model detects landmarks on both super-resolved and HR faces.Source: 1) Datasets and Evaluation Metrics · p. 5
• Figure 5 compares Baseline, SPARNet-V1, and SPARNet on Helen; its average PSNR/SSIM error maps mark larger values as worse and larger improvements as more error reduction.Source: Figure 5 · p. 5, 1) Datasets and Evaluation Metrics · p. 5
• Identity similarity is the cosine distance between SphereFace feature vectors extracted from super-resolved and HR faces, following.Source: 1) Datasets and Evaluation Metrics · p. 5

2) Ablation Study:

• The ablations define a residual network without attention as the Baseline, vary attention-branch counts in SPARNet-VN, and use SPARNet-V16-S4 as the full SPARNet model.SPARNet-VN varies N across 1, 2, 4, 8, and 16 branches, increasing branches from the backend.Source: 2) Ablation Study · p. 5
• 58.51% AUC for SPARNet versus 56.23% AUC for the Baseline; SPARNet also improves PSNR and SSIM and produces sharper facial structures, especially around the eyes.The reported improvements concentrate around key facial structures, which are difficult to recover.Source: 2) Ablation Study · p. 6
• PSNR scores rise with the number of attention branches, and stacked FAUs progressively focus attention on different key facial structures, unlike the noisier single-FAU attention maps.SPARNet, composed entirely of FAUs, performs better than SPARNet-V1, which has one FAU at the end.Source: 2) Ablation Study · p. 6, Figure 6 · p. 6, Figure 7 · p. 6
• Models using more feature scales generally perform better, but gains diminish as bottleneck size M decreases; SPARNet uses M = 4 to balance performance and computation.SPARNet-S4 performs similarly to SPARNet-S2.Source: 2) Ablation Study · p. 6, Figure 7 · p. 6

B. Comparisons of SPARNet against Other Methods

• The comparison evaluates RCAN, CBAM, URDGN, Wavelet-SRNet, Attention-FH, FSRNet, SICNN, PFSRNet, and DICNet.Source: B. Comparisons of SPARNet against… · p. 6
• For RCAN and CBAM comparisons, SPARNet substitutes the corresponding attention module while keeping network depth the same; all models use the same training dataset except FSRNet and SICNN.FSRNet provides only test code, while SICNN relies on a much larger prealigned training set.Source: B. Comparisons of SPARNet against… · p. 6

1) Overall Results:

• SPARNet outperforms generic and face super-resolution methods on Helen in both PSNR and SSIM.Table I covers Helen and UMD tests with 16 × 16 inputs and an 8× upscale factor.Source: 1) Overall Results · p. 6, Table I · p. 7

2) Detailed Comparisons:

• 26.40 vs 26.38: RCAN shows little PSNR improvement over Baseline; Attention-FH cannot recover details, while CBAM pooling loses shape and edge details.Unlike RCAN's channel attention, SPARNet's multiscale spatial attention maps outline details such as eyes and mouth more clearly than CBAM's blurry maps.Source: 2) Detailed Comparisons · pp. 6–7, Figure 9 · p. 7
• SPARNet outperforms FSRNet in PSNR and SSIM despite FSRNet's additional face-parsing supervision.For a fair comparison, SPARNet was retrained and tested on FSRNet's dataset of 18K CelebA images.Source: 2) Detailed Comparisons · p. 7, Table II · p. 7
• SPARNet uses fewer parameters and is more computationally efficient and flexible than Wavelet-SRNet, while retaining better performance without increasing complexity across upscale factors.Wavelet-SRNet's parallel subnetworks make parameters grow quadratically with upscale factor, and its GFLOPs rise when that factor is smaller.Source: 2) Detailed Comparisons · pp. 7–8, Figure 10 · p. 8
• On DICNet's Helen test set, SPARNet achieves the best PSNR and SSIM, uses fewer parameters and runs faster than DICNet, and outperforms PFSRNet with similar computation complexity.Using public models trained on CelebA, SPARNet also recovers key face components—especially eyes—better than DICNet without extra landmark information; PFSRNet performs badly on unaligned faces.Source: 2) Detailed Comparisons · p. 8, Table III · p. 8, Figure 11 · p. 8
• SPARNet achieves state-of-the-art results at both 4× and 16× upscale factors, especially at 16×.The 4× model was trained from scratch, while the 16× model was initialized from the pretrained 8× model.Source: 2) Detailed Comparisons · p. 8, Table IV · p. 8

Landmark Detection Results

• SPARNet achieves the best performance in landmark detection.Landmark detection is used to evaluate how well the method recovers important face structures.Source: 3) Landmark Detection Results · p. 8
• Although Wavelet-SRNet performs similarly in landmark detection, SPARNet is more efficient and performs much better at the 16× upscale factor on PSNR/SSIM.The passage directs readers to Fig. 10 for efficiency and Table IV for the 16× PSNR/SSIM comparison.Source: 3) Landmark Detection Results · p. 8

4) Identity Similarity Results:

• SPARNet achieves the best performance on the UMD-Face test set among models trained on CelebA without pre-alignment.The passage does not provide the metric values.Source: 4) Identity Similarity Results · p. 8
• GAN-based methods such as URDGN may produce sharper textures but can disturb identity information because they prioritize realistic textures over consistency with the high-resolution image.Source: 4) Identity Similarity Results · pp. 8–9
• SPARNet∗ outperforms SICNN in identity similarity without explicit identity supervision, supporting the benefit of spatial attention for preserving identity.SPARNet∗ was retrained on the same dataset as SICNN for a fair comparison.Source: 4) Identity Similarity Results · p. 9, Table V · p. 9

C. Evaluation of SPARNetHD

• SPARNetHD targets real-world low-resolution faces, which may be noisy, blurry, and lack fixed upscale factors, and requires high-resolution training data to produce high-resolution outputs.The paper gives old photos and unclear images captured by low-end devices as typical scenarios.Source: C. Evaluation of SPARNetHD · p. 9

1) Degradation Model:

• The degradation model generates low-resolution inputs online by blurring and downsampling high-resolution images, adding AWGN, and applying JPEG compression, with hyperparameters randomly selected per image.The resulting low-resolution image, upsampled by the selected scale factor, serves as SPARNetHD’s input.Source: 1) Degradation Model · p. 9
• The blur kernel is randomly chosen from Gaussian, average, median, or motion blur, with parameter ranges 3–15 for the first three and 5–25 for motion blur.Source: 1) Degradation Model · p. 9
• AWGN noise levels are sampled from 0 to 0.1 × 255, while JPEG compression quality is randomly selected from 60 to 85, with higher compression strength at higher quality values.Source: 1) Degradation Model · p. 9

2) Datasets and Evaluation Metrics:

• Training uses 70,000 aligned FFHQ images resized to 512×512; synthetic CelebAHQ-Test has 2,824 test faces scored by LPIPS, while 1,051 natural CelebA-TestN faces use FID against HR CelebAHQ.CelebAHQ-Test is synthetically degraded; CelebA-TestN comprises collected natural low-resolution faces and old photos, with no ground-truth high-resolution images.Source: 2) Datasets and Evaluation Metrics · p. 9

3) Ablation Study of Loss Functions:

• Removing adversarial loss most degrades sharpness and quantitative performance; Lpix affects subtle details, omitting Lfm destabilizes GAN training, and omitting Lpcp distorts eyes and lips.The ablation compares models trained for 100k iterations, taking about three days on one GPU.Source: 3) Ablation Study of Loss Functions · p. 9, Figure 12 · p. 10

4) Comparison with Other Methods:

• SPARNetHD outperforms other methods on both datasets, surpassing Pix2PixHD [5] and SFTGAN [60], while Pix2PixHD [5] performs better than ESRGAN [59].The paper attributes SPARNetHD's advantage over SFTGAN to spatial attention providing both coarse semantic and low-level texture guidance, unlike parsing maps alone.Source: 4) Comparison with Other Methods · pp. 10–11, Table VI · p. 10
• On CelebAHQ-Test, SPARNetHD restores key face components and hair and teeth textures, whereas ESRGAN [59] produces unrealistic faces and SFTGAN [60] lacks detailed textures.Pix2PixHD [5] produces better-looking results than ESRGAN [59] but shows artifacts around the right eyes; these qualitative findings are consistent with the quantitative comparison.Source: Figure 13 · p. 10, 4) Comparison with Other Methods · p. 11
• On natural CelebA-TestN faces, SPARNetHD's results are more robust and natural than the artifacts produced by competing methods, indicating generalization to real low-resolution images.The paper reports similar qualitative observations for natural faces in Fig. 14.Source: 4) Comparison with Other Methods · p. 11, Figure 14 · p. 11
• SPARNetHD produces better texture details than GFRNet [55] on natural low-resolution faces, even without additional training-stage information.GFRNet [55] uses a low-resolution image and a high-resolution image of the same person as input.Source: 4) Comparison with Other Methods · p. 11

V. CONCLUSION

• Extensive experiments across multiple metrics show SPARNet’s advantages over current state-of-the-art methods.Source: §V · p. 11
• SPARNet stacks Face Attention Units, which extend vanilla residual blocks with a spatial attention branch that focuses less on less feature-rich regions.The paper says this makes SPARNet training more effective and efficient.Source: §V · p. 11
• SPARNetHD trained on synthetic datasets generates realistic 512 × 512 outputs for low-resolution faces and generalizes well to real-world low-resolution faces.The paper also reports that quantitative and qualitative comparisons indicate spatial attention benefits the restoration of texture details.Source: §V · p. 11