Source-linked AI summary
Deep Face Super-Resolution with Iterative Collaboration between Attentive Recovery and Landmark Estimation
Cheng Ma, Zhenyu Jiang, Yongming Rao, Jiwen Lu, Jie Zhou
TL;DR
Existing face super-resolution methods may rely on inaccurate facial priors estimated from low-resolution or coarse reconstructions. This paper introduces recurrent recovery and landmark-estimation branches that collaborate iteratively, with attentive fusion for facial components, and reports superior results on two benchmark datasets.
Problem
Existing FSR methods may estimate landmark and component priors from low-resolution or coarse super-resolved images, limiting prior accuracy and guidance.
Method
The method uses recurrent SR and alignment branches that exchange outputs across steps, while an attentive fusion module separately processes facial components using landmark guidance.
Results
The method achieves the best PSNR and SSIM on CelebA and Helen and shows progressively improving SR and landmark performance across recurrent steps.
Takeaways & Limitations
Iterative collaboration progressively improves both reconstructed face images and landmark estimates, while attentive fusion exploits component-specific facial structure.
Abstract
from arXiv · showhide
Recent works based on deep learning and facial priors have succeeded in super-resolving severely degraded facial images. However, the prior knowledge is not fully exploited in existing methods, since facial priors such as landmark and component maps are always estimated by low-resolution or coarsely super-resolved images, which may be inaccurate and thus affect the recovery performance. In this paper, we propose a deep face super-resolution (FSR) method with iterative collaboration between two recurrent networks which focus on facial image recovery and landmark estimation respectively. In each recurrent step, the recovery branch utilizes the prior knowledge of landmarks to yield higher-quality images which facilitate more accurate landmark estimation in turn. Therefore, the iterative information interaction between two processes boosts the performance of each other progressively. Moreover, a new attentive fusion module is designed to strengthen the guidance of landmark maps, where facial components are generated individually and aggregated attentively for better restoration. Quantitative and qualitative experimental results show the proposed method significantly outperforms state-of-the-art FSR methods in recovering high-quality face images.
1. Introduction
Face super-resolution restores high-resolution faces from low-resolution inputs, using facial structure as prior knowledge. The proposed method addresses inaccurate priors by iteratively coupling face recovery with landmark estimation and attentively fusing facial-component information.
- FSR restores high-resolution facial images from low-resolution counterparts and supports applications including surveillance, enhancement, recognition, and alignment.
- Facial configuration provides strong prior knowledge for recovering global structures and local details, especially at high upscaling factors such as 8×.
- Existing methods may estimate landmarks or component priors from low-resolution or coarse super-resolved images, making their guidance inaccurate.
- The proposed method uses recurrent recovery and landmark-estimation branches that feed outputs to each other across steps for progressively better images and landmarks.
- An attentive fusion module extracts features for facial components individually and aggregates them using component-specific attention.
- Experiments on CelebA and Helen report superior high-quality face super-resolution compared with state-of-the-art FSR methods.
2. Related Work
Related work spans deep face hallucination, facial-prior methods, perceptual single-image super-resolution, and recurrent architectures. The proposed framework builds on recurrent processing while using two collaborating branches for super-resolution and alignment.
- Face Super-Resolution: Deep face super-resolution methods include generative, wavelet-based, attribute-guided, identity-aware, and perceptual approaches.
- Facial Priors: Facial-prior methods use component heatmaps, landmark and parsing maps, attention losses, or alternating correspondence and hallucination networks.
- Facial Priors: The cascaded correspondence framework can be redundant and inflexible, while inaccurate dense fields may cause severe distortions.
- Single Image Super-Resolution: Single-image super-resolution research progressed from SRCNN and VDSR toward perceptual and adversarial methods that target photo-realistic outputs.
- Recurrent Networks: Recurrent SISR methods use recursive, residual, dense, dual-state, and feedback designs to iteratively fuse features or LR and HR signals.
- Proposed Framework: The proposed framework applies recurrent processing to both SR and alignment, enabling collaboration between the two branches.
3. Approach
The method iteratively couples face recovery with landmark estimation, using progressively improved outputs to refine both processes. Its attentive fusion module separately processes facial components and aggregates them under landmark-derived guidance.
- 3.1. Deep Iterative Collaboration: Accurate landmark maps guide higher-quality face recovery, while improved SR images support more accurate landmark localization.The paper describes this mutual enhancement as progressive collaboration between recovery and alignment.
- 3.1. Deep Iterative Collaboration: The deep iterative collaboration network estimates super-resolved faces and landmark maps recursively, with each process guiding the other.The SR branch uses landmark results and prior-step feedback, while the alignment branch uses SR images as guidance.
- 3.2. Attentive Fusion Module: The attentive fusion module uses group convolutions to generate individual component features, weights them with attention maps, and sums them into output features.Because the module lies in the recurrent SR branch, its gradients can propagate to both SR and alignment branches.
- 3.1. Deep Iterative Collaboration: Losses are imposed at every recurrent step so mutual supervision progressively corrects inaccurate SR and alignment factors.The final output uses the SR image from the last recurrent step.
- 3.2. Attentive Fusion Module: Landmark channels are grouped into facial components, converted into attention maps, and used to guide component-specific feature extraction.Grouping highlights local structures and reduces the number of channels.
- 3.3. Objective Functions: Adversarial and perceptual losses are added to the objective for perceptual-quality training, while DIC sets both corresponding trade-off parameters to zero.The complete losses produce the perceptual-oriented DICGAN model.
- 3.4. Experiments: On CelebA and Helen, DIC achieves the best PSNR and SSIM among the compared state-of-the-art FSR methods.The paper specifically reports a large margin over FSRNet.
4. Experiments
Experiments on CelebA and Helen evaluate reconstruction quality, alignment accuracy, iterative refinement, attentive fusion, and perceptual quality. The results show progressive improvement across steps, strong performance against state-of-the-art methods, and benefits from landmark guidance and component-specialized generation.
- Experimental Setup: Experiments use CelebA and Helen, evaluating SR with PSNR and SSIM and landmark accuracy with normalized NRMSE.The datasets contain 128×128 HR faces downsampled to 16×16 LR inputs; NRMSE is normalized by face width.
- Comparison with State-of-the-Arts: DIC achieves the best PSNR and SSIM on both CelebA and Helen, while DICGAN preserves pixel-wise accuracy while increasing perceptual quality.The comparison reports that DIC outperforms FSRNet by a large margin, whereas DICGAN has comparable performance with the PSNR-oriented FSRNet.
- Qualitative Comparison: Visual comparisons show DIC recovers facial details under large pose and rotation variations, while DICGAN produces more realistic textures than competing methods.The paper links improved structural preservation to progressively more accurate landmarks guiding reconstruction at each step.
- Comparison with State-of-the-Arts: DICGAN outperforms other methods on NRMSE across both datasets, indicating stronger landmark-based alignment accuracy.The paper attributes this to revising landmark estimates at every step and integrating the resulting guidance through attentive fusion.
- Study of Iterative Learning: From step 1 to step 4, PSNR, SSIM, and NRMSE improve progressively on CelebA and Helen.The authors identify three steps as a suitable balance between good recovery and efficient computation.
- Ablation Study: Ablations show that removing landmark guidance severely degrades SR quality, while concatenated landmarks improve results but remain below the attentive-fusion DIC model.The attentive fusion module also supports component-specialized generation by extracting and selectively aggregating features for facial parts.
5. Conclusion
The paper proposes a deep iterative collaboration network in which recurrent recovery and alignment branches progressively improve one another for face super-resolution.
- A recurrent SR branch uses estimated landmarks to produce improved face images that help the alignment branch estimate more accurate landmarks.
- An attentive fusion module supports component-specialized generation by guiding facial components individually before aggregation.
A. More Details on Network Architecture
The method uses recurrent SR and alignment branches that pass feedback features through dedicated processing modules to generate super-resolved images and landmarks.
- The SR branch extracts features from input LR images, concatenates feedback features, and processes them through convolution, attentive fusion, and recurrent SR modules.The resulting features serve as feedback signals and inputs for subsequent generation.
- The alignment branch uses preprocessing and postprocessing blocks around a recurrent hourglass module, with both input and output containing two components.
B. User Study
A user study evaluated perceptual quality by asking 39 raters to rank outputs from four versions of 30 CelebA test images.
- 39 human raters ranked DICGAN, FSRGAN, PFSR, and HR versions for 30 randomly selected CelebA test images.The four versions were displayed in random order and judged by perceptual satisfaction.
- DICGAN received substantially more rank-1 and rank-2 votes than FSRGAN and PFSR.The study passage reports that PFSR scored worst among the three FSR methods, particularly for faces with large pose and rotation variations.
C. Visual Results
Qualitative comparisons and a user study report stronger perceptual face recovery for the proposed method than for state-of-the-art FSR methods.
- Visual Results: Qualitative comparisons evaluate the proposed method against RDN, FSRNet, FSRGAN, and PFSR.
- Visual Results: The user study reports that the proposed method performs better than state-of-the-art FSR methods in recovering perceptually pleasant face images.