Source-linked AI summary
Attention-Aware Face Hallucination via Deep Reinforcement Learning
Qingxing Cao, Liang Lin, Yukai Shi, Xiaodan Liang, Guanbin Li
TL;DR
Face hallucination must recover high-resolution faces from low-resolution inputs while existing patch-to-patch approaches often neglect dependencies among facial parts. Attention-FH uses recurrent reinforcement learning to select and enhance facial regions sequentially from whole-face context, and experiments report substantially better performance than compared methods, including state-of-the-art results on standard datasets.
Problem
Face hallucination seeks to generate high-resolution faces from low-resolution inputs, but existing patch-based methods often ignore contextual interdependencies among facial parts.
Method
Attention-FH jointly trains a recurrent policy network to select facial regions and a local enhancement network, using prior whole-face results and a long-term global reward.
Results
Attention-FH substantially surpasses compared methods, achieves state-of-the-art performance on evaluation datasets, and improves PSNR over an attention-free recurrent variant by 0.67 dB on LFW 4×.
Takeaways & Limitations
Sequential attention lets the model incorporate correlation cues among facial parts while producing attentional-region sequences that accord with human perception.
Abstract
from arXiv · showhide
Face hallucination is a domain-specific super-resolution problem with the goal to generate high-resolution (HR) faces from low-resolution (LR) input images. In contrast to existing methods that often learn a single patch-to-patch mapping from LR to HR images and are regardless of the contextual interdependency between patches, we propose a novel Attention-aware Face Hallucination (Attention-FH) framework which resorts to deep reinforcement learning for sequentially discovering attended patches and then performing the facial part enhancement by fully exploiting the global interdependency of the image. Specifically, in each time step, the recurrent policy network is proposed to dynamically specify a new attended region by incorporating what happened in the past. The state (i.e., face hallucination result for the whole image) can thus be exploited and updated by the local enhancement network on the selected region. The Attention-FH approach jointly learns the recurrent policy network and local enhancement network through maximizing the long-term reward that reflects the hallucination performance over the whole image. Therefore, our proposed Attention-FH is capable of adaptively personalizing an optimal searching path for each face image according to its own characteristic. Extensive experiments show our approach significantly surpasses the state-of-the-arts on in-the-wild faces with large pose and illumination variations.
1. Introduction
Face hallucination generates high-resolution faces from low-resolution inputs, but existing patch-based methods often ignore dependencies among facial parts. Attention-FH addresses this by sequentially selecting and enhancing regions with recurrent reinforcement learning while using whole-face context.
- Face hallucination generates a high-resolution face image from a low-resolution input and supports face analysis tasks including recognition and alignment.
- Existing methods commonly learn LR-to-HR patch mappings, but they usually ignore contextual dependencies among facial parts.Human perception instead begins with the whole image and successively explores attended regions.
- Attention-FH recurrently discovers facial parts and enhances them using the global interdependency of the face.The framework adapts its enhancement route to image characteristics such as blur, pose, illumination, and appearance.
- The framework jointly optimizes a recurrent policy network for selecting facial parts and a local enhancement network for hallucinating them from prior whole-face results.This allows enhancement of one region to incorporate clearer information from previously enhanced regions, such as using an eye region when enhancing the mouth.
- A global reward based on overall super-resolved-face performance trains the recurrent policy over a long-term sequence of region selections.An LSTM captures past attended parts, while memorized actions help avoid repetitive action cycles.
- Experiments report that Attention-FH substantially surpasses compared methods and produces attentional-region sequences that accord with human perception.
2. Related Work
Related work treats face hallucination as a specialized form of image super-resolution and has progressed from hand-crafted priors and component mappings to deep convolutional models. Attention and reinforcement learning provide related foundations for sequentially using contextual information.
- Face Hallucination and Image Super-Resolution: Face hallucination is a special case of image super-resolution requiring more informative facial structure priors and handling more challenging blur.
- Face Hallucination and Image Super-Resolution: Early face hallucination methods assumed controlled settings with small variations, using eigen transformations, sparse priors, or facial-component mappings.Component matching can depend on landmark detection, which may be unavailable at large down-sampling factors.
- Face Hallucination and Image Super-Resolution: Deep convolutional networks and cascaded or local auto-encoder architectures were later applied to face hallucination and general image super-resolution.
- Reinforcement Learning and Attention Networks: Attention mechanisms have been used across vision tasks, while reinforcement learning optimizes sequential attention models with delayed rewards.These approaches formulate attention over multiple regions as sequential decision problems to incorporate contextual information.
3. Attention-Aware Face Hallucination
Attention-FH formulates face hallucination as sequential local-part selection and enhancement, using recurrent policies conditioned on the evolving whole-face result and action history. The policy and enhancement networks are jointly optimized so each selected patch updates the current hallucination result.
- Attention-FH sequentially locates and enhances attended facial parts using a recurrent policy network and a local enhancement network.The policy dynamically determines the facial part for the current step, while the enhancement network processes the selected part.
- At step t, the policy selects a local patch location from the current image and encoded action history, then the enhancement network replaces that patch with an enhanced version.The selected patch is cropped from I_t−1, enhanced, and written back to produce I_t.
- The recurrent policy outputs probabilities over all W × H image locations after encoding the current hallucination result and fusing it with action history through an LSTM.During testing, the location with the highest probability is selected.
- The policy is trained with a global reward based on final whole-face MSE, while the enhancement network uses supervised patch-level MSE loss.REINFORCE optimizes the policy from the end-of-sequence reward, and back-propagation minimizes enhancement loss at each step.
- The enhancement network combines holistic information from the whole face with the selected patch before producing the enhanced local patch through cascaded convolutional layers.Its architecture uses two fully connected layers with 256 neurons and eight cascaded convolutional layers.
- Joint training introduces a non-stationary policy objective because enhancement-network updates change the final hallucination result, motivating variance reduction during training.The sequential procedure uses a maximal number of T = 25 local patch-mining steps.
4. Experiments
Experiments evaluate Attention-FH on BioID and LFW using standard image-quality metrics, comparisons with state-of-the-art methods, ablations, visualizations, and computational-cost measurements. The method achieves strong quantitative and qualitative results, while ablations support sequential attention, contextual patch selection, and patch-wise enhancement.
- Experimental setup: Experiments use constrained BioID and unconstrained LFW datasets, with PSNR, SSIM, and FSIM as evaluation metrics.BioID contains 1,521 images, while LFW contains 13,233 images; the reported train/test splits follow the dataset settings.
- Quantitative and qualitative comparisons: Attention-FH substantially beats compared methods on LFW and BioID in PSNR, SSIM, and FSIM.Average PSNR gains over the second-best method are 2.59dB, 1.66dB, 2.43dB, and 1.8dB across the reported experiments.
- Quantitative and qualitative comparisons: Qualitative results show clearer images than GLN and VDSR despite large variations, including successful recovery of eyes missed by the comparison methods.Figures 3 and 4 present comparisons on LFW and BioID under scaling factors 8 and 4.
- Ablation studies: Patch-wise attention improves PSNR over a recurrent whole-image model by 0.67dB on LFW 4× and 0.46dB on LFW 8×.Compared with CNN-16, improvements are 3.82dB and 2.14dB on LFW at factors 4 and 8, respectively.
- Ablation studies: Random patch selection reduces PSNR by 1.33dB and 0.41dB, while omitting previous enhancement information reduces it by 0.83dB and 0.25dB on LFW 4× and 8×.These ablations support agent-selected attention sequences and conditioning on previous enhanced information.
- Visualization of attended regions: The agent’s attended sequence moves from image corners to facial components and finally to detailed high-frequency regions.Figure 5 visualizes intermediate enhancement results and the corresponding attended patches.
- Computational cost: Despite multiple passes, the 8-layer local enhancement network has running time comparable to one-pass full-image super-resolution methods.The reported testing comparison uses single 128 × 128 images on a TITANX GPU.
5. Conclusion
Attention-FH frames face hallucination as a Markov decision process and optimizes it with deep reinforcement learning to incorporate correlations among facial parts. Experiments show state-of-the-art performance and better visual results on evaluation datasets.
- Attention-FH explicitly incorporates rich correlation cues among different facial parts by casting face hallucination as a Markov decision process.
- Attention-FH is optimized using deep reinforcement learning.
- The model achieves state-of-the-art performance on popular evaluation datasets.
- The model demonstrates better visual results on popular evaluation datasets.