Source-linked AI summary
Deep Cascaded Bi-Network for Face Hallucination
Shizhan Zhu, Sifei Liu, Chen Change Loy, Xiaoou Tang
TL;DR
The paper tackles face hallucination when faces are extremely low-resolution and spatially unaligned, where correspondence estimation and hallucination are mutually difficult. It alternates these complementary tasks in a cascade and adds a gated bi-network with common and high-frequency branches. Experiments report high PSNR and superior perceptual quality on low-resolution faces, including 5pxIOD settings, while failure cases remain for occlusions, large poses, and unreliable gaze direction.
Problem
Low-resolution blur and down-sampling make dense correspondence estimation ill-posed and degrade facial analysis, while hallucination itself benefits from spatial configuration.
Method
CBN alternates cascaded dense correspondence estimation and face hallucination, using a gated bi-network with common and high-frequency branches.
Results
The method achieves high PSNR and superior perceptual quality against general super-resolution and face hallucination approaches across various benchmarks.
Takeaways & Limitations
The formulation supports face hallucination for unconstrained poses and very low-resolution inputs by using progressively refined spatial cues and high-frequency priors.
Takeaways & Limitations
Failure cases include over-synthesis of occluded parts, ghosting under inaccurate spatial prediction for very large poses, and incorrect gaze direction when the input lacks reliable gaze information.
Abstract
from arXiv · showhide
We present a novel framework for hallucinating faces of unconstrained poses and with very low resolution (face size as small as 5pxIOD). In contrast to existing studies that mostly ignore or assume pre-aligned face spatial configuration (e.g. facial landmarks localization or dense correspondence field), we alternatingly optimize two complementary tasks, namely face hallucination and dense correspondence field estimation, in a unified framework. In addition, we propose a new gated deep bi-network that contains two functionality-specialized branches to recover different levels of texture details. Extensive experiments demonstrate that such formulation allows exceptional hallucination quality on in-the-wild low-res faces with significant pose and illumination variations.
1 Introduction
The paper addresses face hallucination from very low-resolution, unconstrained faces by alternating dense correspondence estimation with hallucination. Its gated bi-network uses complementary branches to recover texture details with spatial guidance.
- Motivation: Low-resolution face images degrade facial analysis because down-sampling and blur remove information and interfere with processing.Face hallucination is presented as a way to improve low-resolution face processing, including identification and enhancement.
- Problem: At 5pxIOD, estimating a high-resolution dense correspondence field is ill-posed, while blur also hinders alignment and parsing methods.These difficulties create a chicken-and-egg problem: hallucination needs spatial configuration, but spatial estimation needs a high-resolution face.
- Framework: The cascaded framework alternates dense correspondence estimation and face hallucination, progressively refining spatial fields as face resolution increases.Image resolution is adaptively upscaled using increasingly refined correspondence fields.
- Architecture: The gated deep bi-network contains common and high-frequency branches specialized for recovering different levels of facial texture detail.The common branch uses details detectable in the low-resolution input, while the high-frequency branch additionally uses warped spatial prior information.
- Contributions: The authors characterize CBN as avoiding assumptions of pre-aligned inputs or available landmarks or parsing maps while exploiting face spatial prior.The contribution also explores the lower resolution bound for recovering reasonable, high-quality details.
- Results: The method is evaluated against general super-resolution and face hallucination approaches, reporting high PSNR and superior perceptual quality.The experiments cover various benchmarks and include very-low-resolution settings such as 5pxIOD.
2 Related Work
Prior face hallucination methods use spatial cues in different forms, but low-resolution correspondence is difficult to obtain and some methods assume alignment or canonical views. CBN instead alternates dense correspondence estimation and face hallucination.
- Face hallucination and spatial cues: Previous face hallucination methods use spatial cues through warping functions, facial landmarks, or aligned input faces.These approaches differ in how they connect local reconstruction or input faces to high-resolution facial structure.
- Face hallucination and spatial cues: Low-resolution correspondence can be insufficient or ill-posed, especially when grayscale values are distributed across adjacent interpolated pixels.This limits landmark-based assumptions at 5pxIOD and complicates high-resolution spatial configuration estimation.
- Face hallucination and spatial cues: Methods relying on identical similarity transforms handle canonical-view low-resolution faces, whereas CBN estimates dense correspondence and hallucinates faces alternately.The paper describes the two tasks as mutually aiding refinement in a cascaded framework.
- Scope and representation: The paper assumes correspondence from pixel to pixel when defining its dense face correspondence field.The field maps points from the mean face template domain to coordinates in the target face image.
3 Cascaded Bi-Network (CBN)
CBN alternates dense face correspondence estimation with spatially guided face hallucination across cascades. Each hallucination stage combines a common branch, a high-frequency branch, and a learned gate, while correspondence is refined using shared deformation coefficients.
- 3 Cascaded Bi-Network (CBN): The framework predicts a high-resolution facial image from a low-resolution input while jointly refining dense correspondence across K cascades.Each cascade updates the deformation coefficients and then performs spatially guided hallucination.
- 3 Cascaded Bi-Network (CBN): The dense field maps mean-face-template coordinates to target-image coordinates and is parameterized by deformation bases and sample-specific coefficients p.When p = 0, the field equals the mean-face template.
- 3 Cascaded Bi-Network (CBN): The hallucination network predicts an image residual from the upscaled previous image and the current dense correspondence field.The residual is added to the upscaled image to produce the hallucinated output.
- 3 Cascaded Bi-Network (CBN): The gated bi-network combines a common branch and a high-frequency branch with a pixel-wise soft gate that controls their residual contributions.The common branch uses only the interpolated low-resolution image, whereas the high-frequency branch also receives a warped high-frequency prior.
- 3 Cascaded Bi-Network (CBN): The high-frequency prior is constructed by warping training residual maps into the mean-face-template domain and averaging their magnitudes to identify high-frequency locations.The resulting prior supplies spatial guidance to the high-frequency branch.
- 3 Cascaded Bi-Network (CBN): Training pre-trains the common and high-frequency branches separately, then fine-tunes the complete gated network with a final loss.The gate-related parameters use a learning rate 10 times that of the two branches during final fine-tuning.
- 3 Cascaded Bi-Network (CBN): Each cascade estimates correspondence coefficients with cascaded regression from shape-indexed landmark features, transferring landmark deformation to the dense field through shared coefficients.Dense-field and landmark bases are one-to-one related and share the same coefficient vector.
4 Experiments
Experiments evaluate CBN on constrained and in-the-wild faces under conventional down-sampling, unknown Gaussian blur, surveillance video, and component ablations. CBN outperforms compared methods across the reported settings, while its performance becomes unrealistic below 5pxIOD and its components are all important.
- Datasets and metric: The evaluation covers MultiPIE, BioID, PubFig, and HELEN, spanning constrained and in-the-wild faces with varied poses and illumination.PSNR is reported on the luminance channel of the facial region; SSIM is omitted for in-the-wild faces because of irregular facial shape.
- Evaluation settings: CBN is compared under conventional super-resolution and Gaussian-blurred settings, including fixed scale factors and inputs beginning at 5 or 8 pxIOD.The Gaussian-blur setting uses unknown Gaussian blur before down-sampling, following the comparison protocol of.
- Conventional SR results: CBN outperforms general super-resolution and face hallucination methods in both conventional scenarios, especially when hallucinating from very low resolution.The authors attribute the advantage to recovering details using the high-frequency prior and estimated dense correspondence field rather than only detectable input details.
- Blur and surveillance results: CBN again significantly outperforms all compared approaches under unknown Gaussian blur and is also evaluated on real surveillance videos with appealing visual quality.The authors attribute blur robustness to spatial guidance from the face high-frequency prior; surveillance-video inference uses single frames.
- Ablation study: The ablation results suggest that the gated bi-network, progressively updated dense correspondence field, and cascade are all important components of CBN.The study compares removing one branch, fixing the correspondence estimate from the low-resolution input, and replacing the cascade with a single network.
5 Input resolution lower bound
The study evaluates hallucination from 3, 5, 8, and 10 pxIOD inputs and identifies 3pxIOD as below a practical recovery lower bound. Inputs no smaller than 5pxIOD yield a significant PSNR increase and motivate using 5/8pxIOD faces thereafter.
- Input resolution lower bound: The experiment compares hallucination results across 3, 5, 8, and 10 pxIOD input face sizes.Models are trained to assess how small an input face can be recovered in-the-wild.
- Input resolution lower bound: 3pxIOD outputs are mostly unrealistic and visually dissimilar to the full-resolution image.The dense correspondence field is likely incorrectly predicted, while the input provides very little information.
- Input resolution lower bound: Inputs no smaller than 5pxIOD produce a significant PSNR increase.This observation supports selecting 5pxIOD and 8pxIOD faces as low-resolution inputs in Section 4.
6 Conclusion
The framework targets faces with substantial shape deformation and appearance variation. By adaptively refining correspondence fields and hallucinating faces alternately, it achieves state-of-the-art performance and visually appealing qualitative results while using high-frequency spatial cues.
- Conclusion: The framework hallucinates faces under substantial shape deformation and appearance variation.Its intended setting includes challenging changes in facial shape and appearance.
- Conclusion: Alternating refinement adaptively improves dense correspondence fields and face hallucination.The framework uses guidance between the two tasks during the hallucination process.
- Conclusion: The framework achieves state-of-the-art performance and visually appealing qualitative results.The conclusion reports both quantitative performance and visual quality outcomes.
- Conclusion: High-frequency priors provide spatial cues that the framework leverages during hallucination.These cues are incorporated through the estimated correspondence field.