PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning-based Face Super-Resolution: A Survey

Junjun Jiang, Chenyang Wang, Xianming Liu, Jiayi Ma

arXiv:2101.03749v2cs.CV

TL;DR

Face super-resolution seeks to recover high-resolution face images from low-resolution inputs, but the field has lacked a systematic deep learning-focused summary. This survey synthesizes formulations, metrics, losses, facial information, datasets, methods, evaluations, applications, and future directions, while highlighting trade-offs among reconstruction quality, perceptual quality, and training stability.

  • Problem

    FSR addresses recovery of high-resolution face images from low-resolution inputs, while the survey notes that few summaries of deep learning-based FSR studies were available.

  • Method

    The survey systematically reviews FSR formulations, metrics, loss functions, facial characteristics, datasets, method categories, evaluations, joint tasks, applications, and future prospects.

  • Results

    The survey organizes deep learning-based FSR into five facial-characteristic-based categories and compares representative methods, including their advantages and disadvantages.

  • Takeaways & Limitations

    FSR design involves trade-offs: perceptual or adversarial losses can improve visual quality but may lower PSNR and SSIM, introduce artificial details, or destabilize training.

  • Takeaways & Limitations

    Adversarial-loss training can be unstable, may cause model collapse, and tends to introduce artificial details that worsen PSNR and SSIM.

Abstract

from arXiv · show

Face super-resolution (FSR), also known as face hallucination, which is aimed at enhancing the resolution of low-resolution (LR) face images to generate high-resolution (HR) face images, is a domain-specific image super-resolution problem. Recently, FSR has received considerable attention and witnessed dazzling advances with the development of deep learning techniques. To date, few summaries of the studies on the deep learning-based FSR are available. In this survey, we present a comprehensive review of deep learning-based FSR methods in a systematic manner. First, we summarize the problem formulation of FSR and introduce popular assessment metrics and loss functions. Second, we elaborate on the facial characteristics and popular datasets used in FSR. Third, we roughly categorize existing methods according to the utilization of facial characteristics. In each category, we start with a general description of design principles, then present an overview of representative approaches, and then discuss the pros and cons among them. Fourth, we evaluate the performance of some state-of-the-art methods. Fifth, joint FSR and other tasks, and FSR-related applications are roughly introduced. Finally, we envision the prospects of further technological advancement in this field. A curated list of papers and resources to face super-resolution are available at \url{https://github.com/junjun-jiang/Face-Hallucination-Benchmark}

1 INTRODUCTION

FSR recovers high-resolution face images from low-resolution inputs, addressing poor image quality caused by imaging limitations. This survey organizes deep learning-based FSR methods, reviews their evaluation and applications, and discusses future directions.

  • 1 INTRODUCTION: FSR recovers high-resolution face images from low-resolution face images and can restore details in low-quality inputs.The problem is motivated by low-quality face images arising from physical imaging systems and imaging conditions.
  • 1 INTRODUCTION: The survey reviews problem definitions, assessment metrics, loss functions, facial characteristics, datasets, methods, applications, and future challenges.Its coverage includes both deep learning-based FSR techniques and related joint tasks.
  • 1 INTRODUCTION: The survey compares state-of-the-art performance and covers methods that jointly perform FSR with other tasks and FSR-related applications.It also discusses limitations and future prospects for the field.
  • 1 INTRODUCTION: Existing FSR methods are categorized into five major groups according to their use of facial characteristics.The categories are general, prior-guided, attribute-constrained, identity-preserving, and reference FSR methods.
  • 1 INTRODUCTION: Each major category is further divided according to network architecture or the specific utilization of facial characteristics.The taxonomy is intended to avoid exhaustive enumeration while incorporating facial information into the organization of methods.

2 BACKGROUND

This section formulates FSR as recovering an HR face from an observed LR image, reviews degradation models, and explains how losses and metrics balance pixel fidelity, structure, and perceptual quality.

  • 2.1 Problem Definition: FSR recovers a super-resolved face from an observed LR image by approximating the inverse of an image degradation model.The degradation model may include blurring, downsampling, and noise; the super-resolution model produces ISR from ILR.
  • 2.1 Problem Definition: Synthetic training pairs model degradation because real-world degradation parameters are unavailable and only the LR image is observed.Simple downsampling is insufficient, so more realistic models combine downsampling, blur, noise, and compression.
  • 2.2 Assessment Metrics and Loss Functions: Loss functions guide training, while assessment metrics evaluate reconstruction quality after the network is trained.Common losses include pixel-wise, SSIM, perceptual, adversarial, and cycle-consistency losses; evaluations include objective and subjective measures.
  • 2.2.1 Image Quality Assessment: PSNR and SSIM quantify pixel or structural similarity, whereas LPIPS, FID, NIQE, and MOS assess perceptual, distributional, no-reference, or human-rated quality.PSNR can disagree with human perception because it focuses on pixel distances; LPIPS is reported as more aligned with human judgment.
  • 2.2.2 Loss Functions: Perceptual loss usually improves visual pleasantness but lowers PSNR, while adversarial loss can introduce artificial details and worsen PSNR and SSIM despite improving visual quality and FID.Adversarial training is also described as unstable and susceptible to model collapse; Wasserstein GAN variants are proposed to alleviate training difficulties.

3 CHARACTERISTICS OF FACE IMAGES

Face images contain structural, semantic, and identity characteristics that constrain face super-resolution. These characteristics and associated datasets provide information for recovering plausible facial details and preserving identity.

  • Facial structural priors: Landmark availability varies across datasets, from five landmarks in CelebA to 194 in Helen.
  • Facial structural priors: Structural priors include facial landmarks, heatmaps, and parsing maps that describe facial components and their arrangement.Landmarks locate key points, heatmaps represent landmark probabilities, and parsing maps segment components such as eyes, nose, mouth, skin, ears, and hair.
  • Facial structural priors: Structural prior information can regularize or guide FSR models toward more reasonable target face images.
  • Facial attributes: Attributes such as gender and hair color provide semantic information that can constrain recovered attributes and facial details.Attributes can help address one-to-many LR-to-HR mappings, such as recovering whether a person wears eyeglasses.
  • Identity information: Identity information is used to maintain consistency between super-resolved and ground-truth faces and to support face recognition.
  • Datasets: Public face datasets differ in sample counts and available facial characteristics, while other datasets require generating LR images with a degradation model for FSR use.

4 FSR METHODS

Deep learning-based FSR methods either design efficient networks without facial characteristics or explicitly exploit facial characteristics and reference images to assist restoration.

  • Existing deep learning-based FSR methods explore network architecture, facial characteristics, and additional high-quality reference faces for restoration.

4.1 General FSR

General FSR methods focus on network design without using facial characteristics, spanning CNN, GAN, reinforcement-learning, and ensemble approaches. Their trade-offs involve fidelity, perceptual detail, computational cost, and robustness to real-world degradation.

  • Overview: General FSR methods design efficient networks without using facial characteristics, while basic CNN methods include global, local, and mixed processing strategies.
  • CNN-based methods: Global methods recover faces as wholes, local methods process facial components or patches, and mixed methods combine global structure with local detail recovery.
  • CNN-based methods: Wavelet-domain methods aim to avoid over-smoothed results by super-resolving textural and contextual information in wavelet coefficients.
  • GAN-based methods: GAN-based methods seek realistic-looking faces with more detail than pixel-wise CNN outputs, but their adversarial training can produce artificial details and lower PSNR or SSIM.
  • GAN-based methods: Unpaired real-world FSR methods learn degradation from unpaired LR and HR data before training low-to-high restoration networks.LRGAN uses a high-to-low GAN to generate realistic LR images and create paired data for the restoration generator.
  • Generative-prior methods: Generative-prior methods use pretrained face generators; PULSE searches latent space for an HR face whose downsampled output matches the LR input.Later methods embed pretrained generators in new networks to better preserve spatial information than randomly sampled low-dimensional latent codes.
  • Reinforcement-learning methods: Reinforcement-learning methods recurrently locate and enhance facial regions while modeling global dependencies among facial parts.
  • Comparative discussion: CNN methods generally favor higher PSNR and smoother outputs, whereas GAN methods favor perceptual detail; ensemble and reinforcement-learning methods increase memory, computation, or training cost.

4.2 Prior-guided FSR

Prior-guided FSR uses facial structure information to improve reconstruction and is organized by when prior estimation occurs relative to super-resolution. The four designs trade estimation accuracy, inference participation, and computational cost.

  • Taxonomy: Prior-guided FSR extracts facial prior information and uses it to facilitate face reconstruction, with methods classified as pre-, parallel-, in-, or post-prior.
  • Pre-prior methods: Pre-prior methods estimate structure from LR images before FSR, using priors such as heatmaps, landmarks, or parsing maps as model inputs.
  • Pre-prior methods: Landmark-based pre-prior methods face inaccurate landmarks on tiny LR images, motivating methods that use facial parsing maps instead.The cited example specifically identifies 16×16 LR images as especially challenging for accurate landmark extraction.
  • Parallel-prior methods: Parallel-prior methods jointly estimate priors and perform super-resolution, sharing features or feeding prior-branch features into the super-resolution branch.JASRNet uses a shared encoder, while ATSENet additionally sends prior-estimation features to its feature fusion unit.
  • In-prior methods: In-prior methods first enhance LR images, extract priors from intermediate results, and use those priors in subsequent reconstruction.
  • Post-prior methods: Post-prior methods extract priors from SR results and use them to design training losses rather than as inference inputs.
  • Discussion: Prior-guided methods require ground-truth structural priors during training, while all except post-prior methods estimate priors during testing.
  • Discussion: Pre-prior estimation is limited by degraded LR information, in-prior procedures increase memory and computation, and post-prior methods cannot use input-specific priors during inference.

4.3 Attribute-constrained FSR

Attribute-constrained FSR uses semantic facial attributes to guide restoration, either by providing attributes directly or estimating them when unavailable. Methods differ in how they integrate attributes, and estimated-attribute approaches broaden applicability but make attribute accuracy difficult to guarantee.

  • Facial attributes such as eyewear provide semantic knowledge that can guide face super-resolution.
  • Given-attribute methods condition FSR on supplied attributes, which may be concatenated directly with the LR image or its extracted features.
  • AACNN instead extracts attribute features to enhance FSR because attributes and LR images belong to different domains.
  • Estimated-attribute methods predict attributes and use attribute-based loss, avoiding the requirement that all attributes be provided.
  • Given-attribute methods are restricted by missing attributes, whereas estimated-attribute methods apply more broadly but may estimate attributes inaccurately in practice.

4.4 Identity-preserving FSR

Identity-preserving FSR seeks to keep the person consistent after super-resolution and support face recognition. Methods use pretrained recognition networks or pairwise identity relationships, with the latter reducing dependence on costly identity-labeled datasets.

  • Identity-preserving FSR maintains identity consistency between super-resolved and target faces while improving downstream face recognition.
  • Recognition-based methods define identity loss using pretrained face-recognition networks, sometimes operating on normalized features, residual maps, or multiple scales.
  • SPGAN injects pretrained identity information into its discriminator at multiple scales and uses attention-based identity loss.
  • Pairwise methods such as SiGAN use same-identity and different-identity relationships to construct contrastive identity losses from weak labels.
  • IADFH uses a three-way discriminator that distinguishes fake, genuine, and imposter face pairs.
  • Recognition-based methods require costly well-labeled datasets, whereas pairwise methods avoid that requirement and therefore have wider application potential.

4.5 Reference FSR

Reference FSR supplements an LR face with high-quality images to recover identity-aware details, using single-face, multi-face, or dictionary-guided references. Reference availability and alignment determine applicability, while dictionary guidance removes the same-identity requirement at the cost of harder reconstruction.

  • Reference FSR uses high-quality face images to provide identity-aware details for restoring an LR face.
  • Single-face guided methods align one same-identity reference with the LR face before jointly reconstructing the super-resolved result.
  • Multi-face guided methods exploit several same-identity references to provide complementary information for FSR.
  • ASFFNet selects a pose- and expression-similar reference, applies weighted least-square alignment and AdaIN, then adaptively fuses reference and LR features.
  • Dictionary-guided methods use facial components from references without requiring identity consistency between the reference and LR face.
  • Same-identity reference methods facilitate restoration but are limited when references are unavailable, while dictionary guidance broadens application and increases reconstruction difficulty.

4.6 Experiments and Analysis

The survey evaluates FSR methods across datasets, scales, and metrics, finding that strong general super-resolution backbones can outperform specialized FSR methods on PSNR and SSIM. Facial characteristics may improve perceptual quality despite weaker distortion metrics, while reference methods require separate comparisons because they use additional high-quality inputs.

  • Experimental setting: The evaluation compares PSNR, SSIM, and LPIPS on CelebA, VGGFace2, and CASIA-WebFace at ×4, ×8, and ×16 scales.
  • Experimental setting: CelebA experiments use 168,854 training images and 1,000 test images, resized to 128×128 and degraded using the survey’s degradation model.
  • Comparison results: Retrained RCAN and NLSN can outperform the best FSR methods on PSNR and SSIM, while SPARNet performs best among the listed FSR methods.
  • Comparison results: General-image models trained on face images outperform their general-image pretrained versions, so direct pretrained-model comparisons are unsuitable for FSR.
  • Comparison results: Facial-characteristic methods may perform poorly on PSNR and SSIM yet well on LPIPS, indicating stronger recovery of contours and discriminative details under perceptual evaluation.
  • Comparison results: The relative effectiveness of different facial characteristics is difficult to determine because methods use different backbone networks.
  • Reference FSR results: Reference FSR is evaluated separately because it receives additional high-quality reference images unavailable to the other methods.
  • Reference FSR results: ASFFNet outperforms the compared single-face guided methods, with alignment, illumination handling, and adaptive feature fusion identified as contributing factors.

4.7 Joint FSR and Other Tasks

Joint FSR methods address real-world degradations and facial transformations that make isolated super-resolution inadequate. The survey covers occlusion, blur, illumination, alignment, and non-frontal poses through task-integrated or sequential approaches.

  • Overview: Joint methods target face images affected simultaneously by low resolution and factors such as occlusion, shadow, blur, abnormal illumination, or pose.These conditions motivate combining FSR with completion, deblurring, illumination correction, alignment, or frontalization.
  • Occlusion and shadow: Sequentially completing occluded regions before super-resolution can accumulate errors and produce large artifacts.FCSR-GAN instead combines a pretrained face completion model with super-resolution.
  • Occlusion and shadow: Pro-UIGAN progressively recovers occluded LR faces by using facial landmarks to capture geometric prior.
  • Blur: SCGAN jointly deblurs and super-resolves blurry LR faces, while later methods incorporate facial parsing or prior information to restore high-frequency details.
  • Abnormal illumination: SeLENet decomposes faces into normal appearance, albedo, and lighting components before reconstructing a neutral-light face.
  • Alignment and frontalization: Alignment methods insert spatial transformer networks into generators, while integrated frontalization methods address the sharp quality drop on non-frontal faces.Alignment targets the pose of HR references, whereas frontalization recovers frontal faces from non-frontal LR inputs.

4.8 Related Applications

Related FSR applications extend reconstruction beyond single images to video, old-photo restoration, audio-guided reconstruction, and 3D face data. These settings exploit temporal, cross-modal, latent-space, or geometric information.

  • Scope: Related applications include face video super-resolution, old-photo restoration, audio-guided FSR, and 3D FSR.
  • Face video super-resolution: Video methods exploit inter-frame correlation by fusing multiple frames or estimating a central frame from per-frame super-resolved outputs.
  • Old-photo restoration: Old-photo restoration addresses complex, difficult-to-simulate degradation by mapping real old images toward artificial LR images before applying FSR methods.BOPBL performs this transformation in latent space rather than image space.
  • Audio-guided FSR: The first audio-guided FSR method encodes image and audio separately, fuses their representations, and feeds the result to a generator.The approach exploits face-related information carried by audio but faces challenges from cross-modal differences.
  • 3D FSR: 3D FSR methods use low-resolution 3D scans or depth information to provide features for face reconstruction and recognition.

5 CONCLUSION AND FUTURE DIRECTIONS

The survey organizes deep learning-based FSR into five facial-characteristic categories, reviews representative methods and performance, and identifies open challenges in network design, facial priors, evaluation, real-world degradation, discrimination, and multimodal inputs.

  • Conclusion: The taxonomy divides FSR into general, prior-guided, attribute-constrained, identity-preserving, and reference methods, with architecture- or characteristic-based subcategories.General methods include CNN-, GAN-, reinforcement-learning-, and ensemble-learning-based approaches.
  • Conclusion: The survey provides a pedagogical synthesis of typical FSR methods while acknowledging possible omissions in this rapidly expanding field.
  • Design of Network: Lighter networks with lower computational cost remain necessary because large models are difficult to deploy in real-world applications.The survey also identifies backbone design as crucial to performance, especially for PSNR and SSIM.
  • Exploitation of Facial Prior: Future FSR systems should better discover and represent facial priors ranging from identity and attributes to landmarks, heatmaps, parsing maps, and 3D models.
  • Metrics and Loss Functions: Pixel-wise losses favor high PSNR and SSIM, whereas perceptual and adversarial losses favor visually pleasant results measured by LPIPS and FID.The survey emphasizes that metric choice guides optimization and affects final outputs.
  • Discriminate FSR: FSR should pursue discriminative HR faces or features that improve face recognition, using weakly supervised positive and negative sample information.
  • Real-world FSR: Synthetic degradation creates a gap with real-world data, and assuming a shared degradation across real LR faces remains an urgent limitation.Real images can have substantially different degradation processes.
  • Multi-modal FSR: Audio-guided and hyperspectral FSR motivate broader use of multimodal information such as audio, depth, and near infrared.
Loading 2101.03749v2…