PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Face Recognition: A Survey

Mei Wang, Weihong Deng

arXiv:1804.06655v9cs.CV

TL;DR

Face recognition needed methods that could handle unconstrained facial variation and increasingly demanding evaluation settings. This survey synthesizes deep FR architectures, losses, processing methods, databases, protocols, and application scenes, reporting rapid benchmark gains alongside persistent dataset and scenario challenges. Its review organizes face processing into augmentation and normalization, and highlights technical issues for future research.

  • Problem

    Face recognition remains affected by domain shifts, dataset biases, difficult scenarios, and variations including pose, illumination, expression, and occlusion.

  • Method

    The survey reviews architectures and loss functions, categorizes face processing as one-to-many augmentation or many-to-one normalization, and compares databases, protocols, metrics, and application scenes.

  • Results

    DeepFace reached 97.35% on LFW versus 97.53% for humans, while LFW performance later rose above 99.80% within three years.

  • Takeaways & Limitations

    Deep FR has reshaped algorithm design, datasets, application scenarios, and evaluation protocols, while difficult scenario-specific problems remain active research challenges.

Abstract

from arXiv · show

Deep learning applies multiple processing layers to learn representations of data with multiple levels of feature extraction. This emerging technique has reshaped the research landscape of face recognition (FR) since 2014, launched by the breakthroughs of DeepFace and DeepID. Since then, deep learning technique, characterized by the hierarchical architecture to stitch together pixels into invariant face representation, has dramatically improved the state-of-the-art performance and fostered successful real-world applications. In this survey, we provide a comprehensive review of the recent developments on deep FR, covering broad topics on algorithm designs, databases, protocols, and application scenes. First, we summarize different network architectures and loss functions proposed in the rapid evolution of the deep FR methods. Second, the related face processing methods are categorized into two classes: "one-to-many augmentation" and "many-to-one normalization". Then, we summarize and compare the commonly used databases for both model training and evaluation. Third, we review miscellaneous scenes in deep FR, such as cross-factor, heterogenous, multiple-media and industrial scenes. Finally, the technical challenges and several promising directions are highlighted.

I. INTRODUCTION

Deep learning reshaped face recognition after 2014 by learning hierarchical, increasingly invariant representations and rapidly improving benchmark performance. This survey reviews deep FR architectures, losses, processing methods, databases, evaluation protocols, and challenging application scenes.

  • Motivation: Traditional face-recognition methods improved accuracy slowly and generally addressed individual variations such as lighting, pose, expression, or disguise.Their shallow representations and separate preprocessing, descriptor, and transformation improvements were insufficient for unconstrained changes.
  • Motivation: Deep learning uses cascaded processing layers to learn hierarchical representations with increasing invariance to pose, lighting, and expression.The architecture progressively transforms low-level pixels into higher-level face representations.
  • Deep-learning breakthrough: In 2014, DeepFace achieved 97.35% accuracy on LFW, approaching human performance at 97.53% under unconstrained conditions.DeepFace used a nine-layer model trained on 4 million facial images.
  • Deep-learning breakthrough: LFW accuracy rose from around 60% to above 90% as representation pipelines deepened, then reached 99.80% within three years of deep learning’s breakthrough.The survey presents this progression as a milestone in face-representation research.
  • Survey scope: The survey systematically reviews deep FR network architectures and loss functions, including Euclidean, angular or cosine-margin, and softmax-based objectives.It covers mainstream models such as DeepFace, DeepID, VGGFace, FaceNet, and VGGFace2.
  • Survey scope: It categorizes deep face processing into “one-to-many augmentation” and “many-to-one normalization,” then compares databases, protocols, and scenario-specific solutions.Covered challenges include anti-attack, cross-pose, cross-age, heterogeneous, multiple-media, and industrial recognition.

1) Face Processing:

Deep face recognition combines face processing, deep feature extraction, and matching to address unconstrained facial variations. The survey organizes face processing into augmentation and normalization, while reviewing architectures, losses, databases, and applications.

  • Face Processing: Pose, illumination, expression, and occlusion variations still affect deep face recognition performance, motivating dedicated face processing methods.
  • Face Processing: “One-to-many augmentation” generates multiple patches or images from one face so networks learn pose-invariant representations.
  • Face Processing: “Many-to-one normalization” recovers a canonical face view from one or more nonfrontal images before recognition.
  • Face Processing: The survey focuses mainly on deep face processing for pose variation because pose is widely regarded as a major challenge in automatic face recognition.
  • Architectures and Losses: Deep face recognition uses backbone or assembled networks and loss functions designed to produce separable, discriminative identity features.
  • Face Matching: Deep face recognition systems extract test-image features and use matching algorithms such as cosine or L2 distance for verification or identification.

III. NETWORK ARCHITECTURE AND TRAINING LOSS

Deep FR evolved through larger training sets, increasingly discriminative loss functions, and architectures inherited from object classification. Euclidean, triplet, center, and angular-margin approaches address feature separability and intra-class variation in different ways.

  • Training Data: Deep FR training benefits from many identities, but academic datasets remain limited to roughly 10^3–10^5 IDs.Large private datasets and GPU clusters are generally inaccessible to academic researchers.
  • Euclidean-distance-based Loss: Euclidean-distance losses embed faces so intra-class variance decreases while inter-class variance increases.Contrastive loss uses positive and negative pairs, whereas triplet loss uses anchor, positive, and negative samples with a relative-distance objective.
  • Euclidean-distance-based Loss: Contrastive loss combines identification and verification supervision in DeepID2, while later variants expand representations, early-layer supervision, or network architectures.DeepID2+ increases hidden-representation dimensions and adds early convolutional supervision; DeepID3 introduces VGGNet and GoogleNet.
  • Euclidean-distance-based Loss: Center loss and its variants reduce intra-class variation by learning class centers, but can require substantial GPU memory and balanced, sufficient data per identity.These methods address training instability associated with selecting effective samples for contrastive and triplet losses.
  • Evolution of Discriminative Loss Functions: Angular/cosine-margin losses impose stricter feature separation, while the field progressed from Euclidean losses toward angular margins, normalization, and softmax variations.The survey organizes loss functions into Euclidean-distance-based, angular/cosine-margin-based, and softmax-variation families.

2) Angular/cosine-margin-based Loss

Angular/cosine-margin losses reformulate classification boundaries to enforce larger angular or cosine separation between face features. L-Softmax introduced angular margins, while A-Softmax normalized weights and learned features on a hypersphere, with optimization and noise sensitivity remaining concerns.

  • Angular/cosine-margin-based Loss: Angular/cosine-margin losses impose larger angular or cosine separation to make difficult face samples more discriminable.Their decision boundaries modify softmax geometry through an angular margin.
  • Angular/cosine-margin-based Loss: L-Softmax introduces an angular margin through a positive integer m and uses a piece-wise function to preserve cosine monotonicity.The piece-wise construction addresses the nonmonotonicity of the cosine function.
  • Angular/cosine-margin-based Loss: L-Softmax is commonly combined with softmax because its optimization is difficult and convergence needs facilitation.The combined formulation includes a dynamic hyper-parameter λ.
  • Angular/cosine-margin-based Loss: A-Softmax normalizes weights to unit L2 norm, placing learned face features on a hypersphere manifold with an angular margin.SphereNet further adopts hyperspherical convolution and angular-margin supervision.
  • Angular/cosine-margin-based Loss: Angular/cosine-margin losses perform better on clean data but become worse than center loss and softmax in high-noise regions.The survey identifies vulnerability to noisy training data as a limitation of this loss family.

3) Softmax Loss and its Variations

Softmax variations normalize features, weights, or both to improve discriminative learning and reduce bias from sample distributions. Feature scaling is necessary because unit-normalizing both features and weights can trap softmax at a high training loss.

  • Softmax Loss and its Variations: Normalizing both features and weights to unit norm can trap softmax at a very high training-set loss.Scaling features to a fixed radius α avoids this issue.
  • Softmax Loss and its Variations: Some methods normalize weights before adding angular or cosine margins, while others normalize features to reduce softmax bias toward sample distributions.L2-softmax enforces a feature norm, and Ring loss encourages sample norms toward a learned value R.
  • Softmax Loss and its Variations: Feature and weight normalization became a common strategy for improving softmax-based face recognition models.The normalized feature and weight formulation is used to modify losses such as softmax.
  • Softmax Loss and its Variations: Mainstream deep FR architectures followed object-classification networks and evolved rapidly from AlexNet toward SENet.The survey presents influential architectures in chronological order.

1) Backbone Network

Deep FR backbones largely inherit the rapid evolution of object-classification architectures, while specialized designs address deployment constraints, adaptive construction, alignment, pose, and nuisance factors. These networks trade depth and capacity against efficiency and robustness across recognition settings.

  • Backbone Network: Deeper networks improve accuracy but require powerful GPUs and large memory, limiting practicality on mobile and embedded devices.This deployment constraint motivates lightweight networks.
  • Backbone Network: Deep FR backbones followed object-classification architectures from AlexNet through VGGNet, GoogleNet, ResNet, and SENet.The survey presents these architectures as the mainstream progression in deep FR.
  • Backbone Network: Lightweight networks reduce computational cost through compact representations, sparse models, fast downsampling, and bottleneck residual blocks.Light CNN uses max-feature-map activation, while MobiFace uses efficient architectural components.
  • Backbone Network: Neural architecture search explores layer operations and hyperparameters automatically instead of relying solely on manually designed architectures.The survey identifies adaptive-architecture networks as a promising direction.
  • Backbone Network: Joint alignment-recognition systems optimize detection, alignment, and recognition modules according to the recognition objective.This produces more adequate and robust inputs than separately optimizing the modules with different objectives.
  • Backbone Network: Multi-input and multi-task networks combine varied views or related tasks to address pose, illumination, expression, and other nuisance factors.Multi-view networks separate view-specific variation from common representations, while multi-task networks share lower layers and disentangle higher layers.

C. Face Matching by deep features

Deep face recognition matches extracted features using distance, classification, metric-learning, and ensemble strategies, while face processing expands or normalizes appearance variation before matching.

  • Face matching: Cosine and L2 distances measure similarity between deep features before threshold comparison or nearest-neighbor decisions.These decisions support verification and identification.
  • Face matching: Metric learning makes identity classes more separable by learning a new metric over extracted deep features.The JB model represents each feature as identity information plus intra-personal variation.
  • Face matching: The similarity score r(x1, x2) is a log likelihood ratio comparing same-identity and different-identity probabilities.The two probabilities are defined as P(x1, x2|HI) and P(x1, x2|HE), respectively.
  • Face matching: Transfer learning addresses distribution shifts by learning transferable features from labeled source data and unlabeled target data.The survey notes that adaptation is often embedded in deep models rather than applied only at the matching stage.
  • Face processing: Face processing methods evolved from data augmentation and 3D or autoencoder models toward GAN-based one-to-many augmentation and many-to-one normalization.GANs were introduced into face processing in 2017 and support both categories.
  • One-to-many augmentation: 3D reconstruction enriches training diversity by modeling pose transformations, while autoencoders generate target-pose images directly from pose codes.Iterative 3D CNNs use a secondary channel containing the previous output; autoencoders learn pose-invariant representations before decoding.

B. Many-to-One Normalization

Many-to-one normalization produces frontal faces to reduce appearance variability and facilitate alignment and comparison. Autoencoder, CNN, and GAN approaches implement this normalization through different mappings and losses.

  • Many-to-One Normalization: Many-to-one normalization produces frontal faces and reduces test-data appearance variability for easier alignment and comparison.The category includes autoencoder, CNN, and GAN models.
  • Autoencoder model: Autoencoder normalization learns pose-invariant representations and decodes frontal faces without pose codes.This differs from augmentation autoencoders, which use pose codes to generate desired target views.
  • Autoencoder model: Stacked progressive autoencoders narrow pose variation layer by layer by mapping large-pose inputs toward progressively smaller virtual poses.The final objective is a frontal-view reconstruction through several shallow autoencoders.
  • CNN model: CNN normalization directly learns a 2D mapping from non-frontal to frontal images in pixel space.LDF-Net uses a displacement field to learn pixel-shifting relationships before translation.
  • CNN model: GridFace warps pixels from the original image to a canonical image according to a computed homography matrix.Its rectification network uses local homography transformation.
  • GAN model: TP-GAN combines four landmark-located patch networks with a global encoder-decoder and uses adversarial, symmetry, and identity-preserving losses.The design targets frontal views while preserving global structure and local detail.

V. FACE DATABASES AND EVALUATION PROTOCOLS

Face databases progressed from small, controlled collections to large, diverse, unconstrained datasets that support deep-model training and task-specific evaluation. The survey also highlights noise, collection bias, demographic imbalance, and industry-oriented data engineering as major boundaries.

  • Database evolution: Face databases evolved from small, controlled, single-source collections toward large, diverse, real-world unconstrained datasets.The survey states that database development largely leads the direction of face-recognition research.
  • Large-scale training datasets: Effective deep face recognition requires sufficiently large training data; leading systems were typically trained on more than 0.5M images and 20K people.Examples include DeepFace with 4M images of 4K people and FaceNet with 200M images of 3M people.
  • Large-scale training datasets: CASIA-WebFace provided the first widely used public training dataset with 0.5M images of 10K celebrities, enabling fairer academic comparisons.Its moderate size and identity count may not reflect advanced deep-learning methods fully.
  • Dataset structure: VGGFace2 emphasizes dataset depth with many images per subject, whereas MS-Celeb-1M and Megaface emphasize breadth with many subjects and fewer images per subject.Depth targets intraclass variation; breadth expands subject coverage.
  • Data engineering: Benchmark data engineering favors industrial participants with exceptional hardware and data scale, which may hinder new academic model development.The survey links this observation to leaderboard concentration among companies.
  • Data quality: Large-scale datasets contain label noise that increases with dataset scale and is especially damaging for 10,000-class face recognition.Label-flip noise severely deteriorates model performance.
  • Data bias: Dataset biases reduce cross-dataset generalization, with VGGFace performance falling from 98.95% on LFW and 97.30% on YTF to 26%, 52%, and 85% on GBU partitions.The cited GBU values correspond to the Ugly, Bad, and Good partitions, respectively.

B. Training protocols

Face-recognition evaluation separates subject-dependent from subject-independent training and distinguishes verification, close-set identification, and open-set identification tasks. These settings require different metrics and reflect different deployment conditions.

  • Training protocols: Subject-dependent protocols include testing identities in training and treat recognition as classification over known identities.They were common in early small-scale studies and remain used by MS-Celeb-1M Challenge 2.
  • Training protocols: Subject-independent protocols use testing identities disjoint from training, making recognition more challenging and closer to practical deployment.Generalized representations are essential because test identities cannot be classified as known training identities.
  • Evaluation tasks: Verification computes one-to-one similarity, whereas identification computes one-to-many similarity against gallery identities.Closed-set identification contains only gallery identities; open-set identification also includes unknown subjects.
  • Evaluation metrics: Verification performance is reported with TAR and FAR, including TAR@10^-2FAR, TAR@10^-3FAR, TAR@10^-6FAR, and TAR@10^-9FAR across benchmarks.Customers generally prioritize TAR when FAR is very low in security certification scenarios.
  • Evaluation metrics: Close-set identification commonly uses rank-N and CMC metrics, with IJB-A/B/C emphasizing rank-1 and rank-5 recognition rates.These metrics measure whether the correct gallery identity appears within a ranked list.
  • Evaluation tasks: Open-set identification requires rejecting unknown subjects and is evaluated by DET curves relating FNIR to FPIR.The survey notes that few databases cover open-set face recognition.

D. Evaluation Scenes and Data

The survey organizes realistic face-recognition evaluation into cross-factor, heterogeneous, multiple-media, and industrial scenes, each requiring specialized datasets and methods. These settings expose limitations of benchmark performance, data availability, and real-world deployment.

  • Scene taxonomy: Realistic face-recognition scenes are divided into cross-factor, heterogeneous, multiple-media, and industrial categories.The survey associates each category with testing datasets and specialized recognition methods.
  • Cross-factor FR: Cross-factor recognition covers pose, age, makeup, and disguise variations caused by changes in facial appearance.Examples include CALFW, MORPH, CACD, FG-NET, CFP, CPLFW, and DFW.
  • Heterogeneous FR: Heterogeneous recognition matches faces across visual domains such as visible light and near-infrared or photos and sketches.CUFSF is harder than CUFS because of lighting variation and shape exaggeration.
  • Multiple-media FR: Multiple-media recognition addresses low-shot enrollment and set-to-set matching when subjects are represented by limited images, image sets, or videos.These conditions differ from conventional image-to-image recognition with many training images per subject.
  • FR in industry: Industrial face recognition must consider factors beyond accuracy, including anti-attack protection and 3D recognition.Public 3D databases remain limited in scans and subjects, hindering 3D deep FR development.
  • Deployment boundary: Despite high LFW and MegaFace accuracy, face-recognition models still struggle to meet real-world requirements, while privacy concerns constrain future data collection and annotation.The survey notes that larger target-scene datasets improve generic models only to a certain degree.

3) Makeup Face Recognition:

Makeup and other disguises create substantial facial appearance changes that complicate recognition. Reviewed approaches include synthesizing nonmakeup faces, adapting pretrained models, and translating between heterogeneous visual representations.

  • Makeup face recognition: BLAN generates nonmakeup images from makeup faces and uses the synthesized images for verification.This approach is illustrated as a bi-level adversarial network.
  • Disguise: Disguise introduces obfuscation and impersonation, respectively increasing within-person variation and reducing between-person dissimilarity.Accessories include wigs, beards, hats, mustaches, and heavy makeup.
  • Heterogeneous FR: Heterogeneous recognition methods address near-infrared, low-resolution, photo-sketch, and photo-caricature matching through adaptation, synthesis, or common-space learning.Examples include VIS-to-NIR transfer, superresolution with feature extraction, transfer learning, and image-to-image translation.

C. Multiple (or single) media Face Recognition

Multiple- and single-media recognition targets practical settings with few enrollment samples, mixed image-video representations, and difficult video frames. Methods therefore synthesize data, aggregate media, and learn representations robust to blur, pose, and occlusion.

  • Low-shot FR: Low-shot recognition must identify people from very few training samples or even one sample.Reviewed methods either synthesize training data or learn more powerful features.
  • Low-shot FR: Single-sample domain adaptation can generate varied poses with a 3D face model while handling blur, occlusion, and expression variations.The approach is presented as a deep domain-adaptation framework.
  • Set-to-set FR: Set-to-set recognition uses heterogeneous collections of images and videos as the smallest representation unit.A common strategy aggregates media representations into one representation per set before comparison.
  • Video FR: Video recognition must aggregate information across frames while handling severe blur, pose variation, and occlusion.NAN uses two memory-driven attention blocks to produce a 128-dimensional video-face representation.
  • 3D FR: 3D face recognition benefits from inherent advantages over 2D methods but remains constrained by scarce annotated 3D data and underexplored feature extraction.Many studies use one-to-many augmentation, 2D-CNN fine-tuning, or depth-data pretraining.
  • Partial FR: Partial recognition addresses arbitrary-size face patches in surveillance and embedded-vision applications.Reviewed methods compare multiscale patches or use dynamic feature matching.

4) Face Anti-attack:

Face anti-attack is necessary because spoofing and adversarial manipulation threaten deployed recognition systems. The survey places these risks within broader challenges involving security, privacy, fairness, interpretability, transferability, efficiency, and multimodal evidence fusion.

  • Face anti-attack: Face spoofing attacks present fake faces through photographs, masks, or electronic displays, motivating dedicated anti-attack methods.Examples include two-stream CNNs, depth maps, multiframe inputs, LSTMs, and spoof-noise decomposition.
  • Security challenges: Deep face-recognition systems also face adversarial, template, and digital-manipulation attacks.The survey identifies presentation attacks using realistic 3D silicone masks as a challenge for current antispoofing methods.
  • Privacy: Privacy-preserving recognition seeks recognizable biometric templates that hide private information encoded in facial images.The survey highlights gender, age, race, and genetic information as privacy concerns.
  • Evaluation limits: Benchmark saturation does not resolve cross-age, cross-pose, cross-sensor, or cross-style recognition, and demographic bias remains a required evaluation concern.The survey notes over 99.9 percent accuracy on LFW and MegaFace while fundamental challenges remain.
  • Deployment: Ubiquitous recognition remains difficult because innumerable real-world scenes lack sufficient labeled samples and general transfer solutions remain open.Deep domain adaptation has been applied to different scenes and races but does not provide a general solution.
  • Accuracy and efficiency: High-assurance applications require very low alarm rates such as 10^-9, while mobile deployment requires compact representations and compressed networks.The survey identifies simultaneous accuracy and efficiency as an unresolved challenge.
  • Evidence fusion: Face recognition alone is insufficient for some biometric and forensic tasks, including distinguishing identical twins or matching faces before and after surgery.The survey recommends consolidating multiple sources of biometric evidence.
Loading 1804.06655v9…