Source-linked AI summary
The Elements of End-to-end Deep Face Recognition: A Survey of Recent Advances
Hang Du, Hailin Shi, Dan Zeng, Xiao-Ping Zhang, Tao Mei
TL;DR
End-to-end deep face recognition depends on coordinated face detection, alignment, and representation, yet existing reviews often emphasize only representation or isolated elements. This survey systematically reviews all three stages, their evaluations and challenges, and their interactions within the holistic system. It concludes that understanding these connections is necessary for constructing high-performance systems, while particular methods remain bounded by issues such as annotation ambiguity and anchor-based detection trade-offs.
Problem
Existing surveys often focus on face representation or isolated elements, leaving the intrinsic connections among detection, alignment, representation, and the holistic system insufficiently reviewed.
Method
The survey reviews recent methods for face detection, face alignment, and face representation across algorithms, metrics, datasets, comparisons, challenges, future directions, and inter-element effects.
Results
The survey provides a comprehensive, multi-perspective account of the three elements and discusses their relationships within the end-to-end framework.
Takeaways & Limitations
Understanding every element and its intrinsic connections supports selecting methods and establishing high-performance end-to-end face recognition systems.
Takeaways & Limitations
Facial landmark definitions can be ambiguous, occluded regions can produce imprecise annotations, and anchor-based detectors require dataset-specific tuning while increasing computation and imbalance.
Abstract
from arXiv · showhide
Face recognition is one of the most popular and long-standing topics in computer vision. With the recent development of deep learning techniques and large-scale datasets, deep face recognition has made remarkable progress and been widely used in many real-world applications. Given a natural image or video frame as input, an end-to-end deep face recognition system outputs the face feature for recognition. To achieve this, a typical end-to-end system is built with three key elements: face detection, face alignment, and face representation. The face detection locates faces in the image or frame. Then, the face alignment is proceeded to calibrate the faces to the canonical view and crop them with a normalized pixel size. Finally, in the stage of face representation, the discriminative features are extracted from the aligned face for recognition. Nowadays, all of the three elements are fulfilled by the technique of deep convolutional neural network. In this survey article, we present a comprehensive review about the recent advance of each element. To start with, we present an overview of the end-to-end deep face recognition. Then, we review the advance of each element, respectively, covering many aspects such as the to-date algorithm designs, evaluation metrics, datasets, performance comparison, existing challenges, and promising directions for future research. Also, we provide a detailed discussion about the effect of each element on its subsequent elements and the holistic system. Through this survey, we wish to bring contributions in two aspects: first, readers can conveniently identify the methods which are quite strong-baseline style in the subcategory for further exploration; second, one can also employ suitable methods for establishing a state-of-the-art end-to-end face recognition system from scratch.
1 INTRODUCTION
The survey examines 2D end-to-end deep face recognition as a pipeline of face detection, face alignment, and face representation, emphasizing their interactions and recent advances. It organizes methods, evaluations, datasets, performance comparisons, challenges, and future directions to support systematic understanding and system construction.
- Scope and pipeline: End-to-end deep face recognition takes images or video frames as input and outputs deep face features through detection, alignment, and representation.Detection locates faces, alignment normalizes them to a canonical view, and representation extracts discriminative features.
- Three elements: Face detection localizes face regions, face alignment calibrates detected faces to a canonical view, and face representation extracts features for recognition.These stages form the standard end-to-end pipeline.
- Element interactions: Recognition performance depends on all three elements, so weakness in any detection, alignment, or representation component can bottleneck the final system.The survey therefore analyzes the elements together rather than treating representation in isolation.
- Recent advances: The survey emphasizes recent work, noting that publications from 2018–2020 exceeded those published before 2018.Its stated goal is to provide an up-to-date review and identify future research trends.
- Survey coverage: The survey reviews algorithm designs, evaluation metrics, datasets, performance comparisons, challenges, and promising directions for each element.It also discusses how each element affects subsequent stages and the holistic system.
2 OVERVIEW
The overview presents end-to-end recognition as a sequential pipeline and frames the survey around technical reviews of detection, alignment, and representation. It highlights multi-perspective categorization and the importance of understanding connections among the stages.
- Pipeline overview: The standard pipeline localizes faces, normalizes detected faces to a canonical layout, and extracts discriminative features used for identity similarity decisions.The extracted features support deciding whether faces belong to the same identity.
- Survey structure: The survey’s technical sections separately review face detection, face alignment, and face representation, while other sections provide overall introduction and discussion.This organization is illustrated as functional contents on the left and technical contents on the right.
- Face detection: Face detection methods are categorized across multi-stage, single-stage, anchor-based, anchor-free, multi-task, CPU real-time, and problem-oriented dimensions.The categories are intended to capture development from multiple perspectives rather than form mutually exclusive sets.
- Survey rationale: The survey uses comprehensive categorization because face detection has an advanced state of the art and readers need to understand its development and future trend.Its categorization is presented as useful for interpreting the field’s progress.
- Face alignment: Face alignment calibrates detected faces using facial landmarks or landmark-free methods to generate aligned images for recognition.Landmarks commonly provide references for similarity or affine transformations, while the survey also includes landmark-free alignment.
- Face representation: The face representation review organizes deep feature methods by network architecture and training supervision.The survey distinguishes its broader end-to-end perspective from reviews focused mainly on representation or training loss.
3 FACE DETECTION
This section categorizes deep face detection methods from multiple perspectives, reviews datasets and performance, and examines detection’s effects on later recognition stages. It also highlights accuracy–efficiency trade-offs and challenges affecting practical deployment.
- Overview: Face detection locates face regions and serves as the first input to face alignment and face representation in an end-to-end recognition system.The survey frames detection as the entry point whose outputs affect all subsequent processing.
- Method categorization: The survey organizes methods into seven nonexclusive categories: multi-stage, single-stage, anchor-based, anchor-free, multi-task, CPU real-time, and problem-oriented.The categorization is intended to present deep face detection methods from multiple perspectives.
- Detector architectures: Multi-stage detectors generate candidate boxes and refine them through additional stages, whereas single-stage detectors directly classify candidates and regress boxes from feature maps.Single-stage methods avoid a proposal stage and are generally faster, while multi-stage methods use proposal-to-refine processing.
- Anchor mechanisms: Anchor-based detectors dominate current performance, while anchor-free methods remove preset anchors and aim for better generalization capacity.Anchor-based methods rely on anchor allocation and matching; anchor-free methods include DenseBox, UnitBox, and CenterFace.
- Challenges: Detection faces challenges from scale, occlusion, resolution, pose, illumination, diverse backgrounds, and the trade-off between resizing for speed and retaining tiny-face recall.Anchor settings can also limit generalization and dense anchors can increase computational cost and positive–negative imbalance.
- Multi-task learning: Multi-task detectors improve feature representation and robustness by jointly learning face detection with landmarks or other facial tasks.Examples include MTCNN, HyperFace, and RetinaFace, which combine detection with related supervision.
- Practical deployment: Real-world deployment requires balancing detection accuracy and efficiency, especially because state-of-the-art detectors may be insufficiently efficient on non-GPU devices.The survey identifies CPU real-time detectors that reach at least 20 FPS on a single CPU with VGA-resolution inputs.
- Effect on subsequent elements: Detection quality directly affects later alignment and recognition: inaccurate boxes degrade landmark localization, while more robust detectors can improve recognition accuracy.The survey therefore treats detection as an essential consideration when building a high-performance end-to-end system.
4 FACE ALIGNMENT
Face alignment calibrates detected faces to a canonical layout before recognition, using landmark-based or landmark-free deep methods. Landmark localization approaches include coordinate regression, heatmap regression, and 3D model fitting, while alignment quality affects downstream representation.
- Face alignment calibrates detected faces to a canonical layout and normalized size to facilitate downstream recognition and analysis.
- Landmark-based alignment uses facial landmarks as references and includes coordinate regression, heatmap regression, and 3D model fitting.
- Landmark-based alignment: Coordinate regression learns a nonlinear mapping from face images to landmark coordinates, often refining predictions progressively with cascaded regression or recurrent networks.
- Landmark-based alignment: Heatmap regression predicts likelihood maps for landmarks, with newer architectures maintaining facial structure information and achieving leading localization performance.
- Landmark-based alignment: Ambiguous or occluded landmarks produce inconsistent annotations, motivating geometric, probabilistic, and uncertainty-based methods that can identify alignment failures.
- Landmark-based alignment: 3D model fitting reconstructs face shape from 2D images and supports alignment under large poses, while diverse facial shapes can still produce inaccurate 2D landmarks.
- Landmark-free alignment: Landmark-free methods integrate geometric warping into deep networks and can jointly optimize alignment with subsequent face representation.
- Evaluation and influence: Facial landmark localization is commonly evaluated with normalized mean error, while comparisons cover datasets including 300-W, WFLW-All, ALFW-Full, and COFW.
5 FACE REPRESENTATION
Face representation maps aligned faces into a feature space for verification and identification. The survey organizes learning methods by architecture and supervision, then reviews domain, modality, video, and task-specific challenges.
- Overview: Face representation maps aligned images into a feature space where same-identity features are close and different-identity features are far apart.It supports face verification and face identification by measuring similarity between face images.
- Organization: The survey categorizes representation learning by network architecture and training supervision.Architectures include general and specialized designs, while supervision includes classification, embedding, hybrid, and semi-supervised methods.
- Training supervision: Hybrid methods combine classification and feature-embedding supervision to improve training on long-tail data.Classification performs well on head data, whereas feature embedding provides complementary supervision on tail data.
- Training supervision: Semi-supervised methods exploit labeled and unlabeled data but must address overlapping identities and label noise in pseudo-labeled samples.One approach separates disjoint and overlapping classes using out-of-distribution detection and uses uncertainty-based loss to reduce pseudo-label noise.
- Challenges and tasks: Cross-domain, cross-modality, video, verification, and identification settings introduce distinct representation-learning challenges.Examples include domain gaps, limited heterogeneous data, unequal frame quality, temporal information, and open-set gallery membership.
- Training supervision: Classification training on large-scale datasets can approach open-set face recognition and has dominated recent state-of-the-art performance.The survey relates this trend to the increasing scale of public face datasets.
6 DISCUSSION AND CONCLUSION
The survey identifies challenges across individual face-recognition elements and the holistic system, then outlines research directions addressing robustness, efficiency, interpretability, and joint modeling.
- Major challenges: Face detection, alignment, and representation face challenges involving facial and image variation, data and label distribution, and computational efficiency.Examples include pose, expression, occlusion, scale, blur, illumination, long-tail identities, and domain imbalance.
- Major challenges: The three elements ideally should be jointly modeled and optimized for end-to-end accuracy, but their different learning objectives complicate integration.Joint modeling could support global optimization and allow individual elements to benefit from upstream components.
- Face detection directions: Anchor-based detectors require carefully tuned preset anchors that can limit generalization, while anchor-free detectors need greater robustness to false positives and training instability.The survey proposes generalized anchor settings and further exploration of anchor-free detection.
- Face alignment directions: Future alignment research emphasizes robustness and efficiency, dense landmarks, video temporal information, and semi-supervised localization.These directions target varied input faces, richer alignment keypoints, difficult video conditions, and the cost of precise landmark annotation.
- Face representation directions: Future representation research targets lightweight models, video robustness, noisy-label learning, cross-domain recognition, and learning from imbalanced data.Lightweight models are especially relevant because heavy networks can be impractical on mobile or embedded devices.
- Entire-system directions: System-level directions include interpretable deep models, joint holistic-system modeling, and universal face-representation pretraining.The survey states that explainable deep face recognition remains in its infancy and that end-to-end joint learning remains an open question.
A REPRESENTATIVE SURVEYS OF FACE RECOGNITION
The paper situates its discussion within prior surveys of face recognition, summarized in a dedicated representative-survey table.
- Survey literature: Representative face-recognition surveys from the past twenty years are summarized in Table 12.The table is presented as a reference for earlier survey literature.
B.1 Single-stage and multi-stage face detectors
Single-stage detectors detect faces directly from feature maps, whereas multi-stage detectors generate proposals and refine them through additional stages.
- Single-stage detection: Single-stage face detectors perform detection directly from feature maps without a separate proposal-refinement sequence.This design is contrasted with multi-stage detection in the figure.
- Multi-stage detection: Multi-stage face detectors use a proposal stage to generate candidate regions and one or more stages to refine those candidates.The proposal-to-refinement structure distinguishes them from single-stage detectors.
B.2 Performance comparison of CPU real-time face detection methods
CPU real-time face-detection methods are compared by running efficiency, with accuracy defined as the true positive rate at 1000 false positives on FDDB.
- Efficiency comparison: Table 13 reports the running efficiency of CPU real-time face-detection methods.The table focuses on methods designed for real-time CPU operation.
- Evaluation metric: “Accuracy (%)” denotes the true positive rate at 1000 false positives on FDDB.This metric definition specifies how detection accuracy is reported in the comparison.
C.1 Hourglass network for facial landmark localization
The hourglass network combines bottom-up and top-down processing in stacked bottleneck blocks with intermediate supervision for facial landmark localization.
- Hourglass is a bottom-up and top-down architecture built from stacked bottleneck blocks with intermediate supervision.
- The stacked hourglass illustration depicts consistent feature-channel width across each hourglass structure, with boxes representing residual modules.
C.2 3D model fitting for facial landmark localization
3D model fitting methods estimate a face’s three-dimensional shape and projection through cascaded regression, while landmark-free methods integrate alignment into deep networks.
- Cascaded regression with a dense 3D Morphable Model estimates the 3D face shape for facial landmark localization.
- Landmark-free alignment integrates the transformation into deep convolutional networks and outputs aligned faces without facial landmarks.
D FACE REPRESENTATION
Face representation training commonly uses classification or feature-embedding supervision, while testing centers on verification and identification. As face datasets grow, classification-based supervision becomes the most prevalent scheme because large-scale closed-set training can approach open-set recognition.
- Training and testing: Face representation training widely uses classification and feature-embedding supervision, with hybrid methods also appearing in the publication trend.
- Training and testing: Face verification and face identification are the two major tasks during testing.
- Publication trend: Classification-based training publications exceed feature-embedding and hybrid methods as available face data grow.
- Publication trend: Large-scale closed-set classification training enables approaches to the open-set face-recognition scenario.