Source-linked AI summary
Joint Face Alignment and 3D Face Reconstruction with Application to Face Recognition
Feng Liu, Qijun Zhao, Xiaoming Liu, Dan Zeng
TL;DR
Existing methods commonly separate face alignment and 3D reconstruction and struggle with large-pose images containing invisible landmarks. This paper jointly solves both tasks through alternating 2D landmark and 3D shape regression, producing normalized and expressive 3D faces. Experiments report state-of-the-art alignment and reconstruction performance and improved recognition using reconstructed PEN faces.
Problem
Existing 3D reconstruction methods often require facial landmarks and have difficulty handling large-pose faces with self-occluded, invisible landmarks, while alignment and reconstruction are typically treated separately.
Method
The method alternates cascaded regressors for 2D landmarks and 3D shape, linked by an iteratively updated 3D-to-2D mapping matrix.
Results
The method achieves state-of-the-art 3D reconstruction and face alignment on BU3DFE, AFLW, and AFLW2000 3D, while improving baseline face recognition performance.
Takeaways & Limitations
Reconstructed pose-and-expression-normalized 3D faces provide an additional recognition feature that improves conventional 2D face matchers, particularly under pose variation.
Takeaways & Limitations
The training data assumes dense vertex correspondences across 3D shapes and anatomically correct annotations for invisible landmarks.
Abstract
from arXiv · showhide
Face alignment and 3D face reconstruction are traditionally accomplished as separated tasks. By exploring the strong correlation between 2D landmarks and 3D shapes, in contrast, we propose a joint face alignment and 3D face reconstruction method to simultaneously solve these two problems for 2D face images of arbitrary poses and expressions. This method, based on a summation model of 3D faces and cascaded regression in 2D and 3D shape spaces, iteratively and alternately applies two cascaded regressors, one for updating 2D landmarks and the other for 3D shape. The 3D shape and the landmarks are correlated via a 3D-to-2D mapping matrix, which is updated in each iteration to refine the location and visibility of 2D landmarks. Unlike existing methods, the proposed method can fully automatically generate both pose-and-expression-normalized (PEN) and expressive 3D faces and localize both visible and invisible 2D landmarks. Based on the PEN 3D faces, we devise a method to enhance face recognition accuracy across poses and expressions. Both linear and nonlinear implementations of the proposed method are presented and evaluated in this paper. Extensive experiments show that the proposed method can achieve the state-of-the-art accuracy in both face alignment and 3D face reconstruction, and benefit face recognition owing to its reconstructed PEN 3D face.
1 INTRODUCTION
The paper jointly solves face alignment and 3D face reconstruction by exploiting their shared dependence on 2D landmarks, 3D shape, pose, expression, and camera projection. Its coupled regressors support arbitrary-pose inputs, normalized or expressive 3D outputs, invisible-landmark localization, and face-recognition enhancement.
- Motivation: Existing methods usually separate face alignment and 3D reconstruction, although each task supplies information needed by the other.Landmarks help recover facial geometry, while a 3D shape and mapping function determine landmark positions and visibility.
- Method: The proposed framework jointly estimates 2D landmarks and 3D face shapes from a single image with arbitrary pose and expression.It learns two sets of regressors from paired annotated 2D images and 3D face shapes.
- Method: Cascaded landmark and shape regressors are applied alternately, with updated 3D-to-2D mappings refining landmark positions and visibility.Landmark regressors use image texture, while shape regressors use landmarks as clues for 3D-shape updates.
- Outputs: The method explicitly models expression deformation and reconstructs both pose-and-expression-normalized and expressive 3D faces.It is presented with both linear and nonlinear regression implementations.
- Applications: The approach targets improved recognition of off-angle and expressive faces using reconstructed PEN 3D faces.The paper reports state-of-the-art alignment and reconstruction performance across BU3DFE, AFLW, and AFLW2000 3D, and recognition gains on Multi-PIE and CFP.
2 PRIOR WORK
Prior work commonly treats landmark localization and 3D reconstruction separately, struggles with invisible landmarks and extreme poses, or reconstructs faces in the input pose and expression. This paper combines 2D-image and 3D-space regressions and produces pose-expression-normalized 3D faces for recognition.
- Face alignment: Classical and regression-based alignment methods estimate landmarks from shape or texture models but often degrade on non-frontal faces and do not model visibility.Multi-view templates are complicated and cannot detect self-occluded landmarks.
- Face alignment: 3D-model-based alignment handles self-occlusion better, but prior methods have limited operating ranges for large yaw angles.The cited approaches include methods failing beyond 60° and regressors operating up to 50°.
- Face alignment: Unlike prior 3D-model methods, this approach regresses both 2D image landmarks and 3D landmark coordinates, supporting view angles from -90° to 90°.The combined regressions are intended to improve landmark estimation and handling of self-occluded landmarks.
- 3D reconstruction: Single-image 3D reconstruction is ill-posed and existing shape-from-shading and 3DMM methods depend on priors or constraints while still facing invisible landmarks at large poses.Most existing reconstruction methods also require landmarks as input.
3 PROPOSED METHOD
The proposed framework jointly estimates 2D landmarks, pose-and-expression-normalized 3D shape, and expression deformation from a single face image. It alternates landmark and shape regressors, using 3D-to-2D projection to refine landmark positions and visibility.
- 3.1 Problem Formulation: The method models each 3D face as a PEN identity shape plus expression deformation.The mean frontal, neutral shape is the pose-and-expression-normalized component, while identity and expression deviations form the remaining components.
- 3.2 The Overall Procedure: Given an image, the framework initializes the 3D shape and landmarks, then iteratively updates landmarks, 3D shape, and landmark projections.Initialization uses the mean PEN shape and mean frontal-neutral landmarks placed within the detected face region.
- 3.2 The Overall Procedure: Texture-based landmark regressors estimate landmark adjustments, while shape regressors use those adjustments to predict 3D shape changes.The two regressor types are applied in alternating stages, and the updated shape is then used for landmark refinement.
- 3.6 Refining Landmarks: A dynamically estimated 3D-to-2D mapping projects the updated 3D shape back into the image and recomputes landmark visibility.The mapping matrix approximates the combined effects of pose-induced deformation and camera projection.
- 3.3 Training Data Preparation: Training uses paired images, 3D shapes, and landmarks, including anatomically correct annotations for invisible landmarks.The constructed training sets are based on BU3DFE and 300W-LP, which provide expression and pose variation.
- 3.4 Learning Landmark Regressors: The framework supports linear least-squares regressors and nonlinear regressors such as CNN-based alternatives.The paper also describes nonlinear implementations using DCNN landmark regressors and MLP shape regressors.
4 APPLICATION TO FACE RECOGNITION
The method uses reconstructed PEN 3D faces as an additional recognition feature, matching normalized 3D shapes and fusing their scores with conventional 2D face matchers.
- PEN 3D faces are generated for gallery and probe images, aligned with ICP, converted into normalized similarity scores, and fused with 2D matcher scores by a sum rule.The recognized subject is the gallery sample with the highest fused match score.
5 EXPERIMENTS
Experiments evaluate joint face alignment, 3D reconstruction, and recognition enhancement across pose, expression, ethnicity, and implementation settings. The method achieves strong reconstruction and alignment accuracy, improves face recognition, and operates in real time.
- The experiments evaluate 3D face reconstruction, face alignment, and face recognition in three complementary settings.
- 3D face reconstruction: The proposed method achieves lower average MAE than counterpart methods, with errors remaining comparatively stable as pose increases.This supports its handling of arbitrary-view faces.
- 3D face reconstruction: Across expressions, the proposed method outperforms counterparts and has lower MAE variability, with standard deviations of 0.034mm and 0.029mm versus 0.157mm and 0.179mm.The linear and nonlinear implementations respectively achieve the two lower standard deviations.
- 3D face reconstruction: Training and testing across ethnic groups reveals an other-race effect, with models generally achieving higher accuracy on the ethnicity represented in training.The Caucasian-trained model performs worst on Asian testing, while the Asian-trained model performs worse on other races than on its own race.
- Face alignment: The proposed method achieves the best face-alignment accuracy across poses on both AFLW and AFLW2000-3D, with superior NME robustness.Nonlinear regression performs better for alignment, whereas linear regression has lower reconstruction error on BU3DFE.
- Face recognition: Fusing reconstructed PEN 3D faces with 2D matchers improves recognition, especially at large yaw angles; CenterLoss gains 15.5% at ±90° and 4.0% at ±75°.LDF-Net also improves by 5.1% at ±90° and 1.4% at ±75°.
- Efficiency: The iterative procedure converges in about five linear iterations and two to three nonlinear iterations, while implementations run at approximately 26 FPS and 52 FPS respectively.These measurements indicate real-time landmark detection and 3D reconstruction under the reported hardware settings.
6 CONCLUSION
The paper jointly solves face alignment and 3D reconstruction from single images by alternating between landmark and shape regression. It reconstructs normalized and expressive faces while handling invisible landmarks and supporting real-time processing.
- The method alternates cascaded landmark and 3D-shape regressors, using each task to refine the other from a single 2D image.
- It automatically reconstructs pose-and-expression-normalized and expressive 3D faces from images with arbitrary poses and expressions.
- 3D face models help the method handle invisible and expression-deformed landmarks without requiring a separate face-alignment method.
- Experiments report effectiveness and superiority over state-of-the-art methods in both face alignment and 3D face reconstruction, with benefits for face recognition.