Source-linked AI summary
End-to-end 3D face reconstruction with deep neural networks
Pengfei Dou, Shishir K. Shah, Ioannis A. Kakadiaris
TL;DR
Monocular 3D facial shape reconstruction from a single 2D image is difficult because depth is missing and camera parameters are often unavailable. UH-E2FAR uses an end-to-end DNN with multi-task learning and fusion-CNN components to predict neutral and expressive face parameters from one RGB image. Extensive experiments report improved reconstruction accuracy over comparison methods, while plausible expressions require sampled expression parameters estimated from multiple face databases.
Problem
Single-image monocular 3D facial reconstruction is challenging because image formation loses depth information and camera intrinsic parameters are usually unknown.
Method
UH-E2FAR predicts identity and expression parameters from a single RGB image using an end-to-end DNN with multi-task learning and fused intermediate features.
Results
Extensive experiments report considerable improvements in reconstruction accuracy, including better performance at key facial regions and increased face identification accuracy compared with prior methods.
Takeaways & Limitations
The end-to-end framework simplifies training and deployment by removing iterative updates and the need for initial or intermediate 3D-face rendering.
Takeaways & Limitations
Random expression parameters can generate non-plausible 3D facial shapes, so the method samples parameters estimated from multiple 2D face databases to ensure plausible expressions.
Abstract
from arXiv · showhide
Monocular 3D facial shape reconstruction from a single 2D facial image has been an active research area due to its wide applications. Inspired by the success of deep neural networks (DNN), we propose a DNN-based approach for End-to-End 3D FAce Reconstruction (UH-E2FAR) from a single 2D image. Different from recent works that reconstruct and refine the 3D face in an iterative manner using both an RGB image and an initial 3D facial shape rendering, our DNN model is end-to-end, and thus the complicated 3D rendering process can be avoided. Moreover, we integrate in the DNN architecture two components, namely a multi-task loss function and a fusion convolutional neural network (CNN) to improve facial expression reconstruction. With the multi-task loss function, 3D face reconstruction is divided into neutral 3D facial shape reconstruction and expressive 3D facial shape reconstruction. The neutral 3D facial shape is class-specific. Therefore, higher layer features are useful. In comparison, the expressive 3D facial shape favors lower or intermediate layer features. With the fusion-CNN, features from different intermediate layers are fused and transformed for predicting the 3D expressive facial shape. Through extensive experiments, we demonstrate the superiority of our end-to-end framework in improving the accuracy of 3D face reconstruction.
1. Introduction
Monocular 3D facial reconstruction from one 2D image is difficult because depth is lost and camera parameters are usually unavailable. UH-E2FAR addresses this problem with an end-to-end DNN that predicts morphable-model parameters directly.
- 3D face models are useful for handling pose, expression, and illumination variations in facial images.
- 3D cameras are costly and have limited effective sensing ranges, constraining their practical applicability.
- Monocular reconstruction is challenging because 2D image acquisition loses depth information and the solution space is large.
- Morphable models constrain the solution space by representing a face with weights over basis shapes.
- Earlier approaches either incurred high computation costs under analysis-by-synthesis or produced coarse reconstructions using sparse facial landmarks.
- UH-E2FAR predicts morphable-model parameters from a single 2D image through one end-to-end forward operation instead of iterative reconstruction.
2. Related work
Related work has used morphable models, feature points, manifold learning, radial-basis networks, and iterative regressors for monocular 3D face reconstruction. These approaches have faced computational, alignment, facial-detail, and non-frontal-image limitations.
- 3D morphable models represent faces from single or multiple images, but analysis-by-synthesis fitting is computationally expensive and requires manual initialization alignment.
- Feature-point-based morphable-model estimation improves efficiency but uses sparse image information and remains dependent on facial landmark accuracy.
- Some prior methods cannot handle non-frontal facial images, while later cascade and DNN approaches estimate and update morphable-model parameters iteratively.
3. Method
The method predicts identity and expression parameters from a single facial image using a 3D facial shape subspace and an end-to-end deep neural network. It combines multi-task losses with fused intermediate features and trains against reconstructed 3D faces.
- 3. Method: The 3D face is represented as a mean shape plus neutral-shape and expression basis components weighted by identity and expression parameters.The network predicts these parameters so the reconstructed face minimizes its difference from the ground truth.
- 3. Method: The method uses BFM for detailed 3D face modeling and AFM as a lightweight mesh model for face analysis and pose normalization.BFM has 53,490 vertices and 160,470 triangular faces, while AFM has 7,597 vertices and 14,912 triangular faces.
- 3.1. Deep neural network architecture: The network is based on VGG-Face and adds a fusion-CNN for expression parameters plus a multi-task loss for identity and expression prediction.The fusion-CNN combines intermediate-layer features, while separate objectives train identity and expression outputs.
- 3.2. The end-to-end training: The input is a face-detector-localized 2D facial region cropped as a square and scaled to 180 × 180 pixels.The network outputs identity and expression vectors used with the facial shape model to reconstruct the input face.
- 3.2. The end-to-end training: Training uses real 2D images for initialization and synthetic 2D images for fine-tuning.Synthetic facial images are generated from the 3D facial shape models and their shape and blendshape bases.
- 3.2. The end-to-end training: The total training loss is a weighted sum of identity and expression reconstruction losses based on differences between predicted and ground-truth 3D faces.The vertex-wise discrepancy is measured using a sum of squared errors, with separate weights for the two losses.
- 3.3. Discussion: Unlike iterative alternatives, the model uses one RGB image end-to-end, avoiding iterative simulation during training and 3D rendering of initial or intermediate shapes during deployment.The single-image design also permits initialization with available 2D face databases.
4. Experiments
The experiments evaluate UH-E2FAR for single-image 3D face reconstruction using synthetic training data and three public databases. Across pose, illumination, and expression settings, the method generally achieves strong reconstruction and recognition performance, with fusion-CNN benefits for expressive faces.
- Experimental setup: UH-E2FAR is compared with RSNIEF, RSN, UH-2FCSL, and UH-E2FARMod for single-image 3D face reconstruction.UH-E2FARMod removes the fusion-CNN to test its contribution to expressive-face reconstruction.
- Synthetic data generation: Synthetic training data are generated from BFM and AFM 3D facial shape models, with sampled plausible expression parameters and controlled camera and illumination settings.The synthetic-data procedure addresses the lack of large-scale 3D–2D face databases.
- Datasets and metric: The evaluation uses FRGC2, BU-3DFE, and UHDB31 to test reconstruction under varying illumination, facial expressions, and facial poses, respectively.Reconstruction accuracy is measured by RMSE after rigid alignment and ICP registration.
- Reconstruction results: On UHDB31, UH-E2FAR offers the best reconstruction performance among the compared methods according to RMSE distributions, quantitative results, and spatial error comparisons.The evaluation reconstructs 3D faces from 1,638 successfully detected images.
- Reconstruction results: On FRGC2, UH-E2FAR considerably improves reconstruction accuracy over RSNIEF, RSN, and UH-2FCSL, especially around the mouth, nose, and eyes.The comparison uses RMSE distributions, mean and standard deviation, and facial-region heatmaps.
- Expressive-face results: For expressive faces, UH-E2FAR has larger RMSE than RSNIEF and RSN overall but lower inner-region error than RSNIEF and better key-region performance than UH-E2FARMod.The key regions include the mouth, nose, and eyes.
- Qualitative results: On images captured in the wild, UH-E2FAR is described as more robust for neutral faces, while its reconstructed expressions are more plausible than RSNIEF and RSN and more accurate than UH-E2FARMod.These qualitative comparisons are illustrated in Figures 11 and 12.
- Face recognition: UH-E2FAR considerably increases face-identification accuracy over UH-2FCSL, with some pose-specific rank-1 rates close to those obtained using ground-truth 3D data.Recognition uses manually annotated feature points for pose estimation and an image-gradient facial representation.
5. Conclusions
The paper concludes that UH-E2FAR simplifies monocular 3D face reconstruction through end-to-end inference while retaining reconstruction performance. Its multi-task loss and fusion-CNN support separate neutral and expressive reconstruction, and real-image initialization improves robustness to pose and illumination.
- Conclusions: UH-E2FAR replaces iterative model-parameter updating and additional 3D rendering inputs with end-to-end inference from a single image.The framework does not require initial model parameters or 3D shape rendering as additional DNN input.
- Conclusions: The multi-task loss separates neutral and expressive 3D facial shape reconstruction, while the fusion-CNN combines intermediate features for expressive-face prediction.Different neural-layer types are trained for the two reconstruction subtasks within one DNN model.
- Conclusions: Experiments indicate that the framework simplification does not compromise 3D face reconstruction performance and that real facial images can improve robustness to pose and complex illumination.The paper reports that UH-E2FAR outperforms state-of-the-art approaches, although the supplied passage truncates the final comparison.