Source-linked AI summary
3D Face Morphable Models "In-the-Wild"
James Booth, Epameinondas Antonakos, Stylianos Ploumpis, George Trigeorgis, Yannis Panagakis, Stefanos Zafeiriou
TL;DR
In-the-wild 3D face reconstruction lacks suitable corresponding texture models because existing 3D facial datasets were captured under controlled conditions. The paper combines an identity-and-expression shape model with an in-the-wild texture model, introduces fast fitting, and evaluates the approach using a new relatively unconstrained 3D facial dataset. The authors report state-of-the-art performance, including a considerable margin over classical 3DMM approaches.
Problem
Existing 3D facial datasets are controlled, so a texture model that generalizes to in-the-wild images while corresponding to a strong shape model is difficult to construct.
Method
The paper combines a statistical shape model covering identity and expression with an in-the-wild feature-based texture model and a fast 3DMM fitting algorithm.
Results
The in-the-wild formulation achieves state-of-the-art performance on quantitative evaluations and outperforms classical 3DMM approaches by a considerable margin.
Takeaways & Limitations
The formulation enables 3DMM-based face reconstruction in in-the-wild conditions while recovering more shape detail than purely landmark-driven approaches.
Takeaways & Limitations
Shape-from-Shading comparison methods recover only 2.5D face representations and smooth approximations of facial normals because they lack a facial-surface prior.
Abstract
from arXiv · showhide
3D Morphable Models (3DMMs) are powerful statistical models of 3D facial shape and texture, and among the state-of-the-art methods for reconstructing facial shape from single images. With the advent of new 3D sensors, many 3D facial datasets have been collected containing both neutral as well as expressive faces. However, all datasets are captured under controlled conditions. Thus, even though powerful 3D facial shape models can be learnt from such data, it is difficult to build statistical texture models that are sufficient to reconstruct faces captured in unconstrained conditions ("in-the-wild"). In this paper, we propose the first, to the best of our knowledge, "in-the-wild" 3DMM by combining a powerful statistical model of facial shape, which describes both identity and expression, with an "in-the-wild" texture model. We show that the employment of such an "in-the-wild" texture model greatly simplifies the fitting procedure, because there is no need to optimize with regards to the illumination parameters. Furthermore, we propose a new fast algorithm for fitting the 3DMM in arbitrary images. Finally, we have captured the first 3D facial database with relatively unconstrained conditions and report quantitative evaluations with state-of-the-art performance. Complementary qualitative reconstruction results are demonstrated on standard "in-the-wild" facial databases. An open source implementation of our technique is released as part of the Menpo Project.
1. Introduction
3D facial shape recovery from single in-the-wild images remains difficult because existing data and models do not jointly provide suitable shape and texture priors. The paper addresses this gap with an in-the-wild 3DMM, a fast fitting algorithm, a new evaluation dataset, and an open-source implementation.
- Motivation: Single-image 3D facial shape recovery is an open problem because image intensities combine shape, albedo, normals, camera parameters, and lighting, yielding potentially infinite solutions.Statistical priors for facial shape and texture are therefore needed to resolve the inherent ambiguities.
- Motivation: Existing 3D facial databases were captured in controlled conditions, making in-the-wild statistical priors difficult to learn with current acquisition devices.This limits the availability of training data matching arbitrary capture conditions.
- Motivation: Available data support shape models covering identity and expression, but not a corresponding texture model that generalizes well to in-the-wild images.Consequently, existing methods rely primarily on fitting a statistical shape prior to sparse landmarks.
- Contributions: The paper proposes an in-the-wild feature-based texture model aligned with a statistical shape model covering identity and expression.The feature-based formulation removes the need to optimize illumination parameters during fitting.
- Contributions: The paper introduces a novel fast fitting algorithm for in-the-wild 3DMMs and releases its implementation through the Menpo Project.The authors motivate the release by citing a lack of robust open-source implementations for fitting 3DMMs.
- Evaluation: A new Kinect Fusion dataset provides 3D facial surfaces under relatively unconstrained conditions for quantitative evaluation.The paper also reports qualitative reconstruction results on standard in-the-wild facial databases.
2. Model Training
The model combines PCA-based shape, camera, and feature-based texture components. Its in-the-wild texture model is learned by sampling image features, handling occlusions and corruption, and applying PCA.
- 2. Model Training: A 3DMM comprises parametric shape, camera, and texture models.
- 2.1. Shape Model: The shape model represents corresponded meshes using a mean vector and PCA basis after Procrustes alignment.The basis retains the first ns principal components and generates shapes from ns parameters.
- 2.2. Camera Model: The camera model projects object-centered 3D mesh coordinates into 2D image coordinates using perspective transformation.The implementation uses a pinhole camera; an orthographic model can also be used.
- 2.2. Camera Model: 3D rotation is parameterized with unit-norm quaternions to avoid Euler-angle solution ambiguity and gimbal lock during optimization.The quaternion uses four parameters subject to q^Tq = 1.
- 2.3. “In-the-Wild” Feature-Based Texture Model: The texture model samples dense image features at projected mesh vertices, masks self-occlusions, and reconstructs clean textures from incomplete corrupted observations.Principal Component Pursuit with missing values separates a low-rank texture matrix from sparse noise before PCA creates the final model.
- 2.3. “In-the-Wild” Feature-Based Texture Model: PCA on reconstructed feature-based textures yields a mean texture vector and orthonormal basis for generating novel texture instances.The model retains the first nt principal components and uses nt texture parameters.
3. Model Fitting
The paper fits its 3DMM by minimizing feature-based texture, optional landmark, and regularization costs with Gauss-Newton optimization. A Project-Out variant removes texture increments during iterations, simplifying and accelerating fitting.
- Cost Function: The fitting objective combines feature-based texture reconstruction with optional sparse-landmark and shape or texture regularization terms.The texture term compares sampled input-image features at projected 3D locations with a 3DMM texture instance.
- Cost Function: The proposed in-the-wild texture model removes the need for a parametric illumination model from the cost function.This omission greatly simplifies optimization because illumination parameters do not need to be optimized.
- Gauss-Newton Optimization: Gauss-Newton optimization updates shape, texture, and camera parameters iteratively after first-order linearization of the nonlinear image and landmark terms.Quaternion rotation updates are multiplicative, although they are denoted as additions for simplicity.
- Simultaneous: The Simultaneous algorithm is dominated by a per-iteration complexity of O((ns + nc + nt)^3 + CN(ns + nc + nt)^2), making it too slow in practice.It solves for all parameter increments together.
- Project-Out: Project-Out optimization eliminates texture-parameter increments by optimizing on the orthogonal complement of the texture subspace.Texture parameters are estimated after the iterative procedure, while the resulting per-iteration cost is much lower than Simultaneous optimization.
- Residual Masking: Randomly selecting approximately K ≈5000 visible vertices greatly speeds fitting without an accuracy penalty.The vertices are selected after visibility is determined by z-buffering, with K ≪ N.
4. KF-ITW Dataset
KF-ITW is a 3D facial dataset captured under relatively unconstrained conditions, with subjects recorded across illumination conditions and expressions. Fused meshes provide ground-truth 3D facial shapes for evaluation.
- Dataset Construction: KF-ITW is described as the first dataset of 3D faces captured under relatively unconstrained conditions.It contains 17 subjects recorded under various illumination conditions while performing neutral, happy, and surprise expressions.
- Ground Truth: The dataset uses KinectFusion with a Kinect v1 sensor to acquire 3D representations of the subjects.The fused mesh for each subject serves as ground-truth facial shape for evaluating and comparing reconstruction methods.
5. Experiments
Experiments evaluate the ITW 3DMM on facial shape and surface-normal estimation, using quantitative comparisons on datasets with ground-truth geometry and qualitative fits on unconstrained images.
- 3DMM setup: The ITW model is trained with a Basel-derived shape model containing identity and expression variation and an in-the-wild texture model.The texture model uses annotated iBUG, LFPW, and AFW images.
- 3D shape recovery: On KF-ITW, facial-shape accuracy is measured by normalized dense per-vertex error, with cumulative distributions, AUC, and failure rates reported.All three models are initialized and fitted using the dataset’s ground-truth annotations and evaluated after model-specific correspondence fitting.
- 3D shape recovery: The Classic model performs worst under in-the-wild conditions, the texture-free Linear model performs better, and ITW recovers facial shape most effectively.The comparison uses Classic, Linear, and ITW 3DMM formulations on KF-ITW.
- Qualitative results: Qualitative fits from Helen and 300W cover varied expression, identity, lighting, and occlusion conditions and produce realistic 3D facial shapes.Figure 6 presents examples of in-the-wild fits produced by the ITW 3DMM.
- Surface normals: ITW slightly outperforms IMM for facial surface-normal estimation using cumulative distributions of mean angular error.The evaluation uses 100 Photoface subjects and calibrated four-source photometric stereo to generate ground-truth normals.
6. Conclusion
The paper presents an in-the-wild 3DMM with feature-based texture learning and a new fitting procedure, then evaluates it quantitatively and qualitatively. The formulation recovers more detailed shapes than landmark-driven approaches and outperforms classical 3DMM approaches on the introduced in-the-wild evaluation.
- Contributions: The proposed ITW 3DMM learns an in-the-wild feature-based texture model and fits images without optimizing illumination parameters.The paper also proposes a novel optimization procedure for 3DMM fitting.
- Shape recovery: The method recovers more detailed shapes than purely landmark-driven approaches.This conclusion concerns the level of recovered facial-shape detail.
- Dataset and evaluation: The new in-the-wild KinectFusion dataset enables quantitative evaluation of 3D facial reconstruction techniques in the wild.The dataset is introduced as part of the paper’s evaluation framework.
- Overall result: On these evaluations, the in-the-wild formulation is reported as state of the art and to outperform classical 3DMM approaches by a considerable margin.This is the paper’s overall conclusion from the introduced evaluations.