Source-linked AI summary
FML: Face Model Learning from Videos
Ayush Tewari, Florian Bernard, Pablo Garrido, Gaurav Bharaj, Mohamed Elgharib, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt
TL;DR
Monocular face reconstruction is ill-posed because 2D images leave depth ambiguous and existing priors often come from limited scans. The paper jointly learns facial identity shape, appearance, and 3D reconstruction from in-the-wild videos using multi-frame consistency. The approach matches or outperforms the state of the art in learning-based face reconstruction, although image quality and several difficult visual conditions remain limiting factors.
Problem
Monocular 3D face reconstruction is ill-posed because depth is ambiguous, while many existing priors derive from limited scans that can bias generalization.
Method
A self-supervised deep network learns facial identity geometry and albedo while reconstructing 3D faces from multiple frames of in-the-wild videos.
Results
The approach matches or outperforms the state of the art in learning-based face reconstruction.
Takeaways & Limitations
Multi-frame video supervision supports learning a face model from scratch without dense geometry and appearance annotations.
Takeaways & Limitations
Low-resolution, noisy, or blurred in-the-wild data bounds achievable quality, while extreme poses, occlusions, illumination, and reflectance ambiguities remain challenging.
Abstract
from arXiv · showhide
Monocular image-based 3D reconstruction of faces is a long-standing problem in computer vision. Since image data is a 2D projection of a 3D face, the resulting depth ambiguity makes the problem ill-posed. Most existing methods rely on data-driven priors that are built from limited 3D face scans. In contrast, we propose multi-frame video-based self-supervised training of a deep network that (i) learns a face identity model both in shape and appearance while (ii) jointly learning to reconstruct 3D faces. Our face model is learned using only corpora of in-the-wild video clips collected from the Internet. This virtually endless source of training data enables learning of a highly general 3D face model. In order to achieve this, we propose a novel multi-frame consistency loss that ensures consistent shape and appearance across multiple frames of a subject's face, thus minimizing depth ambiguity. At test time we can use an arbitrary number of frames, so that we can perform both monocular as well as multi-frame reconstruction.
1. Introduction
The paper addresses ill-posed monocular 3D face reconstruction by learning shape and appearance directly from unconstrained videos. Its multi-frame training enforces identity consistency across frames while separating identity geometry from expression.
- Problem: In-the-wild monocular reconstruction must disentangle facial shape, albedo, expression, lighting, and camera parameters despite depth ambiguity and occlusions.Albedo and lighting are not easily separable in monocular images, and uncontrolled capture conditions make reconstruction harder.
- Limitations of prior models: Existing methods rely on strong face-model priors that can introduce bias because they are learned from limited scans and may generalize poorly across identities and ethnicities.The paper specifically identifies scan-based 3DMMs as a source of restricted generalization.
- Approach: The proposed approach learns facial identity geometry and albedo from unconstrained Internet video clips without requiring a pre-existing shape or appearance model.Its weak training-data requirements support a virtually endless amount of community data and learning from scratch.
- Approach: A multi-frame consistency loss enforces consistent identity and albedo across frames of the same subject, reducing monocular depth ambiguity.The approach uses multiple images of each subject, including multi-view sequences or monocular videos.
- Approach: Projection onto the blendshapes’ nullspace explicitly separates facial expression from identity-based geometry variations.This separation avoids relying on a large set of handcrafted priors.
2. Related Work
Prior work learns or fits face models from scans, monocular data, photo-collections, or multiple frames, with methods differing in their priors, representations, and supervision. The paper situates its approach among these model-learning and reconstruction strategies.
- Morphable models from scans: Traditional 3D morphable models learn low-dimensional geometry and appearance subspaces from scanner data, while richer models use larger datasets or tensor decompositions.Sparse localized components and Gaussian-process models address localized or nonlinear shape deformations.
- Parametric models from monocular data: Monocular-data methods include personalized models refined from video, corrective models for out-of-space deformations, and morphable models learned from image sequences.These approaches differ in whether they model person-specific detail, corrections, or broader variation.
- Monocular 3D reconstruction: Monocular 3D reconstruction methods optimize personalized or parametric models, regress geometry from single images, or use inverse-rendering losses.Their supervision may come from ground-truth geometry, a morphable model, or unsupervised image reconstruction.
- 3D reconstruction via photo-collections: Photo-collection approaches fit templates or learn models from collections grouped by semantic labels, supporting reconstruction and facial-motion tracking.Examples include low-rank factorization and large photo-collection-based 3DMM learning.
- Multi-frame 3D reconstruction: Multi-frame reconstruction exploits temporal information or multiple views through multilinear fitting, frame averaging, or bundle adjustment.These methods use cross-frame information to improve estimates of facial geometry.
3. Face Model Learning
The method learns a parametric face geometry and appearance model jointly with estimators for frame-specific and shared identity parameters from multi-frame video data. Its graph-based representation, differentiable image formation, and shared identity estimation support self-supervised multi-frame learning.
- Overview: The network jointly learns face geometry and appearance models while estimating shape, expression, albedo, pose, and illumination parameters.Identity parameters are shared across frames, whereas pose, illumination, and expression are estimated per frame.
- Training Data: Training uses multi-frame images sampled from video clips, including 404k samples from VoxCeleb2 videos of more than 6000 celebrities.Frames are sampled from the same video clip to preserve subject identity and avoid variations such as aging or accessories.
- Graph-based Face Representation: The face representation combines a coarse deformation graph with a high-resolution mesh whose vertices encode facial appearance.The graph contains 521 nodes, and its learned deformation is transferred to the high-resolution mesh through linear blend skinning.
- Graph-based Face Representation: The graph identity model uses a learned 500-dimensional deformation subspace, with α representing graph deformation parameters.The mean graph positions are fixed, while the deformation basis and parameters define identity-related shape variation.
- Shape and Expression: High-resolution vertex positions combine the neutral mean shape with graph-based identity deformation, while a fixed blendshape model represents facial expressions.Expression deformations are applied directly to the high-resolution mesh, and orthogonality separates learned identity shape from expression variation.
- Multi-frame Learning: A shared identity estimator pools person-specific shape and appearance parameters across Siamese streams processing different frames.The hard sharing constraint makes identity parameters common to all frames while frame-specific parameters capture pose, illumination, and expression.
- Multi-frame Learning: The multi-frame photometric consistency loss enforces shared identity parameters across frames while allowing illumination and head pose to vary.This consistency objective is presented as model-based non-rigid structure-from-motion during training and supports disentanglement of illumination and appearance.
4. Results
The approach produces strong monocular and multi-frame reconstructions, improving consistency, illumination estimation, geometry, and reflectance separation across challenging conditions. Quantitative evaluations show competitive or superior performance without requiring a precomputed face model.
- Monocular reconstructions recover geometry, reflectance, and scene illumination while generalizing to different ethnicities, facial hair, and makeup.
- Multi-frame reconstruction improves reconstruction consistency and quality, particularly in regions occluded in one image.
- Multi-frame training improves illumination estimation and performance under large poses compared with Tewari et al.
- The learned model produces more accurate results than approaches constrained by synthetic training data or fixed 3D face models.
- The method estimates better geometry and separates reflectance from illumination, unlike Tran et al.'s approach.
- On BU-3DFE, the approach yields higher-quality geometric reconstruction and generalizes to the dataset's ±45 degree head poses.
- On FaceWarehouse, the method competes with leading methods, outperforms [60] and [32], and learns its face model from scratch.
5. Conclusion & Discussion
The paper presents self-supervised joint learning of a face model and reconstruction network from in-the-wild video clips without ground-truth annotations. Despite low-resolution, noisy, and blurred training data, the method matches or outperforms state-of-the-art learning-based reconstruction.
- The approach jointly learns a face model and 3D face reconstruction network through self-supervised multi-frame training.
- The model is learned from a large corpus of in-the-wild video clips without available ground truth.
- Low resolution, noise, and blur in in-the-wild data bound the achievable reconstruction quality.
- The method matches or outperforms the state-of-the-art in learning-based face reconstruction.
— Supplemental Document —
The supplemental document provides implementation details, additional comparisons, limitation discussion, and analyses of the learned model's modes and statistics.
- The main approach jointly learns a face model and 3D reconstruction from in-the-wild video data while disentangling shape, appearance, expression, and illumination.
- The supplement expands the network architecture and empirically determined loss weights.
- It includes additional qualitative and quantitative comparisons and discusses limitations of the approach.
- The document describes extracting statistics from the learned model and visualizing its modes.
1. Network Details
The supplemental network details describe a 124M-parameter implementation and a PCA-based reparameterization for interpreting variation in the learned face and appearance model.
- The network contains 124M parameters, including parameters for the learned geometry and appearance model.
- The networks are trained on commodity Titan Volta GPUs.
- PCA is computed over reconstructions from more than 10k training-set images to reparameterize the learned model.
- The reparameterized model exposes global modes such as gender and local modes such as nose and eye deformations.
- The visualization separates the learned shape identity model from the expression model.
2. Weights of the Energy
The section specifies fixed energy weights and provides architectural details for feature extraction and shared identity processing, alongside visualizations of learned shape modes.
- Weights of the Energy: The energy weights are fixed empirically across all experiments.The reported values include λ_pho = 1.6/|V̄|, λ_lan = 4.7, λ_smo = 0.001, λ_spa = 1e−7, and λ_ble = 1e−8.
- Shape Model: The learned shape model is visualized through multiple learned modes, including side-view variations.
3. Results
The results show that multi-frame training and inference improve reconstruction consistency and quality, while the learned model generalizes to challenging identities, poses, lighting, and appearance conditions.
- Monocular vs multi-frame reconstruction: Multi-frame reconstruction improves overall consistency and resolves ambiguities caused by occlusions in individual images.Single-image input cannot completely resolve these ambiguities, although training with multiple frames still yields plausible monocular identity reconstruction.
- Comparison to state-of-the-art methods: The approach reconstructs faces under large poses, harsh low-frequency illumination, beards, and non-average facial shapes.The authors attribute this generalization to learning the model from in-the-wild data rather than relying only on synthetic data or a fixed 3DMM.
- Comparison to state-of-the-art methods: The method disentangles facial geometry, reflectance, and scene illumination, producing relatively clean reflectance estimates.The reflectance estimate has a small global scaling ambiguity, while competing texture models can include shading effects.
- Quantitative evaluations: Larger learned shape and appearance models produce lower photometric errors on 1000 CelebA images and outperform the compared 3DMM-based optimization method.Table 4 reports average photometric error over the CelebA dataset and compares the method against an existing face-model approach.
- Quantitative evaluations: Multi-frame training and two-image test-time reconstruction improve geometry quality on 324 BU-3DFE images compared with monocular reconstruction.The approach also outperforms Tewari et al. on this dataset; average and max pooling produce geometric errors differing by less than 0.004mm in the reported setting.
4. Limitations
The approach remains limited by extreme poses, occlusions, unusual facial characteristics, illumination conditions, and ambiguities between reflectance and lighting.
- Limitations: Extreme head poses remain difficult despite the method handling large poses relatively well.
- Limitations: Accessories and thick facial hair can reduce reconstruction quality, while beards are modeled in reflectance rather than physically reconstructed.
- Limitations: A global scale ambiguity prevents reliable separation of global skin tone from ambient illumination brightness.
- Limitations: Strong colorful directional lighting, specular reflections, cast shadows, and non-standard facial shapes can challenge reconstruction.Specular reflections and cast shadows are not modeled by the differentiable renderer and may be baked into reflectance.