Source-linked AI summary
Self-supervised Multi-level Face Model Learning for Monocular Reconstruction at over 250 Hz
Ayush Tewari, Michael Zollhöfer, Pablo Garrido, Florian Bernard, Hyeongwoo Kim, Patrick Pérez, Christian Theobalt
TL;DR
Single-image face reconstruction is ill-posed and existing parametric priors restrict real-world facial diversity. The paper jointly learns a regressor and multi-level face model with differentiable self-supervised rendering, achieving favorable reconstruction quality and operation above 250 Hz. Its scope is bounded by external occlusions and low-dimensional corrective spaces that cannot recover fine-scale detail.
Problem
Single-image reconstruction must recover geometry, reflectance, and illumination despite existing parametric priors limiting generalization beyond restricted face subspaces.
Method
The method jointly learns a feed-forward regressor and multi-level face model combining 3DMM regularization with trainable corrective spaces and differentiable self-supervised rendering.
Results
The approach compares favorably with state-of-the-art reconstruction methods, handles facial hair and makeup robustly, and runs at over 250 Hz.
Takeaways & Limitations
Learned corrective spaces improve generalization and robustness while retaining parametric regularization for high-quality monocular face reconstruction.
Takeaways & Limitations
External occlusions can be baked into correctives, and the low-dimensional corrective space cannot recover fine-scale surface detail.
Abstract
from arXiv · showhide
The reconstruction of dense 3D models of face geometry and appearance from a single image is highly challenging and ill-posed. To constrain the problem, many approaches rely on strong priors, such as parametric face models learned from limited 3D scan data. However, prior models restrict generalization of the true diversity in facial geometry, skin reflectance and illumination. To alleviate this problem, we present the first approach that jointly learns 1) a regressor for face shape, expression, reflectance and illumination on the basis of 2) a concurrently learned parametric face model. Our multi-level face model combines the advantage of 3D Morphable Models for regularization with the out-of-space generalization of a learned corrective space. We train end-to-end on in-the-wild images without dense annotations by fusing a convolutional encoder with a differentiable expert-designed renderer and a self-supervised training loss, both defined at multiple detail levels. Our approach compares favorably to the state-of-the-art in terms of reconstruction quality, better generalizes to real world faces, and runs at over 250 Hz.
1. Introduction
Monocular face reconstruction is an ill-posed problem because geometry, reflectance, and illumination collapse into one pixel measurement. Existing parametric priors regularize reconstruction but restrict facial diversity, motivating a jointly learned multi-level model and fast self-supervised inference.
- The method targets monocular reconstruction for applications including motion capture, content creation, virtual and augmented reality, and communication.
- Single-image reconstruction must disentangle geometry, reflectance, and illumination from one color measurement per pixel.
- 3D Morphable Models constrain faces to a restricted low-dimensional subspace learned from limited scan databases.
- These priors can miss facial details, produce incorrect features, and fail on characteristics such as beards or unusual noses.
- The proposed method jointly learns a regressor and corrective face model to improve generalization beyond predefined parametric subspaces.
- The approach combines a CNN with differentiable image formation and multi-level self-supervision for training on in-the-wild images without dense annotations.
3. Method Overview
The method combines a low-dimensional parametric base model with learned medium-scale geometry and reflectance correctives. A feed-forward encoder and corrective representation are trained jointly through differentiable rendering and self-supervision.
- At test time, a CNN computes a latent face representation in under 4ms from a coarse 3DMM plus trainable per-vertex correctives.
- The base model uses affine geometry and reflectance spaces, with separate shape and expression components derived from scan and blendshape data.
- Combining the base and corrective levels yields the final face model for geometry and reflectance.
- The base representation uses 80 shape vectors, 80 reflectance vectors, and 64 expression vectors, but remains limited to coarse-scale geometry.
- Trainable corrective mappings add out-of-space variations while learning both the generative model and corrective parameters without dense geometry, reflectance, or illumination annotations.
5. Differentiable Image Formation Model
The differentiable image formation model maps predicted face geometry and reflectance through camera and illumination models into rendered images. Its differentiability enables end-to-end backpropagation through the reconstruction pipeline.
- The image formation model is differentiable, enabling end-to-end training with backpropagation.
- A rigid transformation and full perspective camera project model-space 3D points into camera-space and then onto image pixels.
- The illumination model assumes distant lighting and represents incoming radiance with spherical harmonics depending on surface normals.
- Three spherical-harmonic bands approximate smooth incident radiance with an average error below 1%, using nine variables per color channel.
- The rendering layer accepts base-level or corrective-enhanced per-vertex shape and reflectance before transforming and projecting vertices.
- Training learns face, camera, illumination, and corrective-space variables jointly.
6. Self-supervised Learning
The self-supervised objective combines image alignment with coarse- and medium-scale face priors, using sparse landmarks, dense photometric terms, and regularization for geometry and reflectance.
- Objective: The loss combines a data-fitting term with regularization to fit the base model and learn per-vertex correctives end-to-end.The regularizer encodes prior assumptions at coarse and medium scales.
- Data Terms: Dense multi-level photometric alignment measures misalignment between coarse and corrected fits while using robust color distances and visibility filtering.The ℓ2,1-norm combines color-distance measurement with sparsity across pixel-wise residuals; visibility uses backface culling.
- Data Terms: Automatically detected 66 facial landmarks provide weak supervision through fixed points and pose-dependent sliding contour points.Sliding-point correspondences are updated by selecting the mesh vertex nearest the back-projected feature ray.
- Regularization Terms: Statistical regularization constrains base-level 3DMM parameters, while stabilization keeps corrected geometry close to the base reconstruction.These constraints address degeneration in the ill-posed monocular reconstruction setting.
- Regularization Terms: Corrective smoothness applies Laplacian regularization to vertex displacements, and local reflectance sparsity further regularizes the full reflectance estimate.The reflectance sparsity term uses chromaticity similarity between input colors and fixed α = 50 and p = 0.9.
- Regularization Terms: Global reflectance constancy compares randomly sampled skin vertices while excluding facial-hair regions, helping remove shading from reflectance.The skin mask covers the fixed skin region, and each vertex uses six random mask-region samples.
7. Results
The jointly trained encoder and multi-level face model produce high-quality reconstructions across qualitative and quantitative comparisons, including challenging facial appearance variations. The method also improves photometric fitting while retaining fast inference and requiring no dense ground truth.
- Training: The jointly trained encoder and multi-level face representation are trained self-supervised on in-the-wild images without densely annotated ground truth.The network regresses pose, shape, expression, reflectance, and illumination.
- Comparisons to the State-of-the-art: Our approach obtains higher-quality reconstructions than Garrido et al. because the jointly learned model generalizes better than a manifold-harmonics corrective space.The comparison concerns qualitative reconstruction quality.
- Comparisons to the State-of-the-art: The approach yields a reflectance model, whereas Booth et al.'s texture model contains shading.This distinguishes reconstructed reflectance from shading-inclusive texture modeling.
- Comparisons to the State-of-the-art: The approach is robust to facial hair and make-up, while compared learning-based methods struggle with out-of-subspace shape and reflectance variations.It also reconstructs colored reflectance and illumination jointly.
- Comparisons to the State-of-the-art: 4 ms reconstruction is orders of magnitude faster than Garrido et al.'s 120 s per image and avoids landmarks at test time.The approach also avoids Sela et al.'s offline non-rigid registration.
- Quantitative Results: The method outperforms deep learning techniques on FaceWarehouse, comes close to Garrido et al., and outperforms Garrido et al. on the Volker sequence.The FaceWarehouse comparison uses geometric error; the Volker result is reported in Table 2.
- Quantitative Results: 0.072 mean photometric error is lower than the base level's 0.092 mean, with standard deviations of 0.020 and 0.025, respectively.The error is Euclidean distance in RGB space with channels in [0, 1].
8. Limitations
The approach has limitations involving external occlusion and fine-scale surface detail.
- Limitations: External occlusions such as glasses are baked into the corrective space, so occluded face regions cannot be reconstructed consistently.Resolving this would require semantic segmentation of the training corpus.
- Limitations: The low-dimensional corrective space prevents recovery of fine-scale surface detail.The authors identify this as an orthogonal direction for future work.
9. Conclusion
The paper jointly learns a face model and parameter regressor, combining 3DMM regularization with learned corrective-space generalization. This yields high-quality reconstructions at over 250 Hz and motivates extending the approach beyond faces.
- Conclusion: The approach jointly learns a face model and parameter regressor for face shape, expression, appearance, and illumination.This is presented as the paper's central contribution.
- Conclusion: Combining 3DMM regularization with a learned corrective space improves generalization and robustness for high-quality reconstruction at over 250 Hz.The conclusion frames this combination as overcoming disadvantages of strong-prior approaches.
- Conclusion: The approach can be generalized to further object classes beyond face reconstruction.The authors describe it as a step toward building 3D models from in-the-wild images.
— Supplemental Material —
The supplemental material details robust training, runtime, corrective-space choices, parameter-count studies, and additional results. It also clarifies the multi-level model and its reflectance representation.
- Supplemental Material: The supplement covers robust training, training and testing runtime, corrective-space comparisons, corrective-parameter studies, and additional results.These topics correspond to Sections 1 through 5 of the supplement.
- Supplemental Material: Colored reconstructions visualize reconstructed reflectance and illumination rather than texture copied from the input image.The underlying model uses a 3DMM base level and a final level with learned per-vertex shape and reflectance correctives.
- Supplemental Material: The skin-reflectance representation uses a low-dimensional coefficient vector of size 580 = 500 + 80.The supplied passage introduces this dimensionality while describing the supplemental model.
1. Training
The method is trained in two stages, combining a learned multi-level face model with a reconstruction network. The learned corrective space improves geometry and reflectance over the base 3DMM.
- 1. Training: Jointly learning the multi-level model improves estimated geometry and reflectance over the underlying 3DMM at the coarse base level.The reconstructed nose, lips, and facial hair align better in the learned model.
- 1. Training: Two-stage training first pretrains the network through the base level, then fine-tunes the complete network end-to-end.The base stage uses 200k iterations; subsequent end-to-end fine-tuning uses 190k iterations.
2. Runtime Performance
The trained system performs monocular face reconstruction in real time, with a forward pass taking less than 4 ms. This corresponds to more than 250 Hz on an Nvidia GTX TITAN Xp.
- 2. Runtime Performance: More than 250 Hz: the trained regressor reconstructs monocular faces at this rate on an Nvidia GTX TITAN Xp.A forward pass takes less than 4 ms; training takes 16 hours.
3. Evaluation of the Corrective Space
The corrective space can be modeled linearly or nonlinearly, but experiments select the simpler affine/linear model. Increasing corrective capacity improves photometric re-rendering error over the 3DMM-only baseline.
- 3. Evaluation of the Corrective Space: Corrective mappings transform C-dimensional parameters into per-vertex shape or reflectance corrections.Linear mappings use a trainable matrix and offset; nonlinear variants add ReLU-based layers.
- 3. Evaluation of the Corrective Space: Lower photometric re-rendering error: the affine/linear model outperforms both tested nonlinear corrective spaces.The authors therefore use the simpler affine/linear model in all experiments.
- 3. Evaluation of the Corrective Space: C = 500 corrective parameters achieves the lowest photometric re-rendering error among the tested corrective-space sizes.Networks with corrective parameters outperform the C = 0 baseline using only the 3DMM.
4. Additional Evaluation
Additional evaluations show that corrective parameters and self-supervised priors improve reconstruction quality, including reflectance accuracy on synthetic images. Comparisons also report favorable reconstruction quality and robustness to challenging facial appearance conditions, while external occluders remain a limitation.
- 4. Additional Evaluation: Correctives significantly improve photometric re-rendering error over the 3DMM-only baseline on a 2k-image test set.The evaluation varies shape and reflectance corrective parameters across C = 0, 100, 250, and 500.
- 4. Additional Evaluation: Removing shape regularizers causes training to fail because independent corrective displacements leave the reconstruction problem severely underconstrained.The shape regularizers are stabilization and smoothness terms.
- 4. Additional Evaluation: Removing reflectance sparsity priors makes shading information incorrectly explained by reflectance variation, despite visually plausible overlays.Both local and global reflectance sparsity terms help produce plausible reflectance estimates.
- 4. Additional Evaluation: 0.072 RGB error: predicted reflectance differs from ground truth by this average per-pixel distance over 1k synthetic test images.Global shifts in reflectance are compensated, and rendering uses mean face geometry in a canonical pose.
- 4. Additional Evaluation: The approach compares favorably with optimization-based and learning-based methods and handles occlusions by facial hair and make-up.External occluders can be baked into the recovered model, illustrating a limitation.