PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Metrical Reconstruction of Human Faces

Wojciech Zielonka, Timo Bolkart, Justus Thies

arXiv:2204.06607v2cs.CV

TL;DR

Monocular face reconstruction can align images while failing to recover absolute face dimensions, limiting metrical applications. MICA combines unified metric 3D data, robust pretrained identity features, and supervised shape prediction, and is reported to outperform state-of-the-art methods on non-metric and metric benchmarks.

  • Problem

    Perspective projection makes face scale ambiguous, so existing single-image methods may not recover actual face dimensions despite good 2D alignment; metrical shape is needed for measurement and AR/VR contexts.

  • Method

    MICA unifies small- and medium-scale 3D datasets and trains a supervised metrical shape estimator using robust identity features from a pretrained face recognition network.

  • Results

    MICA outperforms state-of-the-art reconstruction methods on current non-metric benchmarks and the paper’s metric benchmarks.

  • Takeaways & Limitations

    The paper supports evaluating face reconstruction with absolute metrical errors rather than errors after optimal scale alignment.

  • Takeaways & Limitations

    The method does not predict shape and expression in one forward pass and fails when the face detector does not recognize an input face.

Abstract

from arXiv · show

Face reconstruction and tracking is a building block of numerous applications in AR/VR, human-machine interaction, as well as medical applications. Most of these applications rely on a metrically correct prediction of the shape, especially, when the reconstructed subject is put into a metrical context (i.e., when there is a reference object of known size). A metrical reconstruction is also needed for any application that measures distances and dimensions of the subject (e.g., to virtually fit a glasses frame). State-of-the-art methods for face reconstruction from a single image are trained on large 2D image datasets in a self-supervised fashion. However, due to the nature of a perspective projection they are not able to reconstruct the actual face dimensions, and even predicting the average human face outperforms some of these methods in a metrical sense. To learn the actual shape of a face, we argue for a supervised training scheme. Since there exists no large-scale 3D dataset for this task, we annotated and unified small- and medium-scale databases. The resulting unified dataset is still a medium-scale dataset with more than 2k identities and training purely on it would lead to overfitting. To this end, we take advantage of a face recognition network pretrained on a large-scale 2D image dataset, which provides distinct features for different faces and is robust to expression, illumination, and camera changes. Using these features, we train our face shape estimator in a supervised fashion, inheriting the robustness and generalization of the face recognition network. Our method, which we call MICA (MetrIC fAce), outperforms the state-of-the-art reconstruction methods by a large margin, both on current non-metric benchmarks as well as on our metric benchmarks (15% and 24% lower average error on NoW, respectively).

1 Introduction

Metrical face reconstruction is difficult because perspective projection leaves scale ambiguous, while many applications require absolute 3D dimensions. The paper addresses this gap with supervised learning, unified 3D data, identity features, and metric evaluation resources.

  • Motivation: Perspective projection makes face scale ambiguous, allowing differently sized faces at different distances to produce the same image.Consequently, 2D alignment can coexist with incorrect metrical 3D geometry and spatial location.
  • Motivation: Metrical 3D reconstruction is needed when faces are placed in contexts containing known-size objects or when facial distances and dimensions must be measured.The paper specifically identifies AR/VR applications and virtual try-on as examples.
  • Approach: The method predicts neutral-expression head geometry from one RGB image using unified FLAME-topology data and identity features from a face recognition network.The recognition features are intended to support robustness to pose, expression, occlusion, illumination, and focal-length changes.
  • Contributions: The paper contributes a unified 3D reference dataset for about 2300 subjects under common FLAME topology.The dataset combines existing small- and medium-scale datasets.
  • Contributions: The paper introduces MICA, a metrical face shape predictor using supervised learning and generalized identity features, plus a hybrid face tracker and metrical evaluation benchmark.The tracker combines learned metrical reconstruction with optimization-based facial expression tracking.
  • Evaluation: The paper highlights that current evaluation practice can report millimeter errors after scale alignment, which does not represent absolute metrical error.This motivates changing benchmark protocols for metrical evaluation.

2 Related Work

Related work spans model-based and learned monocular face reconstruction, with methods differing in optimization strategy, supervision, rendering losses, and geometric representation.

  • Model-based reconstruction: Monocular face reconstruction methods commonly use priors of face shape and appearance, including 3D morphable models such as Basel Face Model and FLAME.3DMMs represent shape and appearance in a compressed, low-dimensional PCA-based space.
  • Optimization-based reconstruction: Analysis-by-synthesis methods optimize rendered outputs against images, progressing from sparse color sampling to dense color terms with differentiable rendering.These methods have also been adapted for avatar digitization from a single image.
  • Regression-based reconstruction: Learning-based reconstruction methods include supervised regressors trained on synthetic renderings with known 3DMM parameters and self-supervised approaches using image consistency losses.Some methods combine synthetic supervision with real-image multi-view identity or photometric losses.
  • Regression-based reconstruction: Regression methods also separate coarse 3DMM estimates from detailed texture, displacement, bump-map, or UV-space offset predictions.The cited approaches use photometric, landmark, attention, or hybrid losses for refinement.

3 Metrical Face Shape Prediction

MICA predicts a neutral, metrical face shape from one RGB image by combining robust ArcFace identity features, supervised paired 2D/3D training, and a FLAME geometry decoder.

  • Overview: MICA uses a single RGB image to predict a metrical human face shape in neutral expression.Training combines in-the-wild 2D data with metric 3D data.
  • Identity encoder: ArcFace, pretrained on Glint360K, supplies discriminative identity features designed to be robust to illumination, expression, rotation, occlusion, and camera parameters.A mapping network converts ArcFace features into the identity code used by the geometry decoder.
  • Tracking application: The proposed estimation can support analysis-by-synthesis facial-expression tracking that optimizes camera intrinsics, illumination, expression, and pose per frame.The tracker uses the metrical shape estimate as its reconstruction component.
  • Geometry decoder: The FLAME-based geometry decoder represents face geometry as G_3DMM(z) = B · z + A, combining learned principal components with the average-face geometry.The latent code z has dimension 300 and the mesh contains N = 5023 vertices.
  • Supervised learning: The training objective uses paired 2D/3D data and masked mesh error, while most ArcFace layers remain fixed and only the last three ResNet blocks are refined.The restricted refinement is selected to preserve generalization and avoid overfitting the medium-scale dataset.

4 Face Tracking

The tracker uses the learned metrical face shape with optimization-based expression and appearance tracking on monocular RGB sequences. Its objective combines dense color reproduction, sparse landmark alignment, and regularization.

  • 4 Face Tracking: The tracker combines the learned metrical face shape with optimization-based facial expression tracking for monocular RGB sequences.It models non-rigid deformations with FLAME expression basis vectors and linear blend skinning.
  • 4 Face Tracking: The objective combines dense color, 2D landmark, and neutral-pose regularization terms weighted by w_dense, w_lmk, and w_reg.The dense term uses an ℓ1 color-reproduction measure, while the landmark term measures 2D landmark reproduction.
  • 4 Face Tracking: The first-frame optimization estimates albedo, expression, rigid pose, and spherical-harmonic illumination parameters.After initialization, the albedo parameters remain fixed throughout sequence tracking.
  • 4 Face Tracking: Differentiable rendering computes dense photometric error over visible reconstructed-model surface points sampled after the depth test.The error compares RGB input values at projected surface points with reconstructed vertex colors.

5 Dataset Unification

The authors unify heterogeneous 3D face datasets by registering and non-rigidly fitting FLAME to their scans, producing shared-topology neutral meshes. The resulting pool spans varied capture conditions and contains 2315 subjects.

  • 5 Dataset Unification: Available 3D face datasets are unified by registering the FLAME head model to scan data and refining the fit with FLAME-regularized non-rigid deformation.The unified meshes share FLAME topology and provide supervision for reconstruction from 2D images.
  • 5 Dataset Unification: 2315 subjects are included in the unified dataset, all represented with FLAME topology.The dataset provides one registered neutral-expression mesh per subject for the listed data sources.
  • 5 Dataset Unification: The unified datasets vary in capture modality, protocol, facial-expression coverage, imagery type, and geographic region.This variation includes indoor and outdoor imagery, still images and videos, and captures with or without hair caps.
  • 5 Dataset Unification: The combined data pool is more diverse because the source datasets were recorded in different world regions and often carry ethnicity biases.The authors state that combining datasets results in a more diverse pool suitable for generalization.

6 Results

The experiments evaluate face reconstruction on established non-metrical benchmarks and proposed metrical protocols, alongside qualitative robustness tests. MICA shows persistent shape predictions across challenging conditions and handles metrical face shape better than competing methods, while expression tracking remains a separate optimization stage.

  • Tracking: MICA is used to initialize expression tracking, but facial expression is reconstructed separately through an optimization-based tracking method.The method is not designed to predict shape and expressions in one forward pass.
  • Evaluation protocols: Established benchmarks use scale optimization, producing non-metric errors that can improve without improving reconstruction quality.For the average FLAME face on NoW, scaling reduces mean error from 1.92mm to 1.53mm, an improvement of around 20%.
  • Evaluation protocols: The proposed metrical benchmark removes scale optimization and evaluates Euclidean distances after rigid alignment.This protocol compares reconstructions using metrical quantities rather than relative, scale-adjusted error.
  • Quantitative evaluation: On the Stirling benchmark, MICA significantly improves reconstruction accuracy across metrics under the revised evaluation protocol.The Stirling experiment excludes the Stirling dataset from training, and dense ICP replaces the noisy landmark-based alignment for the listed results.
  • Qualitative evaluation: Qualitative results show MICA maintains face-shape predictions across illumination, expression, occlusion, rotation, and perspective changes.Compared with other methods, its predictions are described as more persistent, particularly relative to Deng et al., whose shape predictions vary the most.
  • Qualitative evaluation: MICA reconstructs overall face shape and challenging regions such as the nose and chin, despite a smaller identity and ethnicity pool in its training data.The comparison uses randomly sampled VoxCeleb2 identities and contrasts MICA with baselines including RingNet.
  • Qualitative evaluation: For toddlers, competing methods predict adult-sized faces fitted to image observations, whereas MICA recovers their physiognomy in a common metrical space.The comparison exposes the difference between image-space fitting and metrical 3D reconstruction.

7 Discussion & Conclusion

The discussion argues that metrical reconstruction and metrical evaluation are necessary for applications involving dimensions, distances, or known-size scene objects. It presents dataset unification and supervised learning as steps toward this goal while acknowledging the method’s operational limitations.

  • Discussion: Metrical reconstruction is needed when applications measure distances or dimensions or place reconstructed humans and known-size objects into shared scenes.The discussion highlights virtual reality and augmented reality as especially important application settings.
  • Discussion: The authors argue that scale-optimized benchmarks are misleading because their millimeter errors are not absolute metrical measurements.Their adjustment removes optimal scaling and permits only rigid alignment between prediction and reference shape.
  • Conclusion: Unifying small- and medium-scale paired 2D/3D datasets enables supervised 3D losses for metrical shape prediction.The resulting data collection remains comparably small, at around 2k identities.
  • Supplemental evidence: The supplemental results include qualitative comparisons showing that the method captures detailed face shape while remaining metrically plausible.The examples include intricate regions such as the nose and chin on randomly sampled VoxCeleb2 images.

Towards Metrical Reconstruction of Human Faces –Supplemental Document–

The supplemental document reports additional robustness and quantitative experiments, includes cumulative NoW plots, and examines architectural choices and an alternative model-free approach. It also identifies the document’s institutional authorship context.

  • Document context: The supplemental document is authored by Wojciech Zielonka, Timo Bolkart, and Justus Thies at the Max Planck Institute for Intelligent Systems, Tübingen.
  • Supplemental experiments: The supplemental document demonstrates robustness through additional qualitative and quantitative experiments.It also includes cumulative error plots from the NoW challenge presented in the main paper.
  • Architecture analysis: The document provides a justification of the selected architecture for the unified dataset.This addresses architecture choice in relation to the dataset used for training.
  • Alternative approach: An alternative model-free estimation approach is discussed alongside the main reconstruction method.The supplemental document frames this as an approach that does not rely on a 3DMM decoder.

1 Additional Results

Additional experiments compare encoder choices, decoder capacity, training datasets, and the effect of including FaceScape. They show the importance of pretrained feature selection and dataset composition for reconstruction quality.

  • Additional ablations: The ablation evaluates face encoding networks, decoder dimensionality, training datasets, and FaceScape using metrical reconstruction experiments.The experiments include comparisons involving ArcFace, FaceNet, DECA, FLAME principal components, leave-one-out dataset analysis, and FaceScape.
  • Encoder ablation: Pretrained ArcFace outperforms pretrained FaceNet, while partial ArcFace refinement improves reconstruction without the severe overfitting observed when fine-tuning DECA.Fine-tuning partial layers or the entire DECA pipeline produces significantly worse test reconstructions, whereas partial ArcFace fine-tuning gives the lowest reported mean reconstruction error.
  • Decoder ablation: Using the full linear space of FLAME gives the best performance among the evaluated numbers of principal components.The decoder ablation varies the size of the latent geometry code z by changing the number of FLAME PCA eigenvectors.
  • Dataset scope: FaceScape was processed despite its large scale, but its uncalibrated setup introduces a non-metrical scale that biases predictions.The dataset is therefore not metrically calibrated for this use.
  • Dataset ablation: Leaving out LYHM increases mean error on Stirling from 1.35mm to 1.43mm for HQ images and from 1.46mm to 1.51mm for LQ images.The leave-one-out analysis identifies LYHM as the dataset with the highest influence on reconstruction quality; LYHM contains 1211 subjects.

2 Studies on the Facial Expression Tracking

The metrical face shape prediction initializes facial expression tracking with a perspective camera model, enabling depth evaluation alongside photometric error. On the reported sequence, it improves tracking errors over Face2Face.

  • Tracking setup: The metrical face shape prediction initializes facial expression tracking and uses a perspective camera model to predict depth.The tracking evaluation includes depth and photometric error plots.
  • Tracking results: 5.7mm mean RMSE depth error improves on Face2Face’s 11.0mm mean RMSE depth error.The depth reference maps were reconstructed using a passive stereo system.
  • Tracking results: The method achieves the lowest photometric error under a masked RMSE metric on the colors.Photometric error is computed using an RMSE metric with RGB values in [0, 255].
  • Tracking results: The supplemental video reports temporally stable tracking results across several sequences.

3 Model-free Decoder

The model-free decoder represents face geometry with a SIREN-based coordinate MLP controlled by a mapping network, avoiding dependence on the FLAME PCA basis. It performs on par with the 3DMM-based model but has greater runtime and complexity.

  • Architecture: The model-free decoder uses a SIREN architecture with FiLM conditionings and a mapping network to represent different face geometries.Its sinusoidal activations are controlled by the mapping network, which conditions the geometry representation.
  • Training: The decoder can be trained solely on the unified paired 2D/3D dataset without requiring a 3D morphable model.This dataset is significantly smaller than the 3D-scan dataset used to construct FLAME.
  • Comparison: The model-free approach performs on par with the 3DMM-based model across the different benchmarks.Because it does not rely on FLAME’s PCA basis, it can predict meshes outside the FLAME face space.
  • Trade-offs: The SIREN decoder has greater runtime and complexity than the 3DMM decoder, which represents shape variations with a single linear layer.The SIREN uses 8 hidden layers and 256 feature size, while the 3DMM uses one linear shape-variation layer.
Loading 2204.06607v2…