PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Training for 3D Morphable Model Regression

Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, William T. Freeman

arXiv:1806.06098v1cs.CV

TL;DR

The paper addresses the lack of ground-truth 3D face data and the difficulty of inverse rendering by training a regression network from unlabeled photographs. It uses facial-recognition features from differentiably rendered predictions, with three losses to prevent feature-space fooling. The method improves prior regression accuracy and produces recognizable neutral 3D faces, while remaining below the ground-truth recognition baseline and depending on renderer approximations.

  • Problem

    Regression-based 3D face fitting lacks ground-truth 3D training data, while inverse rendering is difficult and unreliable under general in-the-wild conditions.

  • Method

    The method trains an encoder-decoder regression network on unlabeled photographs and synthetic morphable-model faces, using differentiable rendering, facial-recognition features, and batch, loopback, and multi-view identity losses.

  • Results

    The method improves the accuracy of previous supervised methods, exceeds other regression approaches by a large margin, and produces recognizable neutral 3D faces.

  • Takeaways & Limitations

    Unlabeled photographs can train a regression network that predicts neutral 3D morphable-model coordinates suitable for recognizable-face reconstruction and landmark-based facetracking.

  • Takeaways & Limitations

    The approach does not yet reach the ground-truth recognition baseline, and its renderer approximation could cause problems when losses depend primarily on translation or occlusion.

Abstract

from arXiv · show

We present a method for training a regression network from image pixels to 3D morphable model coordinates using only unlabeled photographs. The training loss is based on features from a facial recognition network, computed on-the-fly by rendering the predicted faces with a differentiable renderer. To make training from features feasible and avoid network fooling effects, we introduce three objectives: a batch distribution loss that encourages the output distribution to match the distribution of the morphable model, a loopback loss that ensures the network can correctly reinterpret its own output, and a multi-view identity loss that compares the features of the predicted 3D face and the input photograph from multiple viewing angles. We train a regression network using these objectives, a set of unlabeled photographs, and the morphable model itself, and demonstrate state-of-the-art results.

1. Introduction

The paper replaces difficult inverse rendering and scarce 3D supervision with unsupervised regression guided by facial-recognition features and differentiable rendering. Three additional losses prevent feature-space fooling, and the resulting predictions improve prior accuracy despite using unlabeled photographs.

  • Motivation: 3D morphable models provide a low-dimensional face space, but conventional inverse rendering requires difficult nonlinear optimization over shape, texture, pose, and lighting.Regression is faster and more robust, yet lacks ground-truth 3D face data because scans are expensive and raise privacy concerns.
  • Approach: The method trains a regression network from unlabeled photographs using facial-recognition features rather than pixel reconstruction or supervised 3D targets.Predicted faces are rendered differentiably, and identity features are matched between the photograph and rendering.
  • Problem: Recognition-feature similarity alone can produce unnatural faces that fool the recognition network.This motivates explicit regularization beyond identity-feature matching.
  • Approach: Three losses address fooling: batch distribution matching, loopback reinterpretation, and multi-view identity comparison across independent predicted views.Together they constrain output statistics, self-consistency, and identity preservation under viewpoint changes.
  • Result: Despite learning from unlabeled photographs, the method improves previous accuracy and often produces faces recognizable as the original subjects.Training uses a face-recognition network, a morphable face model, and unlabeled face images.

2. Related Work

Prior work established morphable-model fitting and regression, but faced optimization instability, limited training data, synthetic-to-real gaps, or entanglement among facial factors. The paper instead combines real and synthetic data with recognition-based verification in an unsupervised framework.

  • Morphable 3D Face Models: Morphable-model reconstruction decomposes geometry and texture into reduced-dimensional PCA spaces, while later models add expression as a separate space.The paper uses PCA weights as regression outputs.
  • Morphable 3D Face Models: Iterative fitting remains unreliable in in-the-wild conditions because convergence depends on initialization and scene complexity.Scene complexity includes lighting, expression, and pose.
  • Regression Approaches: Neural regression avoids iterative fitting but requires enough training data, which is difficult to collect for 3D face parameters.This is the central data problem addressed by unsupervised training.
  • Regression Approaches: Synthetic training pairs provide ground-truth coordinates but can generalize poorly to occlusions, unusual lighting, and underrepresented ethnicities.The paper therefore mixes synthetic faces with real photographs during training.
  • Unsupervised Alternatives: Pixel-based autoencoding on unlabeled photographs can confound related variables such as skin tone and lighting, whereas recognition features compare information across the whole image.Other approaches also predict multiple factors, including shape, expression, texture, pose, and lighting.
  • Training Framework: The framework uses hybrid batches of real and synthetic faces, applying unsupervised losses to real images and 3DMM parameter loss to synthetic images.The identity encoder remains fixed during training.

3. Model

The model uses an encoder-decoder system with a differentiable renderer to learn neutral 3D geometry and texture from varied face images. Its central design isolates the training objective to identity across changing appearance conditions.

  • Architecture: The encoder-decoder architecture enables end-to-end unsupervised learning of 3D geometry and texture morphable-model parameters.A parameterized illumination model and differentiable renderer generate neutral-expression images under varying pose and lighting.
  • Training Data: Training combines real VGG-Face photographs with synthetic Basel Face 3DMM faces in hybrid batches.Synthetic examples provide morphable-model supervision while real images support unsupervised objectives.
  • Design Principle: The approach isolates its loss to identity, preserving identity across expression, pose, and illumination while avoiding network fooling.The authors report robust state-of-the-art recognizability in the predicted faces.

3.1. Encoder

The encoder uses FaceNet features to represent input images for regression, while a different FaceNet layer supplies the identity loss. The selected layers reflect different roles in decoding and identity comparison.

  • Feature Encoder: FaceNet supplies the encoder features, while VGG-Face is identified as another network that may support the identity-preserving loss.The paper uses FaceNet because its features have been effective for generating face images.
  • Feature Encoder: The encoder input is the 1024-D avgpool representation from FaceNet’s NN2 architecture.This penultimate layer was more effective for the decoder than the final normalized representation.
  • Feature Encoder: The final 128-D normalizing layer is reserved for the identity loss rather than decoder input.Thus, the architecture separates features used to decode morphable-model parameters from features used to compare identity.

3.2. Decoder

The decoder predicts Basel Face Model shape and texture parameters, which a differentiable rasterizer converts into rendered faces for training. Its renderer supports perspective and screen-space lighting while approximating derivatives locally away from occlusion boundaries.

  • Decoder: The Basel Face Model represents shape, texture, and expression with PCA-based parameter vectors and bases.Shape and texture vectors have 199 dimensions each, while the expression vector has 100 dimensions.
  • Decoder: The decoder predicts 398 shape and texture parameters, while the expression vector is fixed to zero.It uses two 1024-unit fully connected ReLU layers followed by a 398-unit regression layer.
  • Differentiable Renderer: The differentiable rasterizer interpolates per-vertex attributes at pixels using triangle IDs and barycentric coordinates.This supports full perspective and screen-space lighting models for training.
  • Differentiable Renderer: Rasterization derivatives omit triangle-ID derivatives and extend barycentric derivatives beyond triangle borders, treating faces as locally planar.The approximation is effective away from occlusion boundaries but may cause problems when translation or occlusion drives the loss.
  • Differentiable Renderer: Deferred shading computes illumination per pixel, using the Phong reflection model to improve realism over diffuse-only approximations.Two randomly positioned point lights and varied color temperatures provide lighting during training.

3.3. Losses

The training objective combines parameter, identity, batch-distribution, and loopback losses to train on synthetic and real faces without known real-image 3D parameters. Identity features are matched across rendered views, while auxiliary losses reduce unrealistic outputs and improve robustness.

  • Losses: The loss combines parameter similarity, identity preservation, batch-distribution regularization, and loopback consistency.The batch and loopback weights used for the reported results are ωbatch = 10.0 and ωloop = 0.07.
  • Training: Training first uses synthetic faces with known parameters, then fine-tunes the partially trained model to convergence on batches combining real and synthetic images.Synthetic training alone cannot capture correlations between independently sampled shape and texture parameters.
  • 3.3.2 Identity Loss: Face-recognition features provide an identity loss that is robust to expression, pose, and illumination variation.The loss compares cosine similarity between FaceNet embeddings of the photograph and a randomly posed, illuminated rendering.
  • 3.3.2 Identity Loss: Using three random poses per real face improves identity training by reducing occluded mesh regions and providing multiple vertex-level backpropagation paths.The figure contrasts this multi-view setting with a single frontal view.
  • 3.3.3 Batch Distribution Loss: The batch-distribution loss matches the predicted shape and texture distributions to zero mean and unit variance.It regularizes the lowest two moments through separate mean and variance terms for shape and texture.
  • 3.3.4 Loopback Loss: The loopback loss re-encodes randomly rendered versions of predicted real-face outputs and penalizes parameter differences from the first prediction.Its two backpropagation paths generalize near real-face parameters and discourage unrealistic decoder outputs.

4. Experiments

Experiments show that the method produces more accurate, consistent, recognizable, and condition-robust 3D face reconstructions than prior approaches across qualitative and quantitative evaluations.

  • 4.1. Qualitative Comparison: The method improves likeness, eyebrow texture, nose shape, coloration, and skin-tone fidelity over previous approaches on MoFA-Test.It is also more resistant to identity–expression and skin-tone–lighting confounding than MoFA.
  • 4.2. Neutral Pose Reconstruction on MICC: 20-25% lower absolute error is achieved against MICC ground-truth scans, with less than half Tran et al.’s standard deviation.Results remain similar across cooperative, indoor, and outdoor test conditions after ICP alignment with isotropic scale.
  • 4.3. Face Recognition Results: The method reaches an average VGG-Face photo-rendering similarity of 0.403 on MoFA-Test.This exceeds the similarity of 22.7% of same-person LFW photo pairs and nearly all different-person pairs, of which 0.04% score above 0.403.
  • 4.3. Face Recognition Results: Its similarity distributions are closer to same-person than different-person LFW distributions, whereas competing methods are closer to the different-person distribution.The comparison uses Earth Mover’s distance between VGG-Face similarity distributions.
  • 4.4. Face Clustering: Identity clustering achieves 87% Top-1 recall on MoFA-Test versus 25% for Tran et al. and 19% for MoFA, and 51% Top-5 recall on LFW.MoFA-Test contains 84 images and 78 identities, while LFW contains 12,993 images and 5,749 identities.
  • 4.5. Robustness and Non-Photorealistic Inputs: The reconstructions remain robust to changes in pose, expression, lighting, occlusion, and resolution, and can produce plausible likenesses from non-photorealistic artwork.These evaluations use consistent outputs for varied conditions and recognition-focused features rather than pixel-level matching.

5. Discussion and Future Work

The method learns neutral, expressionless 3D morphable-model faces from unlabeled photographs, improving on prior regression and supervised approaches. Its scope and performance vary with predicted factors and demographic representation.

  • Unlabeled photographs suffice to train neutral, expressionless 3D morphable-model regression that improves on supervised methods.
  • The predicted face can be used directly for landmark-based face tracking.
  • The current system focuses on expressionless faces rather than pose, expression, or lighting.Future work proposes adding an inverse-rendering stage to predict those factors while retaining neutral-pose losses.
  • Performance is generally superior for young adults and Caucasian ethnicities, with possible causes including scan, feature, and training-data representation.The authors identify improving performance across ages and ethnicities as future work.

A. Appendix

The appendix evaluates robustness and qualitative reconstruction across lighting, pose, expression, occlusion, resolution, viewpoints, and benchmark examples. Results indicate stability in several conditions but degradation under severe pose, occlusion, and blur.

  • Pose Stress Test: The pose stress test remains consistent under 45° rotation, but global shape changes under 90° rotation.
  • Expression Stability Test: The expression stability test finds the method invariant to expression while remaining sensitive to identity.
  • Occlusion Stress Test: Increasing occlusion causes facial features to degrade smoothly once necessary input information is no longer available.
  • Resolution Stress Test: Progressive blur removes fine eyebrow detail, while extreme blur can subtly alter global shape for the female subject.

A.1. Fitting Pose and Expression

The system reconstructs shape and texture while omitting pose, expression, and lighting, then uses landmark-based optimization to fit pose and expression to the source image.

  • The system reconstructs face shape and texture while ignoring pose, expression, and lighting.These omitted components are needed to exactly match the reconstruction to the source image.
  • Gradient descent starts from the predicted face and fits pose and expression by minimizing landmark distances between the mesh and image.The procedure uses the 68-landmark configuration from the Multi-PIE database.
  • Figure 18 illustrates the sequence from landmarked input through neutral reconstruction to shaded-geometry and albedo overlays with fitted pose and expression.
Loading 1806.06098v1…