PaperlayerView exact revision on arXiv

Source-linked AI summary

On Learning 3D Face Morphable Model from In-the-wild Images

Luan Tran, Xiaoming Liu

arXiv:1808.09560v2cs.CV

TL;DR

Conventional 3DMMs depend on costly 3D scans, controlled images, and linear bases that limit representation of unconstrained facial variation. This paper learns a nonlinear 3DMM and its fitting algorithm jointly from in-the-wild 2D images using neural decoders and differentiable rendering. The resulting model improves representation power and performance in face alignment, reconstruction, and editing.

  • Problem

    Conventional 3DMMs rely on 3D scans, controlled 2D images, and linear bases, limiting their representation of in-the-wild facial variation.

  • Method

    A CNN encoder and two nonlinear decoders jointly learn projection, lighting, shape, and albedo through weakly supervised differentiable rendering from in-the-wild 2D images.

  • Results

    The nonlinear 3DMM has greater representation power than the traditional linear counterpart and improves face alignment, face reconstruction, and face editing.

  • Takeaways & Limitations

    3DMM learning can be performed end-to-end from large collections of in-the-wild face images without collecting 3D face scans.

  • Takeaways & Limitations

    The method refrains from supervising lighting and albedo because pseudo-ground-truth estimates are unreliable for in-the-wild images with occlusions.

Abstract

from arXiv · show

As a classic statistical model of 3D facial shape and albedo, 3D Morphable Model (3DMM) is widely used in facial analysis, e.g., model fitting, image synthesis. Conventional 3DMM is learned from a set of 3D face scans with associated well-controlled 2D face images, and represented by two sets of PCA basis functions. Due to the type and amount of training data, as well as, the linear bases, the representation power of 3DMM can be limited. To address these problems, this paper proposes an innovative framework to learn a nonlinear 3DMM model from a large set of in-the-wild face images, without collecting 3D face scans. Specifically, given a face image as input, a network encoder estimates the projection, lighting, shape and albedo parameters. Two decoders serve as the nonlinear 3DMM to map from the shape and albedo parameters to the 3D shape and albedo, respectively. With the projection parameter, lighting, 3D shape, and albedo, a novel analytically-differentiable rendering layer is designed to reconstruct the original input face. The entire network is end-to-end trainable with only weak supervision. We demonstrate the superior representation power of our nonlinear 3DMM over its linear counterpart, and its contribution to face alignment, 3D reconstruction, and face editing.

1 INTRODUCTION

The paper proposes learning a nonlinear 3DMM from large collections of in-the-wild 2D face images without 3D scans, addressing limitations from controlled data and linear bases. It jointly learns the model and fitting algorithm, enabling improved representation and downstream facial-analysis performance.

  • Motivation: Conventional 3DMMs require expensive 3D scans and controlled 2D images, limiting coverage of identity, expression, texture, and unconstrained conditions.Existing models were built from small, laboratory-captured datasets, including early models with 200 subjects and neutral expressions.
  • Motivation: Linear PCA bases inadequately model nonlinear facial variation and out-of-subspace texture such as facial hair.Fitting errors can be absorbed by altering surface normals or shrinking face shape, degrading linear 3DMM-based applications.
  • Research question: The paper asks whether nonlinear shape and albedo 3DMMs can be learned from in-the-wild 2D face images without collecting 3D scans.This question motivates a new learning paradigm based on deep neural networks.
  • Method: Two convolutional decoders replace PCA spaces, mapping shape and albedo parameters to dense 3D meshes and face-skin reflectance.The decoders form the nonlinear 3DMM, while a CNN encoder estimates fitting parameters from an input face image.
  • Method: A differentiable rendering layer fuses predicted shape, albedo, lighting, and projection to reconstruct the input, supporting joint weakly supervised end-to-end learning.The encoder and decoders are trained jointly by minimizing the difference between reconstructed and input faces.
  • Contributions: The nonlinear 3DMM has greater representation power than the traditional linear counterpart and improves face alignment, reconstruction, and editing.The framework also represents shape and albedo as 2D images to preserve spatial relations and leverage CNN-based synthesis.

2 PRIOR WORK

Prior work improves 3DMMs through richer statistical models, in-the-wild texture modeling, nonlinear components, and unsupervised fitting, but remains constrained by PCA bases or limited representations.

  • PCA-based 3DMMs represent shape and texture with linear subspaces and have motivated subsequent efforts to improve their modeling mechanisms.
  • Gaussian Mixture 3DMMs replace the single Gaussian population model with multiple Gaussian subpopulations sharing covariance.
  • Feature-based texture models extend 3DMM texture representation beyond controlled settings toward in-the-wild conditions.
  • Deep Boltzmann Machine face models address linearity but use only 2D faces and sparse landmarks, limiting handling of large poses and occlusions.
  • A corrective model adds learned nonlinear corrections to a linear model, whereas the proposed approach uses a unified model with 2D shape and albedo representations.
  • Prior reconstruction methods lose full 3DMM correspondence or rely on linear-3DMM-synthesized training data, limiting out-of-subspace variations such as facial hair.

3 THE PROPOSED NONLINEAR 3DMM

The proposed nonlinear 3DMM replaces PCA components with deep shape and albedo decoders, jointly learned with an encoder and differentiable rendering layer from image reconstruction. It uses UV-space representations, physically based rendering, perceptual loss, and landmark supervision to preserve spatial structure and improve fitting.

  • 3.2.1 Problem Formulation: Unlike linear 3DMMs based on PCA, the model uses deep networks to increase representation power and learn nonlinear facial variations.The paper views linear factorization as a one-layer network without activation and extends it to a deep architecture.
  • 3.2.1 Problem Formulation: The framework jointly learns an encoder for projection, lighting, shape, and albedo parameters with deep decoders that generate 3D shape and albedo.The rendered output is optimized to reconstruct the input face image.
  • 3.2.2 Albedo & Shape Representation: The model represents albedo as an unwrapped UV-space image and shape as a three-channel UV position map for x, y, and z coordinates.These representations preserve vertex spatial relations and enable CNN decoders.
  • 3.2.2 Albedo & Shape Representation: UV-space shape representation replaces an MLP with a CNN decoder, avoiding wide fully connected layers and potentially modeling more complex shape variations.The paper reports better fitting results from this design in its experiments.
  • 3.2.3 In-Network Physically-Based Face Rendering: The differentiable rendering layer combines albedo, surface normals, lighting, projected shape, and Z-buffer rasterization to reconstruct face images.Lighting is modeled with spherical harmonics using three bands and nine coefficients per color channel.
  • 3.2.5 Model Learning: Perceptual loss enforces closeness between reconstructed and input images, while landmark supervision is required during training but not testing.At test time, landmarks are predicted from the projection parameters and estimated shape.

4 EXPERIMENTAL RESULTS

The experiments use a large in-the-wild face-image dataset to study the nonlinear 3DMM’s expressiveness, representation power, and facial-analysis applications.

  • 4 EXPERIMENTAL RESULTS: The model is trained on 122,450 in-the-wild face images spanning poses from −90° to 90°.Images are cropped and resized to 256 × 256, with random 224 × 224 crops during training.
  • 4 EXPERIMENTAL RESULTS: The study evaluates expressiveness, representation power, and applications to facial analysis using Basel Face Model mesh topology.The 300W-LP dataset supplies the in-the-wild training images.
  • 4 EXPERIMENTAL RESULTS: Training uses Adam with a learning rate of 0.001 in both stages, with lS = lT = 160.The loss weights are chosen to keep losses at similar magnitudes.

4.1 Ablation Study

Ablation studies show that albedo and shape regularization improve disentanglement and plausibility, while lighting and UV-space shape representation improve alignment and reconstruction quality.

  • 4.1.1 Effect of Regularization: Albedo symmetry resolves global lighting, while albedo constancy further removes shading from albedo and improves shape estimation.Without regularization, lighting is largely explained by albedo; combining both constraints yields plausible albedo and lighting.
  • 4.1.1 Effect of Regularization: Shape smoothness prevents noisy facial geometry, especially near the face sides when imprecise hair segmentation leaves hair regions in training.The comparison is made against a model variant without the shape smoothness constraint.
  • 4.1 Ablation Study: The revised model differs from the preliminary work by incorporating lighting and changing the shape representation.The work disentangles shading from albedo rather than modeling texture directly.
  • 4.1 Ablation Study: The 2D UV position-map representation preserves spatial relations among vertices and enables a CNN shape decoder.Earlier representations treated shape as a vector with independent vertices.
  • 4.1 Ablation Study: 4.70 to 4.30 to 4.12: alignment error decreases first with lighting modeling and then with the 2D representation.The values are reported for variants in Tab. 2 on the ALFW2000 face-alignment task.
  • 4.1 Ablation Study: The model-based approach produces sharper reconstructions than a similarly sized convolutional autoencoder and exposes 3D shape, albedo, lighting, and projection parameters.The autoencoder produces blurry results under large variations in appearance, pose, and background.

4.2 Expressiveness

Feature-space and attribute experiments show that the nonlinear 3DMM represents interpretable shape and albedo variation, including small facial details, while improving texture reconstruction.

  • 4.2 Expressiveness: Varying individual shape and albedo parameters reveals how dimensions contribute to generated facial geometry and texture.The feature-space analysis uses approximately 200k CelebA images to estimate empirical parameter distributions.
  • 4.2 Expressiveness: Attribute-conditioned means produce reconstructed textured meshes whose differences appear in both shape and texture.The analysis feeds images with a shared attribute through the encoder and decodes mean shape and albedo parameters.
  • 4.2 Expressiveness: The nonlinear 3DMM models small attribute-linked details such as bags under the eyes and rosy cheeks.These examples illustrate the attribute meaning embedded in the shape and albedo decoders.
  • 4.2 Expressiveness: The nonlinear model better reconstructs facial texture than the linear comparison model.The texture representation comparison is shown in Fig. 13.

4.3 Representation Power

The nonlinear 3DMM is evaluated against linear and bilinear models for texture and 3D-shape reconstruction, with lower reported reconstruction errors and closer recovery of facial details.

  • 4.3 Representation Power: 0.053 vs. 0.062: the nonlinear model has lower averaged L1 texture reconstruction error than the linear model.The nonlinear texture is closer to groundtruth in the comparison using Basel Face Model albedo bases.
  • 4.3 Representation Power: The model fits faces across diverse skin color, pose, expression, lighting, and facial-hair conditions while recovering these cues.Results are shown for AFLW2000 and CelebA.
  • 4.3 Representation Power: 0.0146 vs. 0.0241: the nonlinear model has smaller normalized mean per-vertex shape reconstruction error than the linear model.The comparison uses ten held-out 3D face scans and evaluates vertex distances normalized by inter-ocular distance.
  • 4.3 Representation Power: On BU-3DFE scans with angry, disgust, fear, happy, sad, and surprise expressions, the nonlinear model is compared with FaceWarehouse’s bilinear model.The models are compared on scans with different facial expressions using Chamfer distance because their mesh topologies differ.

4.4 Applications

The learned nonlinear 3DMM supports face fitting, alignment, reconstruction, lighting transfer, and editing from in-the-wild images. Across these applications, it handles challenging texture, shape, pose, and expression variations while retaining facial correspondence and decomposed appearance components.

  • Model fitting: The encoder fits images into shape, albedo, lighting, and projection parameters, recovering personal facial characteristics and facial hair in albedo.This extends nonlinear 3DMM fitting beyond the limited texture representation of linear models.
  • Face alignment: The nonlinear model significantly outperforms linear-3DMM alignment methods and also provides head pose, albedo, and environment lighting.Evaluation uses Normalized Mean Error with bounding-box-size normalization on AFLW2000-3D.
  • 3D face reconstruction: Compared with prior reconstruction approaches, the model better preserves facial hair, nasolabial folds, smooth surfaces, and correspondence under challenging variations.The comparisons cover Tewari et al., Sela et al., and VRN, while the quantitative FaceWarehouse evaluation reports lower error than PRN and 3DDFA+.
  • 3D face reconstruction: On FaceWarehouse, the model faithfully reproduces expressions and surpasses PRN and 3DDFA+ in dense face alignment.The comparison covers all 19 expressions of 150 subjects.
  • 3D face reconstruction: On Florence, the model consistently outperforms other methods across different view angles.The evaluation includes multiple pitch and raw rotations.
  • Lighting transfer: Lighting transfer achieves performance similar to Shu et al. while reducing per-image processing time from 3 minutes to 150 ms.The model transfers lighting from source images to target images.
  • Face editing: Face editing manipulates decomposed components, and residual addition improves editing quality while preserving identity.The method changes attribute directions by different magnitudes before adding the residual to the original input.

5 CONCLUSIONS

The paper argues that conventional 3DMMs are limited by scan requirements, controlled-image training, and linear shape and texture bases. It demonstrates a nonlinear alternative learned from in-the-wild 2D images without genuine 3D scans, with applications across face analysis and editing.

  • Conclusion: Conventional 3DMMs require 3D scans, controlled 2D images, and linear bases that limit representation of unconstrained facial variation.These limitations affect both training requirements and the model's ability to represent shape and texture.
  • Conclusion: The paper demonstrates that a nonlinear 3DMM can be learned from a large set of in-the-wild 2D face images without collecting 3D scans.The authors frame this as an alternative approach to 3DMM learning.
  • Conclusion: The learned model contributes to face alignment, face reconstruction, and face editing, while the experiments combine quantitative results with subjective judgment.The authors ask readers to interpret these results in the context of learning without genuine 3D scans.
  • Conclusion: The work identifies unsupervised or weakly supervised learning of 3D models from large-scale in-the-wild 2D images as a promising research direction.The paper presents itself as one step along this direction.
Loading 1808.09560v2…