Source-linked AI summary
Nonlinear 3D Face Morphable Model
Luan Tran, Xiaoming Liu
TL;DR
Conventional 3DMMs are limited by controlled training data, scarce 3D scans, and linear bases. The paper learns a nonlinear 3DMM and fitting algorithm from unconstrained 2D images with weak supervision and differentiable rendering. It reports stronger representation power and improved face alignment and reconstruction.
Problem
Conventional 3DMMs rely on small controlled datasets with 3D scans, while linear PCA bases inadequately represent nonlinear facial variation and in-the-wild texture.
Method
An encoder estimates projection, shape, and texture parameters; neural decoders generate 3D shape and texture, and differentiable rendering enables weakly supervised end-to-end reconstruction.
Results
The nonlinear 3DMM has greater representation power than its linear counterpart and improves face alignment and face reconstruction.
Takeaways & Limitations
Nonlinear 3DMMs can be learned from large collections of unconstrained 2D face images without collecting 3D face scans.
Takeaways & Limitations
During landmark-loss optimization, the shape decoder is fixed because updating it would move only a small subset of vertices and could produce unrealistic shapes.
Abstract
from arXiv · showhide
As a classic statistical model of 3D facial shape and texture, 3D Morphable Model (3DMM) is widely used in facial analysis, e.g., model fitting, image synthesis. Conventional 3DMM is learned from a set of well-controlled 2D face images with associated 3D face scans, and represented by two sets of PCA basis functions. Due to the type and amount of training data, as well as the linear bases, the representation power of 3DMM can be limited. To address these problems, this paper proposes an innovative framework to learn a nonlinear 3DMM model from a large set of unconstrained face images, without collecting 3D face scans. Specifically, given a face image as input, a network encoder estimates the projection, shape and texture parameters. Two decoders serve as the nonlinear 3DMM to map from the shape and texture parameters to the 3D shape and texture, respectively. With the projection parameter, 3D shape, and texture, a novel analytically-differentiable rendering layer is designed to reconstruct the original input face. The entire network is end-to-end trainable with only weak supervision. We demonstrate the superior representation power of our nonlinear 3DMM over its linear counterpart, and its contribution to face alignment and 3D reconstruction.
1. Introduction
Conventional 3DMMs are constrained by limited, controlled training data and linear PCA bases. This paper learns a nonlinear 3DMM and its fitting algorithm from unconstrained 2D images using weak supervision and differentiable rendering.
- 3DMM models 3D facial shape and texture with explicit correspondences, supporting morphing and modeling transformations between face types.
- Conventional models require expensive 3D scans and typically rely on small, controlled datasets with limited identity, expression, and texture variation.Earlier models used fewer than 300 training scans and controlled 2D images, limiting coverage of facial variability and in-the-wild texture.
- PCA-based linear formulations cannot represent facial expression and pose variations well because those variations are nonlinear.
- The paper asks whether nonlinear face shape and texture models can be learned from unconstrained 2D images without collecting 3D scans.
- Two neural decoders replace PCA spaces for shape and texture, while an encoder estimates parameters and a differentiable renderer reconstructs the input face.The shape decoder uses an MLP and the texture decoder uses a CNN; the encoder also estimates camera projection parameters.
- The proposed model has greater representation power than the traditional linear counterpart and improves face alignment and face reconstruction.
2. Prior Work
Prior work includes linear and multilinear 3DMMs, 2.5D refinement methods, and encoder-decoder approaches for 3D face reconstruction. The paper distinguishes its approach by jointly learning a nonlinear 3DMM and fitting algorithm from unconstrained 2D images.
- Linear 3DMM: Earlier 3DMM research improved modeling through scan alignment and multilinear models for identity and expression variation.
- Some 3DMM-based refinement methods add facial details but learn only 2.5D depth maps, losing the correspondence property of 3DMM.
- Recent encoder-decoder work reconstructs 3D faces, while this paper focuses on a nonlinear 3DMM learned jointly with its fitting procedure.
3. Proposed Method
The proposed nonlinear 3DMM replaces PCA shape and texture spaces with learned deep decoders, while an encoder estimates parameters and a differentiable renderer reconstructs input images. It uses unconstrained 2D images, unwrapped texture representations, and weak supervision to support end-to-end learning.
- Nonlinear 3DMM: The nonlinear model replaces shallow linear factorization with deep networks to increase representation power and learn from large-scale in-the-wild 2D face images.Two decoders model shape and texture, while an encoder estimates their parameters and the projection.
- Problem Formulation: The encoder maps a face image to projection, shape, and texture parameters, and the decoders map shape and texture parameters to a 3D face and texture.The rendered output is trained to approximate the original input image.
- Shape & Texture Representation: An unwrapped 2D texture preserves side-view information while enabling convolutional processing, unlike per-vertex vectors or frontal-face textures.Vertices are projected into UV space by cylindrical unwrapping, and the texture decoder uses fractionally-strided convolutions.
- In-Network Face Rendering: The rendering layer samples texture values, projects the 3D mesh, and uses Z-buffered barycentric interpolation to produce each image pixel.The closest triangle handles occlusions, and the rendered image is defined on a 2D grid suitable for adversarial loss.
- Network Architecture: End-to-end training combines image reconstruction, adversarial, and landmark losses, with pseudo-groundtruth textures generated by backward rendering from fitted 3DMM outputs.Manual 2D landmarks provide golden groundtruth throughout training, while the shape decoder is learned jointly.
4. Experimental Results
Experiments evaluate the nonlinear 3DMM’s expressiveness, representation power, and applications. It models attribute-linked shape and texture, improves reconstruction over linear models, and supports face alignment and 3D reconstruction.
- Expressiveness: Varying individual latent dimensions produces ordered shape and texture changes, with leading shape variations often related to facial expression.The empirical feature-space analysis uses approximately 200k CelebA images and ranks shape changes across 160 elements.
- Expressiveness: Shape and texture reconstructions vary with shared facial attributes in the learned nonlinear model.The mean shape and texture decoded with selected attributes show differences present in both modalities.
- Representation Power: 0.066 vs. 0.103 L1 reconstruction error: nonlinear texture reconstructs facial texture more accurately than the linear model, especially for in-the-wild images.The comparison uses the linear model trained on controlled images and evaluates texture reconstruction against pseudo-groundtruth texture.
- Representation Power: 0.0196 vs. 0.0241 NME: nonlinear shape reconstruction has significantly smaller error than the linear model.Nonlinear models also achieve similar performance with latent spaces half the size of the compared linear models.
- Applications: The full network recovers personal facial characteristics, including varied skin color and facial hair, from unconstrained face images.The encoder estimates shape, texture, and projection parameters for 3DMM fitting.
- Ablation on Texture Learning: PatchGAN produces higher realism and fewer artifacts than the other tested adversarial-loss options for texture learning.The authors report that a global image-based discriminator can cause severe artifacts, whereas the rendering layer already guarantees global structure.
5. Conclusions
The conclusion presents nonlinear 3DMM learning from unconstrained face images without collecting 3D scans as an alternative to conventional learning. It jointly learns model fitting and identifies unsupervised learning of 3D models from large-scale in-the-wild images as a promising direction.
- Conclusions: A nonlinear 3DMM can be learned from unconstrained face images without collecting 3D face scans.The paper frames this as an alternative approach to conventional 3DMM learning.
- Conclusions: The model fitting algorithm can be learned jointly with the nonlinear 3DMM in an end-to-end fashion.The conclusion emphasizes joint learning as part of the proposed alternative.
- Conclusions: Unsupervised learning of 3D models from large-scale in-the-wild 2D images is identified as one promising research direction.The authors describe this work as one step along that direction.