Source-linked AI summary
MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction
Ayush Tewari, Michael Zollhöfer, Hyeongwoo Kim, Pablo Garrido, Florian Bernard, Patrick Pérez, Christian Theobalt
TL;DR
Detailed 3D face reconstruction from a single unconstrained image remains an open problem amid variation in capture conditions, pose, expression, and lighting. MoFA combines a CNN encoder with a differentiable generative decoder, producing reconstructions of visually similar quality to Tran et al. [55] while also estimating expression and illumination.
Problem
Detailed 3D face reconstruction from a single arbitrary in-the-wild image remains open amid variation in image quality, pose, expression, and lighting.
Source: §1 · p. 1Method
MoFA combines a convolutional encoder with an expert-designed differentiable decoder that analytically implements image formation and encodes semantic face and rendering parameters.
Source: §3 · p. 3Results
Takeaways & Limitations
The authors report better generalization to real-world data than networks trained on synthetic face data.
Source: §1 · p. 2Reconstructions are limited to the modeled face subspace, and strong occlusions can cause the approach to fail.
Source: §8 · p. 8
Abstract
from arXiv · showhide
In this work we propose a novel model-based deep convolutional autoencoder that addresses the highly challenging problem of reconstructing a 3D human face from a single in-the-wild color image. To this end, we combine a convolutional encoder network with an expert-designed generative model that serves as decoder. The core innovation is our new differentiable parametric decoder that encapsulates image formation analytically based on a generative model. Our decoder takes as input a code vector with exactly defined semantic meaning that encodes detailed face pose, shape, expression, skin reflectance and scene illumination. Due to this new way of combining CNN-based with model-based face reconstruction, the CNN-based encoder learns to extract semantically meaningful parameters from a single monocular input image. For the first time, a CNN encoder and an expert-designed generative model can be trained end-to-end in an unsupervised manner, which renders training on very large (unlabeled) real world data feasible. The obtained reconstructions compare favorably to current state-of-the-art approaches in terms of quality and richness of representation.
1. Introduction
MoFA combines a CNN encoder with an analytically differentiable generative face-model decoder whose code has designed semantic meaning. This enables end-to-end unsupervised reconstruction of detailed faces from unlabeled in-the-wild images.
Introduction
- Detailed reconstruction from one arbitrary in-the-wild image remains open because photos vary in resolution, imaging device, pose, expression, and lighting.Meaningful low-dimensional model parameters also support interpretation and manipulation of reconstructions.Source: §1 · p. 1
- Generative fitting can yield detailed, complete faces with meaningful parameters, but its highly nonconvex objective requires near-optimal initialization, capture control, or extra input such as landmarks.These approaches fit parametric models by optimizing alignment between projected models and images.Source: §1 · p. 1
- CNN regression methods estimate face attributes efficiently and robustly, but typically require densely annotated training data and offer less representation richness than leading generative methods.Richardson et al. [41] [42] estimate pose, shape, expression, and surface detail; Tran et al. [55] estimate shape and skin reflectance, while synthetic training data is challenging to create.Source: §1 · pp. 1–2
- MoFA integrates a CNN encoder with a compact, analytically differentiable decoder built on a detailed parametric 3D face model, ensuring the code vector has exact semantic meaning.The decoder implements the generative image-formation model in closed form and avoids training large sets of unintuitive CNN decoder weights.Source: §1 · p. 2
- A single forward pass estimates pose, shape, expression, skin reflectance, and illumination; end-to-end unsupervised training on unlabeled in-the-wild images uses a dense photometric loss.The authors report better generalization to real-world data than networks trained on synthetic face data [41] [42].Source: §1 · p. 2
2. Related Work
Related work spans parametric fitting, monocular optimization, deep coarse and dense reconstruction, autoencoders, and expert vision layers. MoFA is distinguished by unsupervised dense reconstruction of semantic face parameters from real images without ground-truth model parameters.
Parametric Face Models
- AAMs model 2D shape and texture, while 3DMMs extend parametric face modeling and have supported facial-animation learning from 3D scans.Source: §2 · p. 2
Monocular Optimization-based Reconstruction
- Monocular 3DMM reconstruction and other model-fitting methods contrast with MoFA, which regresses parameters without iterative optimization or initialization from a cropped face.These are the two stated differences from optimization-based approaches.Source: §2 · p. 2
Deep Learning for Coarse Face Reconstruction
- Deep landmark and face-parameter predictors are supervised and sparse, whereas MoFA is unsupervised and obtains dense reconstruction by regressing generative-model parameters.Source: §2 · p. 2
Deep Learning for Dense Face Reconstruction
- Existing dense deep face methods require annotated training data, often synthetic when labels are costly; MoFA instead uses real images without ground-truth model parameters.The paper notes that synthetic renderings often lack realistic features, negatively affecting reconstruction accuracy.Source: §2 · p. 3
Autoencoders
- Unlike approaches that use known-varying image groups to disentangle codes, MoFA considers the full set of meaningful parameters without grouping images by known variations.The paper describes autoencoders as learning compact code vectors and notes their generally unsupervised character.Source: §2 · p. 3
Deep Integration of Expert Layers
- Unlike geometric vision layers and single-view Perspective Transformer Nets, MoFA reconstructs semantically meaningful facial geometry, expression, illumination, and camera extrinsics.Source: §2 · p. 3
3. Overview
MoFA combines a convolutional encoder with a differentiable, expert-designed decoder to learn semantically meaningful face and rendering parameters end to end without semantic-parameter supervision. After training, the encoder regresses a dense face model and illumination from a single monocular image.
3. Overview
- The differentiable decoder analytically implements image formation and independently parameterizes pose, shape, expression, skin reflectance, and illumination.It generates a realistic synthetic face image and enforces semantic meaning through a parametric face prior.Source: §3 · p. 3
- Unsupervised end-to-end training compares synthesized and input images with a robust photometric loss that includes statistical face regularization.Optional 2D facial landmarks add a surrogate loss for faster convergence and improved reconstructions; neither scenario requires supervision of semantic parameters.Source: §3 · p. 3
- After training, the encoder regresses a dense face model and illumination from a single monocular image without requiring landmarks or other inputs.Source: §3 · p. 3
4. Semantic Code Vector
The 257-dimensional semantic code vector jointly parameterizes facial expression, shape, skin reflectance, camera pose, and scene illumination. The face model uses PCA-based representations for geometry and per-vertex reflectance.
Semantic code vector
- The 257-dimensional vector encodes expression (64), shape (80), skin reflectance (80), camera rotation and translation (6), and illumination (27).Camera rotation is represented in SO(3), and translation in R^3.Source: §4 · p. 3
Face representation
- The face is represented by a triangle mesh with 24k vertices, whose normals are computed from local one-ring neighborhoods.Source: §4 · p. 3
Shape and expression
- Shape and expression use 80- and 64-dimensional PCA bases; the expression basis, derived from blendshapes and, covers over 99% of their variance.The average shape uses 200 high-quality scans, and the blendshapes were retargeted to that topology using deformation transfer.Source: §4 · p. 4
Skin reflectance
- Per-vertex skin reflectance uses an affine model with an average reflectance based on and an 80-dimensional orthogonal PCA basis for its highest-variation modes.All basis vectors are scaled by the appropriate standard deviations.Source: §4 · p. 4
5. Parametric Model-based Decoder
Given a semantic code vector, the decoder analytically renders a synthetic face image, and its differentiable image formation enables unsupervised end-to-end training.
Backpropagation
- An analytical, differentiable backward pass inverts image formation via backpropagation, enabling unsupervised end-to-end training.It computes image-formation gradients with respect to face and scene parameters in a data-parallel manner for training efficiency.Source: §5 · p. 4
Perspective Camera
- A pinhole camera applies full perspective projection after transforming world-space points into camera space using rotation T and translation t.The mapping is Π ◦ ΦT,t(v), where ΦT,t(v) = T−1(v − t).Source: §5 · p. 4
Illumination Model
- Illumination uses spherical harmonics for distant, low-frequency lighting and assumes purely Lambertian surface reflectance.Three bands provide nine RGB coefficient vectors to parameterize colored illumination.Source: §5 · p. 4
Image Formation
- The forward pass computes each face vertex’s screen-space position and pixel color using the camera and illumination models.World-space normals are transformed into camera space, where illumination is evaluated.Source: §5 · p. 4
6. Loss Layer
The loss combines dense photometric alignment, statistical regularization, and an optional landmark surrogate to train the encoder end to end. During backpropagation, gradients pass through the model-based decoder using the chain rule.
6. Loss Layer
- The robust dense photometric loss combines landmark alignment, photometric alignment, and regularization, with wphoto = 1.92 and wreg = 2.9 × 10−5 balancing the objectives.The landmark term is toggled by binary wland ∈ {0, 1}.Source: §6 · p. 5
Dense Photometric Alignment
- Dense photometric alignment uses a robust ℓ2,1-norm per vertex, iterating over front-facing vertices computed during the forward pass for occlusion awareness.It aims to make the synthetic face image match the monocular training image.Source: §6 · p. 5
Sparse Landmark Alignment
- The optional surrogate landmark loss uses 46 of 66 detected landmarks to speed convergence, but training can be fully unsupervised and requires no landmarks after training.It weights detected 2D landmarks by confidence and aligns their corresponding projected 3D model vertices.Source: §6 · p. 5
Statistical Regularization
- Statistical regularization favors average facial shape, expression, and skin reflectance, using wβ = 1.7 × 10−3 and wδ = 0.8, while leaving pose and illumination unregularized.The face-model basis is scaled by standard deviations.Source: §6 · p. 5
Backpropagation
- During backpropagation, the robust-loss gradient passes to the model-based decoder and combines with Bi(x) via the chain rule to enable stochastic gradient descent.Source: §6 · p. 5
7. Results
The model-based autoencoder achieves low-error reconstructions and competitive quality against prior methods while producing semantic estimates of face parameters, reflectance, and illumination. Unsupervised training and the landmark-based surrogate task further improve reconstruction results.
Comparison to Richardson et al. [41, 42]
- Compared with Richardson et al. [41] [42], the approach obtains higher reconstruction quality and regresses colored reflectance and illumination rather than optimizing grayscale reflectance.The approach is trained unsupervised on real images, whereas [41] [42] use a synthetic training corpus.Source: Figure 4 · p. 6, §7 · p. 6
Comparison to Thies et al. [54]
- Compared with Thies et al., the approach achieves similar or higher quality and runs in 4ms rather than approximately 500ms to fit all parameters from the average model.At test time it requires only face detection, whereas Thies et al. require detected landmarks.Source: Figure 6 · p. 6, §7 · pp. 6–7
Comparison to Tran et al. [55]
- On LFW face verification, the approach reaches 77% accuracy versus the monocular 3DMM baseline at 75%, while Tran et al. [55] report 92%.Reconstructions are visually similar to Tran et al. [55], while also estimating facial expression and illumination; Tran et al. require a photo collection for training.Source: §7 · p. 6, Figure 5 · p. 6
Comparison to Garrido et al. [14]
- The approach obtains comparable quality to Garrido et al. without requiring landmarks, which can help avoid local minima in optimization-based reconstruction.The comparison uses an implementation with coarse layer, photometric, landmark, and regularization terms and 50 Gauss-Newton steps.Source: §7 · p. 7
Evaluation of Different Encoders
- VGG-Face slightly outperforms AlexNet, with landmark error of 4.9 pixels versus 5.3 pixels and photometric error of 0.073 versus 0.075.The photometric error is color distance in RGB space, with each channel in [0, 1].Source: §7 · p. 7, Figure 8 · p. 7
Quantitative Evaluation
- On 5k rendered images, geometric error fell from 21.6mm to 4.5mm, photometric error from 0.33 to 0.05, and landmark error from 31.6 pixels to 3.9 pixels.On 9 Facewarehouse identities (180 images), errors were on par with optimization-based methods, with 4ms inference versus 1min and no test-time landmarks.Source: Figure 10 · p. 7, §7 · p. 8, Figure 11 · p. 8
Quantitative Evaluation of Unsupervised Training
- Unsupervised training decreases dense photometric and landmark error on real validation images even without landmark alignment in the loss; the surrogate reduces landmark error and improves robustness to occlusions and expression quality.With the surrogate, landmark errors are AlexNet: 3.7 pixels vs. 5.3 pixels and VGG-Face: 3.4 pixels vs. 4.9; photometric errors are AlexNet: 0.078 vs. 0.075 and VGG-Face: 0.078 vs. 0.073.Source: §7 · p. 7, Figure 12 · p. 7, Figure 9 · p. 7
Comparison to Autoencoders and Learned Decoders
- Compared with convolutional autoencoders and learned decoders, the model-based approach produces sharper, higher-fidelity images and exposes semantic parameters for dense geometry, reflectance, and illumination.Explicit disentanglement of a convolutional autoencoder requires labeled ground truth data.Source: §7 · p. 8, Figure 13 · p. 8, Figure 14 · p. 8
8. Limitations
The method can produce implausible reconstructions outside its training-data span, is limited to the face model’s subspace, and struggles with strong occlusions and head rotations. Unmodeled backgrounds can also cause slight shrinking.
8. Limitations
- Implausible reconstructions can occur outside the training-data span, though enlarging the corpus can alleviate this in the unsupervised setting.The authors note that expanding the training corpus is easy in their unsupervised approach.Source: §8 · p. 8
- Reconstructions are limited to the subspace represented by the face model.Source: §8 · p. 8
- Strong occlusions, such as facial hair or external objects, can cause failure, and strong head rotations remain challenging.Unsupervised occlusion-aware training is identified as an open research problem.Source: §8 · p. 8
- Because the background is not modeled, reconstructions can shrink slightly; this issue is discussed and addressed in.Source: §8 · p. 8
9. Conclusion
MoFA introduces a deep convolutional model-based face autoencoder that learns meaningful semantic parameters through unsupervised, end-to-end training with a differentiable parametric decoder.
9. Conclusion
- The work presents the first deep convolutional model-based face autoencoder that can be trained unsupervised and learns meaningful semantic parameters.Source: §9 · p. 8
- Its code vector represents pose, shape, expression, skin reflectance, and illumination through an expert-designed parametric model.Source: §9 · p. 8
- A fully differentiable model-based decoder enables end-to-end learning of the network.Source: §9 · p. 8
— Supplemental Material —
MoFA’s encoder learns to predict semantic pose, shape, expression, reflectance, and lighting parameters from a single monocular image in an unsupervised manner. The supplement adds qualitative results and evaluations of reconstruction comparisons, training behavior, encoders, and limitations.
Synthetic results
- On synthetic ground-truth data, MoFA obtains good fits for all parameters.Source: Figure 1 · p. 11
Supplement overview
- The supplement presents sample images from the real-world training corpus and additional evaluations of surrogate tasks, encoders, convergence, limitations, and reconstruction results.It also includes further qualitative results and comparisons to optimization-based and learning-based monocular reconstruction approaches.Source: — Supplemental Material — · p. 11, Figure 3 · p. 11
Additional comparisons
- Against the offline reconstruction approach, MoFA achieves similar quality without landmark inputs, while often gets stuck in a local minimum without them.Source: — Supplemental Material — · p. 11
Training convergence
- On test images unseen during training, MoFA learns facial variation from the average face in an unsupervised manner.Source: Figure 8 · p. 12
Additional comparisons
- On 300-VW [2, 11, 13] and LFW [4], MoFA achieves higher reconstruction quality than Richardson et al. [8, 9], while estimating colored reflectance and illumination.The comparison uses Richardson et al.’s coarse network without refinement; its greyscale reflectance is obtained by optimization, whereas MoFA regresses all parameters at once.Source: Figure 9 · p. 12