Source-linked AI summary
MVF-Net: Multi-View 3D Face Morphable Model Regression
Fanzi Wu, Linchao Bao, Yajing Chen, Yonggen Ling, Yibing Song, Songnan Li, King Ngi Ngan, Wei Liu
TL;DR
Single-view 3DMM reconstruction can leave face geometry ambiguous because it lacks reliable 3D constraints. The paper regresses 3DMM parameters from multiple views with an end-to-end CNN and differentiable cross-view alignment losses. In an ablation, combining photometric and alignment losses gives the best results among the tested variants.
Problem
Single-view 3DMM reconstruction lacks reliable 3D constraints and can leave facial geometry, such as nose and cheekbone height, ambiguous.
Source: §1 · p. 1Method
An end-to-end CNN regresses 3DMM parameters from multiple views, using rendered cross-view projections and differentiable optical-flow alignment errors.
Source: §1 · pp. 1–2Results
Combining photometric and alignment losses gives the best ablation results, with error reductions of 0.046 and 0.024.
Source: §4.3 · p. 6Takeaways & Limitations
The experiments validate the effectiveness of incorporating multi-view geometric constraints into 3DMM regression with a CNN.
Source: §5 · p. 8The photometric consistency assumption applies to non-occluded facial regions, where the observed and rendered images should match in the ideal case.
Source: §3.5 · p. 4
Abstract
from arXiv · showhide
We address the problem of recovering the 3D geometry of a human face from a set of facial images in multiple views. While recent studies have shown impressive progress in 3D Morphable Model (3DMM) based facial reconstruction, the settings are mostly restricted to a single view. There is an inherent drawback in the single-view setting: the lack of reliable 3D constraints can cause unresolvable ambiguities. We in this paper explore 3DMM-based shape recovery in a different setting, where a set of multi-view facial images are given as input. A novel approach is proposed to regress 3DMM parameters from multi-view inputs with an end-to-end trainable Convolutional Neural Network (CNN). Multiview geometric constraints are incorporated into the network by establishing dense correspondences between different views leveraging a novel self-supervised view alignment loss. The main ingredient of the view alignment loss is a differentiable dense optical flow estimator that can backpropagate the alignment errors between an input view and a synthetic rendering from another input view, which is projected to the target view through the 3D shape to be inferred. Through minimizing the view alignment loss, better 3D shapes can be recovered such that the synthetic projections from one view to another can better align with the observed image. Extensive experiments demonstrate the superiority of the proposed method over other 3DMM methods.
1. Introduction
Single-view 3DMM regression can leave face geometry ambiguous, while existing multi-view fitting approaches have practical drawbacks. This paper proposes an end-to-end CNN that regresses 3DMM parameters from multiple views and uses differentiable view alignment to incorporate cross-view constraints.
1. Introduction
- Single-view 3DMM regression lacks reliable 3D constraints, leaving features such as nose and cheekbone height ambiguous from a frontal view.The paper also notes that multi-view pipelines based on SfM/MVS followed by fitting can be error-prone, while direct multi-view optimization can be difficult to solve.Source: §1 · p. 1
- The proposed approach uses an end-to-end trainable CNN to regress 3DMM parameters from multi-view facial images.It is presented as an end-to-end way to exploit both 3DMM and multi-view geometric constraints.Source: §1 · pp. 1–2
- A differentiable dense optical-flow estimator supplies view-alignment errors that backpropagate through rendered cross-view projections to the inferred 3D model.The method samples textures from an input view, renders the textured model into another view, and compares the rendering with the target image.Source: §1 · p. 2
2. Related Work
The related work frames 3DMMs as parametric face representations and reviews single- and multi-view reconstruction approaches. It distinguishes this paper's use of multi-view geometric constraints from prior multi-view methods that do not exploit them.
2.1. Morphable 3D Face Model (3DMM)
- 3DMM represents textured 3D faces using linear combinations of shape and texture bases derived from real 3D face scans.FaceWarehouse later extended the model to include facial expressions; this paper focuses on shape and expression parameters rather than 3DMM textures.Source: §2.1 · p. 2
2.2. Single-view 3DMM-based Reconstruction
- Single-view 3DMM fitting has used analysis-by-synthesis optimization, while CNN approaches have used fitted, scan-based, synthetic, or self-supervised training signals.Optimization can be sensitive to initialization and parameters; the paper also notes that richer geometric representations are outside its scope.Source: §2.2 · p. 2
2.3. Multi-view 3DMM-based Reconstruction
- Prior multi-view approaches include SfM/MVS followed by 3DMM fitting and CNN/RNN feature fusion, but the latter does not exploit multi-view geometric constraints.Other multi-image methods cited here process each image individually rather than in the multi-view setting addressed by this paper.Source: §2.3 · p. 2
3.1. Overview
The overview describes a CNN that takes multiple facial views and regresses a shared 3DMM shape alongside a pose for each view. Rendered cross-view projections are compared with target images using losses, and the rendering layer is differentiable.
3.1. Overview
- The three-view description assumes images of the same face are taken at the same time under the same lighting, and says the approach can generalize to other numbers of views.The paper later illustrates handling lighting variance across views.Source: §3.1 · pp. 2–3
- The model learns features from each input with a shared-weight CNN, concatenates them to regress one set of 3DMM parameters, and regresses pose parameters separately for each view.In the illustrated three-view setting, each textured model shares the underlying shape but uses textures sampled from its respective input image.Source: §3.1 · p. 3
- Each textured model is projected into another input view, where losses compare the rendered image with that view; the rendering layer is differentiable for gradient backpropagation.The overview figure is identified as an overview of the proposed model.Source: §3.1 · p. 3, Figure 2 · p. 3
3.2. Model
The model represents a face using identity and expression parameters and projects 3D points into images with a weak-perspective camera model. It regresses the shape coefficients and camera pose parameters.
3.2. Model
- The regressed 3DMM parameters include identity and expression coefficients, paired with the identity and expression bases to represent the 3D face model.The identity vector is 199-dimensional and the expression vector is 29-dimensional.Source: §3.2 · p. 3
- The model uses weak-perspective projection to map a 3D point to the 2D image plane using camera pose parameters.The pose contains scale, three Euler angles, and 2D translation, for six parameters in total.Source: §3.2 · p. 3
3.3. Parametric Regression
The network extracts a feature vector for each of three views, regresses a shared 3DMM identity-and-expression parameter set, and predicts pose parameters separately for each view.
3.3. Parametric Regression
- The three 512-dimensional view features are concatenated to regress one 228-dimensional 3DMM parameter set: 199 identity and 29 expression parameters.The parameter set is shared across the three input views.Source: §3.3 · pp. 3–4
- Each view receives its own pose parameters, regressed from that view’s individual features through two fully connected layers.The pose parameters are P = {f, α, β, γ, tx, ty}.Source: §3.3 · p. 3
- Feature extraction and pose regression use shared network weights across the three views.Source: §3.3 · p. 4
3.4. Texture Sampling
Using the predicted 3DMM and pose parameters, the method samples textures from each input image to create differently textured versions of the same 3D face model.
3.4. Texture Sampling
- The predicted 3DMM parameters and identity and expression bases define the 3D face model used for texture sampling.Source: §3.4 · p. 4
- Each input image supplies a texture map through differentiable sampling at projected mesh vertices, with barycentric interpolation for points inside triangles.Projection uses the pose predicted for that image.Source: §3.4 · p. 4
- Because texture sampling does not handle occlusions, sampled colors in occluded regions are erroneous; visibility masks address this problem.Source: §3.4 · p. 4
3.5. Rendered Projection and Visibility Masks
The method renders a textured 3D face model into another input view with differentiable rendering, then uses visibility masks to restrict image comparisons to non-occluded facial regions.
3.5. Rendered Projection and Visibility Masks
- A textured model sampled from image IA is rendered into view IB using pose PB, producing the projected image IA→B.Source: §3.5 · p. 4
- The rendering can be expressed per target pixel by back-projecting through the target pose and face surface, then projecting into the source view.The back-projection uses the 3D surface induced by 3DMM parameters X and pose PB.Source: §3.5 · p. 4
- Rendered-image masks exclude regions that may be occluded in another view, using 3D landmarks on the nose tip and eyebrows.Source: Figure 3 · p. 4, §3.5 · p. 4
- Observed-image masks begin with texture-sampling regions, are edge-filtered using the image as guidance, and exclude potentially occluded regions using detected eyebrow landmarks.The comparison is restricted to pixels in the corresponding non-occluded masks.Source: Figure 4 · p. 4, §3.5 · p. 4
3.6. Losses and Training
Training first uses supervised 3DMM and pose labels, then self-supervised multi-view losses to enforce photometric consistency and improve dense correspondences between observed and rendered images.
3.6.1 Supervised Pretraining
- Supervised pretraining on 300W-LP uses multi-view inputs and ground-truth landmarks, 3DMM parameters, and pose parameters, with 3DMM regularization.The CNN is pretrained before self-supervised training on Multi-PIE.Source: §3.6 · pp. 4–5, §3.6.1 · p. 5
3.6.2 Self-supervised Training
- Self-supervised training enforces photometric consistency between observed images and synthetic renderings, using visibility masks that account for misalignment.For three views, photometric and alignment losses are computed across four observed-rendered image pairs, alongside landmark loss for training stability.Source: §3.6.2 · p. 5
- Self-supervised training adds an alignment loss because similar facial pixels make photometric matching prone to mismatches, improving the reliability of dense correspondences.A differentiable dense optical-flow estimator supplies the alignment loss; its smoothed flow fields can suppress individual mismatches.Source: §3.6.2 · p. 5
- The alignment loss uses bidirectional optical flows between an observed image and a rendering, with regions outside visibility masks filled with textures whose flow is easily estimated.Figure 5 shows optical flows between observed and rendered images.Source: §3.6.2 · p. 5, Figure 5 · p. 5
4. Experiments
Experiments evaluate the method on MICC and other multi-view face datasets, test its loss components, and compare it with single-view reconstruction methods. The ablations favor combining photometric and alignment losses, and the method outperforms single-view methods in both MICC evaluation settings.
4.1. Datasets and Metrics
- Supervised pretraining uses 140k 300W-LP triplets; MICC evaluation uses indoor-cooperative and indoor videos and measures point-to-plane L2 errors, excluding subjects with flawed scans, IDs 2 and 27.Source: §4.1 · pp. 5–6
4.2. Implementation Details
- The differentiable optical-flow estimator PWCNet is used for self-supervised training with its weights fixed.Both supervised pretraining and self-supervised training run for 10 epochs.Source: §4.2 · p. 6
4.3. Ablation Study
- Combining photometric and alignment losses gives the best MICC mean-error reductions: 0.046 for indoor-cooperative and 0.024 for indoor images.Photometric-only reduces indoor-cooperative error by 0.026 but not indoor error; alignment-only reduces them by 0.039 and 0.007, respectively.Source: §4.3 · p. 6
- In a MICC side-view example, v4 produces a flatter facial silhouette than v1, better matching the input.Figure 6 uses red for projection beyond the observed facial boundary, green for a smaller projection, and yellow for overlap.Source: §4.3 · pp. 6–7, Figure 6 · p. 6
- With inconsistent lighting across views, photometric-only results are much worse than results with both losses.The authors attribute alignment-loss robustness to the optical-flow estimator's training to handle lighting changes.Source: §4.3 · p. 7, Figure 8 · p. 7
4.4. Comparisons to State-of-the-art Methods
- The proposed method outperforms all single-view methods in both MICC evaluation settings.Table 2 reports mean errors; visual comparisons also show more accurate shapes and expressions than MoFA across facial expressions.Source: Table 2 · p. 7, §4.4 · p. 8, Figure 11 · p. 8
5. Conclusions
The paper presents an end-to-end CNN that regresses 3DMM parameters from multi-view facial images. It incorporates photometric and alignment losses between views, and the authors validate its effectiveness experimentally.
5. Conclusions
- The approach uses an end-to-end trainable CNN to regress 3DMM parameters from multi-view facial images.It incorporates multi-view geometric constraints through photometric and alignment losses between views, using rendered projections from predicted 3D models.Source: §5 · p. 8
- A differentiable dense optical-flow estimator lets alignment errors backpropagate to the 3DMM parameters.Source: §5 · p. 8
- Experiments validate the effectiveness of the proposed approach; the study explores model-based multi-view reconstruction using deep learning.Source: §5 · p. 8