Source-linked AI summary
Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency
Jiaxiang Shang, Tianwei Shen, Shiwei Li, Lei Zhou, Mingmin Zhen, Tian Fang, Long Quan
TL;DR
Monocular face reconstruction remains ambiguous because 2D-only supervision does not reliably constrain face pose and depth. MGCNet uses occlusion-aware view synthesis and multi-view geometry losses for self-supervised training, achieving reported gains on face alignment and 3D reconstruction benchmarks. Its scope includes a calibrated-camera assumption and limitations in illumination and albedo estimation.
Problem
Existing self-supervised methods rely on 2D constraints, leaving monocular face pose and depth estimation ambiguous.
Method
MGCNet combines occlusion-aware view synthesis with pixel, depth, and facial epipolar consistency losses for self-supervised monocular reconstruction.
Results
MGCNet improves face alignment by more than 12% and achieves a largest 17% margin over other methods on challenging 3D face reconstruction datasets.
Takeaways & Limitations
Multi-view geometry consistency provides more reliable pose and depth constraints without costly labeled 3D face data.
Takeaways & Limitations
The method assumes a calibrated camera and has intrinsic illumination and face-albedo ambiguity, while its SH lighting is restricted to RGB-channel settings.
Abstract
from arXiv · showhide
Recent learning-based approaches, in which models are trained by single-view images have shown promising results for monocular 3D face reconstruction, but they suffer from the ill-posed face pose and depth ambiguity issue. In contrast to previous works that only enforce 2D feature constraints, we propose a self-supervised training architecture by leveraging the multi-view geometry consistency, which provides reliable constraints on face pose and depth estimation. We first propose an occlusion-aware view synthesis method to apply multi-view geometry consistency to self-supervised learning. Then we design three novel loss functions for multi-view consistency, including the pixel consistency loss, the depth consistency loss, and the facial landmark-based epipolar loss. Our method is accurate and robust, especially under large variations of expressions, poses, and illumination conditions. Comprehensive experiments on the face alignment and 3D face reconstruction benchmarks have demonstrated superiority over state-of-the-art methods. Our code and model are released in https://github.com/jiaxiangshang/MGCNet.
1 Introduction
Existing self-supervised approaches rely on projected 2D constraints, leaving face pose and depth ambiguous. MGCNet addresses this with occlusion-aware multi-view geometry consistency and three differentiable losses, improving benchmark performance.
- Existing unsupervised methods use 2D landmark and rendering losses that do not penalize incorrect face pose or depth.
- Multi-view images provide geometric constraints, while view synthesis establishes pixel correspondences for self-supervised training.
- MGCNet uses a covisible map, differentiable rendering, and covisible-region consistency to handle self-occlusion and constrain 3DMM parameters.
- The architecture combines pixel consistency, depth consistency, and facial epipolar losses with an existing 2D feature-loss baseline.
- More than 12% improvement was achieved on face alignment, while 3D face reconstruction comparisons showed a largest margin of 17%.The reported gains are especially pronounced for large and extreme face poses.
2 Related work
Prior work includes supervised, self-supervised, and multi-view approaches, but limited scans and monocular ambiguity remain important challenges. MGCNet uses multi-view consistency during training while retaining single-input inference.
- Supervised CNN methods require abundant 3D scans and RGB images, while detail-reconstruction methods require high-quality scans.
- Self-supervised methods reduce dependence on scans but monocular images still leave face pose and depth ambiguous.
- View synthesis aims to generate scene appearance from novel camera viewpoints using multi-view images.
- Existing multi-image methods enforce shape consistency across images or subjects, while MVFNet relies on multi-view aggregation during inference.
- MGCNet exploits geometric and photometric consistency during training to produce view-consistent face geometry from a single input.
3 Method
MGCNet trains monocular 3D face reconstruction with multi-view geometry consistency, combining occlusion-aware view synthesis with 2D feature and geometry-consistency losses. It uses covisibility to avoid self-occlusion errors and enforces pixel, depth, and epipolar agreement across views.
- 3.1 Framework: MGCNet estimates 3DMM coefficients and face poses from multi-view training images while retaining single-image inference.Training uses N-view images of the same face; inference takes a single image.
- 3.2 Model: The model uses a 3DMM with identity, expression, and texture bases, a calibrated pinhole camera, spherical-harmonics illumination, and 257 total parameters.The face model contains approximately 36K vertices and excludes the ear and neck regions.
- 3.3 2D Feature Loss: The baseline 2D feature loss combines render, landmark, identity, and regularization losses for image appearance, alignment, identity, and 3DMM stability.The authors define this combination as L2D and use it as the baseline trained only with 2D feature losses.
- 3.4 Occlusion-Aware View Synthesis: A covisible map masks pixels visible in both target and source views, preventing self-occluded regions from producing redundant view-synthesis errors.Covisible points and adjacent triangles are projected into image space to construct the mask.
- 3.4 Occlusion-Aware View Synthesis: View synthesis establishes pixel correspondences using rendered target depth, camera intrinsics, relative pose, and bilinear sampling, then compares synthesized and target views.The covisible map restricts the pixel comparison to shared visible regions.
4 Experiment
MGCNet is evaluated on face alignment and 3D face reconstruction across diverse datasets, poses, lighting conditions, and expressions. It generally outperforms competing methods, with especially strong results for extreme poses and multi-view geometry reconstruction.
- Evaluation setup: MGCNet is evaluated on AFLW20003D, MICC Florence, BU-3DFE, and FRGC v2.0 against recent state-of-the-art methods.The experiments cover face alignment and 3D face reconstruction tasks.
- Qualitative results: MGCNet reconstructs facial geometry, reflectance, and incident illumination across varied lighting, poses, and expressions.Figure 5 presents input images with reconstruction overlays, while supplementary results provide further shape, albedo, and lighting details.
- 2D face alignment: MGCNet produces better face alignment and geometry than competing methods on large and extreme-pose AFLW20003D samples.The comparisons include 3DDFA, RingNet, PRN, and Deng et al.; the results also show improved mouth shapes and more vivid expressions in selected cases.
- Evaluation caveat: Quantitative 6DoF pose comparisons are not conducted because competing methods use different camera intrinsic assumptions.The paper instead evaluates pose ambiguity through AFLW2000-3D face alignment results.
- 2D face alignment: 12% improvement in normalized mean error is reported over the best compared method on AFLW2000-3D face alignment.NME uses the ground-truth landmark bounding-box size as its normalization factor, and performance degradation in competing methods becomes more pronounced as yaw increases.
- 3D face reconstruction: MGCNet shows obvious improvement under the extreme-pose setting x-axis[-80, 80] in MICC Florence rendered images.The paper links the stronger benefit in extreme poses to reduced 2D features and the additional pose and depth constraints supplied by multi-view geometry consistency.
- 3D face reconstruction: MGCNet outperforms competing methods on FRGC v2.0 and achieves lower reconstruction error than state-of-the-art methods on BU-3DFE.FRGC v2.0 uses point-to-point MAE after landmark and ICP alignment, while BU-3DFE reports point-to-point RMSE after similarity alignment.
- Ablation study: The ablation study evaluates pixel consistency, dense depth consistency, and epipolar losses against a baseline trained with only 2D feature losses.The study is conducted on the MICC Florence 3D Face dataset using RMSE.
5 Conclusion
The paper presents MGCNet as a self-supervised monocular 3D face reconstruction pipeline that uses occlusion-aware view synthesis and multi-view geometry consistency. Its results support more reliable face pose and depth estimation and robust reconstruction.
- Conclusion: MGCNet is a self-supervised monocular 3D face reconstruction pipeline built around multi-view geometry consistency.The paper emphasizes occlusion-aware view synthesis and multi-view losses as central components.
- Conclusion: Multi-view geometry consistency provides more reliable constraints for face pose and depth estimation.The conclusion highlights the method's robustness and its ability to capture both high-level cues and fine-grained information.
Supplemental Material for ”Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view
The supplementary material accompanies the paper with detailed evaluation results and implementation-related material for MGCNet.
- Supplemental material: The supplementary document provides detailed evaluation results accompanying the main paper.It also describes data preprocessing, evaluation datasets, the morphable model, and quantitative render-error evaluation.
1 Overview
The supplementary document details MGCNet’s evaluation, preprocessing, model generality, and comparisons across face reconstruction and alignment benchmarks.
- Evaluation: MGCNet is evaluated through render-error measurements, qualitative ablations, comparisons under geometry, texture, and lighting, and in-the-wild image studies.The supplement also reports qualitative comparisons on MICC Florence and AFLW20003D.
- Framework: The document presents MGCNet as a self-supervised framework designed to mitigate monocular face-pose and depth ambiguity.
- Model scope: MGCNet is described as a general framework rather than one limited to a specific face model.The authors state that other face models can be plugged into the framework.
2 Data Preprocess
The training data are automatically annotated, filtered, resized face images, while multi-view training assumes consistent lighting and view overlap.
- Image preprocessing: Approximately 390K face images from four datasets form the training set after automatic landmark and face detection, filtering, and resizing to 224 × 224.Filtering removes unsuitable pose, face-attribution, low-resolution, and blurred images.
- Multi-view assumptions: Multi-view training uses views captured under consistent lighting because dramatic cross-view lighting differences violate photometric consistency.
- Multi-view assumptions: The method assumes overlap across views and therefore uses N = 3 views in practice.
3 Quantitative Evaluation Dataset
The evaluation spans in-the-wild alignment images and multiple 3D face-scan datasets covering identities, expressions, poses, resolutions, and acquisition conditions.
- Face alignment: AFLW20003D evaluates face alignment on challenging in-the-wild images using 2,000 AFLW images with landmark annotations.
- 3D reconstruction: MICC Florence contains 53 faces with high-resolution ground-truth 3D scans and video sequences varying in resolution, conditions, and zoom level.
- 3D reconstruction: FRGC v2.0 includes 4,007 scans from 466 individuals, mostly frontal, with neutral and spontaneous facial expressions.About 60% of the faces are neutral; the remainder include disgust, happiness, sadness, and surprise.
- Expression diversity: BU-3DFE contains 100 subjects and 2,500 facial-expression models spanning ages 18 to 70 and varied demographic groups.
4 3D Morphable Model
The section motivates 3DMM-based single-image reconstruction, describes MGCNet’s model-agnostic multi-view consistency, and includes photometric-error visualization.
- 3D Morphable Model: 3DMM constrains the solution space and simplifies 3D face reconstruction; this work estimates its parameters from a single photograph.
- 3D Morphable Model: The experiments use 3DMM because it remains widely used by recent single-image methods and enables fair comparisons.
- Framework scope: MGCNet improves single-view reconstruction quality through multi-view consistency and is not limited to a specific face model.
- Photometric evaluation: The photometric-error visualization on CelebA uses an error-map range of [0, 1].
5 Further Evaluation Result
MGCNet is evaluated across datasets, poses, expressions, and competing reconstruction methods, with results indicating accurate geometry and alignment. The method also produces qualitative reconstructions under varied appearance conditions, while illumination and albedo remain intrinsically ambiguous.
- Photometric evaluation: MGCNet achieves low photometric error on in-the-wild CelebA images, supporting its reconstruction capability beyond controlled multi-view data.CelebA images are used for testing and visualization.
- Geometry ablation: The MICC Florence ablation shows the largest geometry improvements around the jaw, nose, cheekbones, and eye contours, especially for large-pose cases.Performance is measured using point-to-plane root mean squared error visualized as a normalized heatmap.
- Comparison with prior methods: MGCNet produces better geometry overlays than Richardson, Sela, Tewari17, Tewari19, and RingNet, while its quantitative geometry result exceeds Tewari19 despite differing face models.The authors note that Tewari19 uses a more complicated representation than MGCNet’s 3DMM.
- Comparison with prior methods: On MICC Florence, MGCNet is more accurate than most compared methods for frontal and large-pose images, including better large-pose results than Deng et al.The comparison includes 3DDFA, RingNet, PRN, and Deng et al.
- Qualitative reconstruction: Qualitative results show accurate alignment and vivid multi-view reconstructions on AFLW20003D, including cases reported as better than the dataset ground truth.The method is also visualized across varied expressions on BU-3DFE, with pose, texture, geometry, and illumination outputs.
- Limitations: Illumination and face-albedo ambiguity remains intrinsic, and the authors note that RGB-channel spherical-harmonics lighting limits the lighting representation.The paper suggests that restricting spherical-harmonics lighting to one channel would help.