Source-linked AI summary
Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set
Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, Xin Tong
TL;DR
The paper addresses scarce ground-truth 3D face data by learning reconstruction from weak supervision. It combines hybrid image- and perception-level losses with confidence-based multi-image aggregation, reporting accurate and robust reconstruction across datasets, including under occlusion and varying pose.
Problem
Ground-truth 3D face shapes are scarce, limiting conventional supervised training for deep 3D face reconstruction.
Method
The method uses hybrid low-level and perception-level weak supervision for single images and a confidence subnet to aggregate complementary multi-image reconstructions without explicit labels.
Results
The method outperforms previous methods in accuracy and robustness, while learned aggregation improves reconstruction over naive shape averaging.
Takeaways & Limitations
Weakly supervised CNN reconstruction can produce accurate single-image faces and exploit image quality and pose diversity to build more accurate shapes from image sets.
Takeaways & Limitations
The illumination model assumes a Lambertian face and approximates scene lighting with spherical harmonics.
Abstract
from arXiv · showhide
Recently, deep learning based 3D face reconstruction methods have shown promising results in both quality and efficiency.However, training deep neural networks typically requires a large volume of data, whereas face images with ground-truth 3D face shapes are scarce. In this paper, we propose a novel deep 3D face reconstruction approach that 1) leverages a robust, hybrid loss function for weakly-supervised learning which takes into account both low-level and perception-level information for supervision, and 2) performs multi-image face reconstruction by exploiting complementary information from different images for shape aggregation. Our method is fast, accurate, and robust to occlusion and large pose. We provide comprehensive experiments on three datasets, systematically comparing our method with fifteen recent methods and demonstrating its state-of-the-art performance.
1. Introduction
The paper targets accurate 3D face reconstruction without ground-truth 3D training shapes, using weak supervision for both single images and image sets. It combines hybrid-level losses with confidence-based aggregation to exploit complementary views and improve robustness.
- Ground-truth 3D face data is scarce, while surrogate labels from synthetic data or traditional fitting can suffer from domain gaps or imperfect labels.
- The single-image method combines robust image-level and perception-level losses because either pixel consistency or perceptual similarity alone can produce suboptimal shapes.The hybrid loss also includes skin-color-based photometric attention for robustness to occlusion, beards, and heavy makeup.
- For image sets, a confidence subnet weights single-image identity-shape coefficients, favoring high-quality and highly visible images without explicit confidence labels.
- Confidence-based aggregation exploits pose differences and complementary information, improving reconstruction beyond naive shape averaging and heuristic strategies.
- Experiments across multiple datasets report state-of-the-art single-image accuracy and substantial advantages over prior supervised and unrestricted-representation methods.
2. Related Work
Earlier 3D face reconstruction relied on costly optimization, reference shapes, or surrogate labels, whereas CNN methods improved efficiency but remained constrained by scarce training data. The paper positions weak supervision and learned aggregation as alternatives for single images and unconstrained photo collections.
- Traditional 3D face reconstruction commonly uses 3DMM analysis-by-synthesis or landmarks, but landmark sparsity limits dense facial fidelity.
- CNN-based methods regress 3DMM coefficients, refine coarse predictions, or reconstruct faces directly, offering alternatives to costly optimization.
- Scarce training data leads CNN methods to use synthetic data, fitted shapes, or unsupervised and weakly supervised learning.
- The paper reports that weak supervision benefits from combining low-level and perception-level information and can outperform synthetic or pseudo-ground-truth training.
- For multi-image reconstruction, prior quality measurement and traditional photo-collection methods provide context for the paper's learned confidence-based fusion approach.
3. Preliminaries: Models and Outputs
The method represents faces with a 3D Morphable Model and predicts identity, expression, texture, illumination, and pose parameters from images. A CNN regresses these parameters for differentiable reconstruction and image regeneration.
- The face shape and texture use affine 3DMM representations built from mean components and PCA bases for identity, expression, and texture.The selected coefficient dimensions are α ∈R80, β ∈R64, and δ ∈R80.
- The model uses the Basel Face Model for identity and texture components and FaceWarehouse-derived expression bases.
- Illumination assumes a Lambertian face and approximates scene lighting with spherical harmonics using three bands and monochromatic lights.
- A perspective camera maps the reconstructed face to the image, with pose represented by rotation R ∈SO(3) and translation t ∈R3.
- The predicted vector x = (α, β, δ, γ, p) ∈R239 is regressed by a modified ResNet-50 called R-Net.
4. Hybrid-level Weak-supervision for Single-Image Reconstruction
The method trains a 3D face reconstruction network without ground-truth shape labels by combining image-level, perceptual, and regularization losses. A skin-aware photometric loss improves robustness, while perceptual supervision produces more faithful shapes than image-level losses alone.
- Network and weak supervision: R-Net regresses 3DMM coefficients from an RGB image and analytically generates a reconstructed image for weakly-supervised training.The network is trained by evaluating a hybrid-level loss on the reconstructed image and backpropagating it.
- Hybrid-level loss: The hybrid loss combines per-pixel color, sparse 2D landmarks, perceptual similarity, coefficient regularization, and texture flattening.The full loss contains two image-level losses, one perceptual loss, and two regularization losses.
- Robust photometric loss: The skin-aware photometric loss weights pixels using a skin-color attention mask to improve robustness to occlusions, beards, and heavy make-up.It operates over the reprojected face region and uses a skin-color probability estimated with a Gaussian-mixture-model naive Bayes classifier.
- Additional robustness: The photometric loss integrates over 2D image pixels and uses z-buffering to identify self-occlusion, enabling handling of large poses.The landmark loss additionally supplies weak supervision from detected 2D landmarks projected from reconstructed 3D landmark vertices.
- Perception-level loss: Image-level losses alone can reach low photometric error while producing less accurate 3D shapes, motivating the added perceptual loss.The perceptual loss compares deep features of the input and reconstructed images using cosine distance.
- Perception-level loss: With perceptual loss, reconstructed textures become sharper and shapes more faithful than with image-level losses alone.The paper reports this qualitative improvement in Figure 3 and notes quantitative benefits in the experiments.
5. Weakly-supervised Neural Aggregation for Multi-Image Reconstruction
For multiple images of one subject, the method learns confidence scores for single-image reconstructions and uses them to aggregate identity-bearing shape coefficients. This aggregation exploits complementary views while accounting for image quality and pose differences.
- Motivation: Images captured under different poses and lighting provide complementary information and can improve robustness to occlusion and bad lighting.The motivation is to use an image set rather than relying on one potentially corrupted image.
- Confidence-based aggregation: C-Net predicts a positive 80-dimensional confidence vector for the 80 identity-bearing shape coefficients and uses it for aggregation.Expression, pose, and lighting coefficients are not included in the confidence prediction.
- Confidence-based aggregation: The learned confidence replaces naive shape averaging by accounting for the quality of individual images and exploiting pose differences.Naive averaging does not handle severe occlusion or fully use pose variation.
- Label-free training: C-Net is trained without labels by reconstructing each image with the aggregated identity shape and minimizing the hybrid loss over the image set.Because the aggregation is differentiable, the error propagates through the aggregated shape to the confidence predictions and C-Net weights.
- Design distinction: The method produces element-wise scores for 3DMM coefficients rather than the scalar quality score used in the cited set-based face recognition work.The paper reports that element-wise scores yield superior results and analyzes their use of face pose differences.
- C-Net architecture: C-Net reuses shallow and deep R-Net features, with shallow features helping measure image corruptions such as occlusion.The network is designed to be lightweight and has 3M parameters, about one-eighth the size of R-Net.
6. Experiments
Experiments show that the hybrid loss improves reconstruction accuracy, while the proposed single- and multi-image methods outperform prior approaches across quantitative and qualitative evaluations.
- Implementation Details: Approximately 260K in-the-wild face images were collected to train R-Net, with pose and race distributions balanced.The images were detected and aligned, resized to 224×224, and used with ImageNet initialization.
- Ablation Study and Analysis: Jointly using image- and perception-level information yields significantly higher accuracy than using either loss separately on MICC and FaceWarehouse.Table 1 evaluates average reconstruction errors in millimeters for R-Net trained with different loss combinations.
- Comparison with Prior Art: Our method significantly outperforms Tran et al. and Genova et al. on all three MICC sequences.The comparison uses ICP alignment and point-to-plane error; qualitative comparisons also show superior results.
- Comparison with Prior Art: Our method has much lower point-to-plane RMSE than PRN for all yaw angles on MICC.The comparison averages RMSE across subjects and pitch angles, and PRN uses a larger model, 160MB versus 92MB.
- Comparison with Prior Art: Qualitatively, our method recovers identity and expression with greater shape variance and handles occlusion and profile faces better than several prior methods.Comparisons include VRN, 3DDFA, Liu et al., and [48].
- Multi-image Reconstruction: All learned aggregation strategies outperform naive shape averaging, with element-wise coefficient aggregation S4 performing best.Confidence scores tend to favor frontal, visible, naturally lit images, while element-wise confidences exploit pose differences for reconstruction.
7. Conclusions
The paper presents weakly supervised CNN methods for accurate single-image reconstruction and confidence-based multi-image aggregation without ground-truth 3D shapes. Experiments report improved accuracy, robustness, and use of complementary image information.
- 7. Conclusions: The proposed CNN reconstructs faces from single images using hybrid-level weak supervision without ground-truth 3D shapes.It combines image-level and perception-level information.
- 7. Conclusions: Experiments show large-margin improvements in accuracy and robustness over previous methods.The paper also introduces CNN-based multi-image aggregation that learns image quality without explicit labels.
- 7. Conclusions: The aggregation method exploits complementary information across images to reconstruct 3D faces more accurately.It learns to measure image quality without explicit confidence labels.