Source-linked AI summary
Learning to Regress 3D Face Shape and Expression from an Image without 3D Supervision
Soubhik Sanyal, Timo Bolkart, Haiwen Feng, Michael J. Black
TL;DR
Single-image 3D face reconstruction needs robustness despite scarce paired 3D data. RingNet learns shape from repeated identities and 2D features, predicts FLAME from one image, and outperforms compared 3D-supervised methods while NoW supports evaluation.
Problem
Robust 3D face and head estimation from a single image lacks sufficient paired 2D-to-3D training data and must handle varied imaging conditions.
Method
RingNet trains on repeated images, automatically detected 2D facial features, and FLAME shape consistency to regress animatable 3D model parameters from pixels without 2D-to-3D supervision.
Results
RingNet recovers more accurate 3D face shape than methods using 3D supervision, and at runtime predicts FLAME parameters from a single image.
Takeaways & Limitations
The method provides full-head, pose- and expression-factorized reconstruction and introduces NoW for evaluation across varied conditions.
Takeaways & Limitations
RingNet requires some people to appear multiple times and assumes an individual's face shape remains constant across images.
Abstract
from arXiv · showhide
The estimation of 3D face shape from a single image must be robust to variations in lighting, head pose, expression, facial hair, makeup, and occlusions. Robustness requires a large training set of in-the-wild images, which by construction, lack ground truth 3D shape. To train a network without any 2D-to-3D supervision, we present RingNet, which learns to compute 3D face shape from a single image. Our key observation is that an individual's face shape is constant across images, regardless of expression, pose, lighting, etc. RingNet leverages multiple images of a person and automatically detected 2D face features. It uses a novel loss that encourages the face shape to be similar when the identity is the same and different for different people. We achieve invariance to expression by representing the face using the FLAME model. Once trained, our method takes a single image and outputs the parameters of FLAME, which can be readily animated. Additionally we create a new database of faces `not quite in-the-wild' (NoW) with 3D head scans and high-resolution images of the subjects in a wide variety of conditions. We evaluate publicly available methods and find that RingNet is more accurate than methods that use 3D supervision. The dataset, model, and results are available for research purposes at http://ringnet.is.tuebingen.mpg.de.
1. Introduction
The paper targets robust full-head 3D reconstruction from a single image without paired 2D-to-3D supervision. RingNet exploits identity-consistent shape across multiple images, while NoW enables evaluation under varied conditions.
- Single-image reconstruction must capture the full face, head, and neck while separating shape from expression and supporting reposing and animation.
- Limited paired 3D data makes direct 3D regression difficult, while synthetic data and 2D-to-3D fitting introduce mismatches or ambiguity.
- RingNet learns from automatically detected 2D facial features by enforcing similar shape across images of one person and distinct shape across different people.
- FLAME factors pose and expression from shape, enabling the network to learn identity-consistent geometry despite changing imaging conditions.
- At inference, the network requires only one image and directly predicts FLAME parameters from pixels.
- NoW contains high-resolution scans and images of 100 subjects in varied conditions, and RingNet recovers more accurate 3D face shape than compared 3D-supervised methods.
- The paper contributes RingNet, a shape-consistency loss, the NoW benchmark, and publicly available model, code, and dataset resources.
2. Related work
Prior approaches include optimization, supervised learning, weak supervision, and unsupervised photometric modeling. RingNet instead learns directly from pixels using 2D features without a photometric image-formation term.
- Monocular face reconstruction methods differ in whether they estimate pixel-tied shape representations or animatable 3D shape models.
- The paper distinguishes paired 2D-to-3D supervision from settings that use 2D features without 3D training data.
- Existing comparisons lack common datasets combining complex images with high-quality ground truth, motivating the complementary NoW benchmark.
- Optimization methods often require tightly cropped faces, while RingNet uses FLAME to model the whole head and neck with shape and expression spaces learned from more scans.
- 3D-supervised learning methods use synthetic data, direct 3D-model regression, or pseudo-ground-truth fitting, with fitting limited by model expressiveness and accuracy.
- Weakly supervised approaches can combine synthetic and real images but may remain tied to tightly cropped faces or produce non-animatable models.
- Unsupervised methods typically use photometric losses and explicit image-formation models, whereas RingNet uses only 2D face features and learns pixels-to-shape mapping.
3. Proposed method
RingNet estimates full 3D head and face shape from a single image by training on 2D landmarks and identity labels, while using FLAME to separate shape from expression and pose. Its ring architecture enforces consistent shapes for images of the same person and distinct shapes for different people.
- During training, RingNet uses loosely cropped, centered face images with 2D landmarks and identity labels, but inference requires only image pixels.The network predicts camera, pose, shape, and expression parameters from each image.
- 3.1. FLAME model: FLAME decodes semantically meaningful shape, expression, and pose parameters into a complete 3D head and face mesh.FLAME models identity- and expression-dependent variations, joint rotations, and returns 5,023 vertices.
- 3.2. RingNet: RingNet shares encoder weights across ring elements and processes multiple images in parallel, with each element combining a feature extractor, regressor, and fixed decoder.The regression network uses iterative error feedback before the differentiable FLAME decoder produces the 3D head mesh.
- 3.3. Shape consistency loss: The shape consistency loss makes matched images close in shape space while requiring unmatched identities to remain separated by a margin.Training batches contain R − 1 images of one identity and one image of another identity, enabling discriminative shape generation.
- 3.4. 2D feature loss: The 2D feature loss compares ground-truth landmarks with landmarks retrieved from the predicted 3D mesh and projected into the image plane.Static and pose-dependent dynamic landmarks are defined on FLAME, and landmark confidence weights determine whether observations contribute to the loss.
4. Benchmark dataset and evaluation metric
NoW is a benchmark for evaluating monocular 3D face reconstruction under varied viewing conditions, with high-resolution images and subject-specific 3D scans. Its scan-to-mesh metric reports reconstruction errors after alignment.
- NoW evaluates 3D face reconstruction under variations in viewing angle, lighting, and common occlusions.
- The dataset contains 2,054 images of 100 subjects and a separate 3D head scan for each subject as evaluation ground truth.
- The four challenges are neutral, expression, occlusion, and selfie, covering multiple views, acted expressions, partial occlusions, and participant-taken selfies.
- The benchmark reconstructs a neutral 3D face from each monocular image, requiring methods to disentangle identity from expression.
- Each subject’s neutral head scan is captured with an active stereo system and contains about 120K vertices.
- Predicted meshes are aligned to scans using corresponding landmarks and further alignment based on scan-to-mesh distance before error computation.
- Reported metrics include cumulative error plots, median distance, average distance, and standard deviation.
- The challenge withholds ground-truth scans to prevent fine-tuning on test data.
5. Experiments
RingNet is evaluated against publicly available 3D-supervised methods on established and newly collected benchmarks. It outperforms competing methods quantitatively and shows qualitative robustness across challenging imaging conditions.
- The experiments compare RingNet with PRNet, Extreme3D, and 3DMM-CNN using qualitative and quantitative evaluations.
- The Feng et al. benchmark contains 2,000 neutral images, including 656 high-quality and 1,344 low-quality images.
- RingNet outperforms the other methods on cumulative error curves for both low-quality and high-quality benchmark images.
- On the NoW face challenge, methods are rigidly aligned to scans and RingNet again outperforms the others on cumulative error curves.
- Qualitative results show single-image 3D face and head reconstruction on CelebA and MultiPIE under varied expression, gender, pose, hair, lighting, and occlusion.
- The ablation study reports lower reconstruction error as the ring uses more elements rather than only a single triplet loss, although error eventually increases.
6. Conclusion
The paper concludes that RingNet learns deformable 3D face shape from single images without paired 3D training data by enforcing identity-based shape consistency. It also introduces NoW and identifies extensions involving supervision and ear detections.
- RingNet addresses estimation of articulated, deformable 3D shape from a single 2D image without paired 3D training data.
- Its ring of pairwise losses encourages shared shape across images of one person and different shape across different people.
- FLAME factors pose and expression from shape, enabling RingNet to constrain identity shape while allowing other parameters to vary.
- The method requires repeated identities in the training data and 2D facial features estimated by existing methods.
- RingNet outperforms methods using 3D supervision and is accompanied by a public dataset with accurate 3D head shape and images across varied conditions.
- The authors suggest that limited supervision and ear detections could improve accuracy, while the current detector does not include ears.
Appendix
The appendix presents cumulative error plots for four NoW challenges and relates them to the aggregate error curve across all challenges.
- The appendix reports cumulative error plots for neutral, expression, occlusion, and selfie challenges in the NoW dataset.
- The right panel of Figure 5 shows cumulative error across all NoW challenges.