Source-linked AI summary
Regressing Robust and Discriminative 3D Morphable Models with a very Deep Neural Network
Anh Tuan Tran, Tal Hassner, Iacopo Masi, Gerard Medioni
TL;DR
Single-view 3D face reconstruction has struggled in unconstrained images because estimates can vary across photos of the same person or become generic, limiting recognition use. The paper regresses 3DMM parameters with a very deep CNN and generates large labeled training sets to overcome data scarcity. The resulting shapes surpass alternative reconstruction methods on MICC and support competitive recognition across LFW, YTF, and IJB-A.
Problem
Single-view 3D face estimates may be unstable or generic in unconstrained conditions, limiting their use as identity representations despite the discriminative nature of true 3D face shapes.
Method
A very deep CNN directly regresses 3DMM shape and texture parameters from face photos, using generated labeled examples to address training-data scarcity.
Results
The estimated shapes surpass alternative reconstruction methods on MICC and support face recognition on LFW, YTF, and IJB-A.
Takeaways & Limitations
Single-image 3D face shapes can serve as robust, discriminative representations under unconstrained viewing conditions.
Takeaways & Limitations
The approach relies on the availability of multiple photos per subject to obtain accurate 3D estimates for training data generation.
Abstract
from arXiv · showhide
The 3D shapes of faces are well known to be discriminative. Yet despite this, they are rarely used for face recognition and always under controlled viewing conditions. We claim that this is a symptom of a serious but often overlooked problem with existing methods for single view 3D face reconstruction: when applied "in the wild", their 3D estimates are either unstable and change for different photos of the same subject or they are over-regularized and generic. In response, we describe a robust method for regressing discriminative 3D morphable face models (3DMM). We use a convolutional neural network (CNN) to regress 3DMM shape and texture parameters directly from an input photo. We overcome the shortage of training data required for this purpose by offering a method for generating huge numbers of labeled examples. The 3D estimates produced by our CNN surpass state of the art accuracy on the MICC data set. Coupled with a 3D-3D face matching pipeline, we show the first competitive face recognition results on the LFW, YTF and IJB-A benchmarks using 3D face shapes as representations, rather than the opaque deep feature vectors used by other modern systems.
1. Introduction
Although true 3D face shapes are discriminative and robust to appearance changes, single-view reconstruction has rarely supported recognition in unconstrained settings because estimates can be unstable or generic. This paper proposes a CNN-based approach that generates robust, discriminative 3DMM estimates and demonstrates recognition across unconstrained benchmarks.
- True 3D face shapes are discriminative and invariant to lighting and texture changes, making them promising face representations.High-resolution 3D scans have even distinguished identical twins.
- 3DMMs were used for face recognition mainly under controlled viewing conditions, with no reported success for single-view reconstruction in the wild.
- Unconstrained reconstruction methods produce either inconsistent shapes across photos of one subject or over-regularized, generic shapes.These behaviors are illustrated in Figure 1 and make the estimates poor recognition representations.
- A very deep CNN directly regresses 3DMM shape and texture parameters from single face photos.The method addresses the shortage of labeled training data by generating huge numbers of labeled unconstrained examples.
- The approach is reported as fast, robust, and accurate, with estimates surpassing other 3D reconstruction methods on MICC.Its 3D shapes also support recognition on LFW, YTF, and IJB-A.
2. Related work
Prior single-view reconstruction work includes multi-image methods, statistical 3DMMs, CNN regression, and scene-assumption approaches. These methods differ in data requirements, representation choices, and robustness to unconstrained imagery.
- Multi-image methods can produce accurate reconstructions but require many images from multiple sources, unlike this paper’s single-image reconstruction.
- Statistical 3DMM methods learn a high-dimensional subspace of aligned 3D faces and reconstruct by finding parameters matching the input image.They generally do not target discriminative facial geometries and were mainly used for recognition under controlled conditions.
- A prior CNN method also regressed 3DMM parameters and addressed limited training data by sampling synthetic faces from the 3DMM distribution.
- The prior CNN approach produced synthetic-looking photos, trained only a seven-layer residual network, and did not show greater robustness or discrimination than alternatives.
- Scene assumption methods: Scene-assumption methods seek correct reconstructions by imposing strong assumptions on the scene and viewing conditions.
3. Regressing 3DMM parameters with a CNN
The method generates pooled 3DMM targets from unconstrained face images, then fine-tunes a very deep CNN to regress 3DMM shape and texture parameters directly from single photos. An asymmetric Euclidean loss is used to reduce bias toward generic mean-face estimates.
- Generating training data: Limited ground-truth 3D scans are replaced by pooled multi-image 3DMM estimates generated from unconstrained face collections.Single-image estimates are aggregated across images of each subject to create one training target per subject.
- 3DMM representation: The face model decouples shape and texture, representing each with a mean vector plus principal-component parameters.The shape and texture parameter vectors are each 99-dimensional.
- Generating training data: The training pipeline fits 3DMM parameters separately to images, pools estimates for each subject, and trains the CNN using those pooled estimates as targets.Pooling uses element-wise weighted averaging, with weights given by normalized per-image landmark-detection confidences.
- Learning objective: An asymmetric Euclidean loss separates under-estimation from over-estimation errors to favor estimates farther from the 3DMM origin.The practical setting λ1 = 1 and λ2 = 3 is intended to help escape under-fitting and produce more detailed, realistic models.
- Learning objective: Figure 3 compares a generic model, regular ℓ2 regression, and the proposed asymmetric ℓ2 regression on the same input image.The proposed loss is associated with more detailed, realistic 3D face models.
4. Experimental results
The experiments evaluate 3D shape accuracy, runtime, and recognition across MICC, LFW, YTF, and IJB-A. The proposed method is most accurate on MICC, substantially faster than iterative alternatives, and competitive for unconstrained face recognition.
- Evaluation setup: The evaluation compares 3D shape accuracy, speed, and recognition against multiple reconstruction and face-recognition baselines across MICC, LFW, YTF, and IJB-A.MICC provides scanned ground-truth 3D shapes, while LFW, YTF, and IJB-A test recognition from unconstrained images or videos.
- 3D reconstruction accuracy: Our method is clearly the most accurate on MICC, with both single-view and multi-frame versions outperforming the 3DMM+pool method used to generate training labels.Errors are averaged across videos and reported with standard deviations; lower values are better.
- Runtime: At 0.088s per image (∼11Hz), the CNN predicts 3DMM parameters several orders of magnitude faster than most tested methods.The second-fastest method, 3DDFA, requires 0.146s (∼7Hz).
- Face recognition in the wild: On LFW, recognition with the estimated 3DMM parameters is not far behind Facebook’s multi-CNN system trained on four million images.Existing 3D reconstruction methods perform substantially worse and suggest instability or excessive generic regularization in unconstrained settings.
- Face recognition in the wild: On YTF, the method performs well despite low-quality, low-resolution frames, trailing Facebook’s CNN ensemble by an AUC gap of only ∼1%.Other 3D methods perform considerably worse, with 3DDFA falling far behind.
- Face recognition in the wild: On IJB-A, the method performs quite well across verification and recognition protocols but is outperformed by a wide margin by a system designed specifically for that benchmark.The same broad pattern observed on LFW and YTF is reported for IJB-A.
5. Conclusions
The paper argues that existing single-view 3D face estimates can be unstable or generic, limiting their ability to capture identity. It proposes a very deep CNN that regresses 3DMM parameters directly from images and reports robust, discriminative shapes in unconstrained recognition tests.
- Existing methods produce 3D face shapes that may vary with unconstrained viewing conditions or become too generic to capture identity.
- A very deep CNN is proposed to regress 3DMM parameters directly from input images.
- Figure 6 compares qualitative outputs from 3DMM, 3DDFA, and the proposed method on LFW and YTF images, including two failure examples.
- The method addresses the shortage of labeled data needed to train the CNN.
- The resulting shapes are reported as more accurate than alternative methods and robust and discriminative under unconstrained viewing conditions.