Source-linked AI summary
3D Face Reconstruction by Learning from Synthetic Data
Elad Richardson, Matan Sela, Ron Kimmel
TL;DR
Single-image facial reconstruction lacks large annotated 3D-face datasets and existing methods often depend on landmarks or manual initialization. The paper trains an iterative CNN on synthetic faces with known geometry, recovering facial shapes from real images across varied expressions and illumination. The approach generalizes to in-the-wild faces, while its model remains constrained by the scanned-face and synthetic-data distributions.
Problem
Accurate single-image facial reconstruction requires large annotated datasets, but suitable large 3D-face datasets are unavailable and existing methods rely on landmarks or manual initialization.
Method
The method trains an iterative CNN on synthetic, nearly photo-realistic facial images with known geometry represented in a low-dimensional facial space.
Results
The approach successfully reconstructs facial geometry from real in-the-wild images across different illumination conditions, viewpoints, and expressions, including details beyond sparse landmarks.
Takeaways & Limitations
Direct image-wide CNN reconstruction can recover relatively accurate facial geometry from a single image and support efficient handling of varied real-world conditions.
Takeaways & Limitations
Performance is constrained by the ethnic origins represented in the 3DMM scans and by facial attributes absent from the synthetic training data.
Abstract
from arXiv · showhide
Fast and robust three-dimensional reconstruction of facial geometric structure from a single image is a challenging task with numerous applications. Here, we introduce a learning-based approach for reconstructing a three-dimensional face from a single image. Recent face recovery methods rely on accurate localization of key characteristic points. In contrast, the proposed approach is based on a Convolutional-Neural-Network (CNN) which extracts the face geometry directly from its image. Although such deep architectures outperform other models in complex computer vision problems, training them properly requires a large dataset of annotated examples. In the case of three-dimensional faces, currently, there are no large volume data sets, while acquiring such big-data is a tedious task. As an alternative, we propose to generate random, yet nearly photo-realistic, facial images for which the geometric form is known. The suggested model successfully recovers facial shapes from real images, even for faces with extreme expressions and under various lighting conditions.
1. Introduction
Facial reconstruction from a single image is ill-posed and existing approaches depend on manual initialization or accurate landmarks. The paper proposes CNN-based direct reconstruction trained on synthetic, nearly photo-realistic faces using a low-dimensional facial model.
- Single-image facial geometry reconstruction is ill-posed because different textured surfaces can produce the same camera image.
- Optimization-based methods iteratively update geometry and lighting but require manual initialization despite achieving highly accurate results.
- Landmark-based methods improve robustness to lighting and albedo changes but depend strongly on landmark accuracy and miss details beyond sparse landmarks.
- The proposed CNN reconstructs geometry from the image as a whole, can implicitly model different rendering methods, and supports real-time systems.
- The network is trained with synthetic photo-realistic facial images whose known geometries are generated across varied pose, expression, race, and gender.
- Facial textures and geometries are represented in a low-dimensional linear space used both to synthesize training data and predict reconstruction coefficients.
2. Related Efforts
Earlier approaches use morphable models, shape-from-shading, landmark alignment, or CNNs trained on augmented datasets. This work extends synthetic-data training to automatically generated facial reconstruction examples.
- The 3D Morphable Model represents facial geometry and texture as linear combinations of vectors learned from registered scans of a few hundred subjects.
- Analysis-by-synthesis methods alternate rendering with refinement of geometry, texture, and illumination parameters.
- Shape-from-shading approaches constrain reconstruction by manually aligning a single 3D reference face and recovering lighting, depth, and albedo.
- Landmark-based methods automate alignment or reconstruct geometry from detected feature points, but automated initialization can be less accurate than manual initialization.
- CNN approaches integrated with 3DMMs recover pose variations when trained on augmented face-alignment datasets, assuming a suitable dataset exists.
- The paper extends synthetic-data training by automatically generating a large dataset for facial shape recovery.
3. Overview
The proposed pipeline masks an input facial image, repeatedly updates its geometry representation through a CNN, and trains the network on synthetic textured 3D-face images.
- The input image is masked using the projection of a generic 3D facial model posed by an automatic alignment algorithm.
- The masked image is propagated through the network multiple times while the geometry representation is iteratively updated.
- Training uses synthetic images of textured three-dimensional faces rendered from sampled geometric and photometric structures.
4. The Face Model
The face model uses a 3D Morphable Model to represent geometry through identity and expression bases and texture through a principal-component basis.
- The geometry representation decomposes facial shape into an average shape, identity basis, and expression basis coefficients.
- The identity basis is a compact principal-component basis, while expression vectors represent offsets from neutral identity to specific expressions.
- The model is restricted to geometries and expressions spanned by its bases, so training examples must be collected carefully.
- The identity basis contains 200 vector elements and is augmented by 84 vectors representing various expressions.
- Facial texture is modeled as a linear combination of an average texture, texture basis, and coefficient vector learned from registered scans.
5. Learning Framework
The learning framework predicts face geometry directly with an iterative CNN that repeatedly corrects a geometry estimate. It trains using geometry-level error rather than coefficient-level error.
- Framework: The network predicts face geometry from image inputs and operates at the center of the recovery pipeline.The architecture is designed to reconstruct geometry directly from the image.
- Iterative Formulation: Iterative error feedback feeds the original image and a shading image of the previous prediction into the CNN to produce a correction.A linear layer combines the current network output and previous prediction to determine the actual correction.
- Iterative Formulation: Approximately 3 iterations update the geometry vector, shading image, and input masking during evaluation.Evaluation begins from the average shape, α0 = 0, and repeats the refinement procedure.
- Training Criterion: The training criterion uses mean squared error between reconstructed geometries rather than between output representation vectors and ground-truth vectors.Because geometries are aligned, corresponding vertices can be compared directly; the differentiable loss supports training and better convergence.
- Network Architecture: The architecture is based on ResNet and receives a 200x200x2 input, with a 568-to-284 linear layer optimized using ADAM.The figure describes building blocks b[n] with n output maps and padding in all layers.
6. Data Generation
The paper generates large labeled training sets by sampling facial geometries and rendering them into varied synthetic images. It also simulates iterative refinement by pairing input images with intermediate geometry estimates.
- Data Generation: Existing 3D-face datasets contain only a few hundred subjects, so the method generates geometries directly from a morphable model instead of scanning millions of faces.The generated geometries avoid inheriting the shortcomings of a separate reconstruction algorithm used to label 2D images.
- Rendering The Geometries: Synthetic faces are rendered with random Phong shading, including varied ambient, diffuse, specular, and frontal light-direction parameters.The shininess constant is fixed at 10, while the other reflectance constants are sampled around specified means.
- Rendering The Geometries: The shaded geometry is projected onto the image plane using a parallel weak perspective projection.Rotation, translation, and scaling parameters are normally sampled around a front-facing centered-face mean.
- Iterative Data Simulation: Each training sample combines a cropped rendered face image, a current geometry estimate, and the corresponding ground-truth geometry label.The current estimate generates the shading image and masks the rendered facial image before training.
- Iterative Data Simulation: Current geometry estimates are uniformly sampled between a random geometry and the ground truth, creating examples at multiple refinement distances.The same projection is used for both geometries.
- Iterative Data Simulation: The generated data pipeline includes a facial image, shaded secondary geometry, its binary mask, and a cropped facial image, with training pairs marked in red.These stages correspond to the sample-generation sequence illustrated in Figure 6.
7. Extracting Fine Details
The 3DMM constrains reconstructions to its affine subspace, limiting fine-detail recovery; shape-from-shading refines predicted geometry to recover such details.
- 3DMM-based reconstructions are constrained to the affine subspace spanned by 3DMM vectors, preventing fine details such as wrinkles around the eyes.
- Shape-from-shading takes the predicted geometry as input and refines the reconstruction as a post-processing step.
- The refinement requires the predicted geometry to be accurate at a coarse scale before recovering finer details.
8. Results
The method is evaluated on real faces in-the-wild using textured renderings, visual comparisons, and quantitative error analysis. It reconstructs faces across varied conditions and recovers details beyond sparse landmarks.
- Texture missing from naturally occluded facial regions is filled using a linear texture model before rendering the results.
- The method successfully reconstructs in-the-wild faces under different illumination conditions, viewpoints, and expressions.
- Shape-from-shading effects are examined by comparing the input image, network reconstruction, and reconstruction after refinement.
- Compared with reconstruction from 68 image landmarks, the proposed network recovers details including nose shape and wrinkles around the mouth.
- Figure 11 evaluates reconstruction accuracy using pointwise Euclidean distances after non-rigid registration to FRGCv2.0 scans and similarity alignment.
9. Discussion
The approach recovers relatively accurate geometry, with weak-perspective projection outperforming full perspective in the reported experiments. Its main limitations arise from model and synthetic-data coverage.
- Weak-perspective projection outperformed full-perspective projection in the reported experiments.The authors attribute this to ambiguity when initializing face-mask parameters under full perspective.
- More realistic rendering techniques could be incorporated without changing the learning framework, and richer data or models could address observed failures.
- Limitations: The network can fail on faces outside the ethnic origins represented in the scanned-face 3DMM.
- Limitations: Facial attributes absent from the synthetic data, such as thick lips with lipstick or beards, can be misclassified as other image regions.
10. Conclusion
The paper presents an iterative CNN trained on synthetic data to recover facial geometry from single images, with optional shape-from-shading refinement. It generalizes to varied in-the-wild faces and conditions.
- The proposed approach uses an iterative Convolutional-Neural-Network trained with synthetic data to reconstruct facial geometry from a single image.
- An optional shape-from-shading algorithm provides detail refinement after reconstruction.
- The method successfully generalizes to varied expressions and illumination conditions in in-the-wild face images.