Source-linked AI summary
Learning Detailed Face Reconstruction from a Single Image
Elad Richardson, Matan Sela, Roy Or-El, Ron Kimmel
TL;DR
Detailed face reconstruction from a single image is difficult because facial appearance and geometry are highly variable and ambiguous. The paper proposes an end-to-end coarse-to-fine CNN with a differentiable rendering layer, trained first on synthetic data and then unsupervised on real facial images. The framework produces detailed reconstructions and outperforms recent state-of-the-art approaches, while synthetic-data training can limit generalization to unseen facial features.
Problem
Recovering detailed facial geometry from one image remains difficult because faces vary in expression, pose, texture, material, lighting, viewing direction, and intrinsic geometry.
Method
An end-to-end coarse-to-fine CNN uses CoarseNet for rough geometry and FineNet for detail refinement, connected by a differentiable rendering layer and trained with synthetic and unlabeled real images.
Results
The proposed framework produces detailed reconstructions and outperforms recent state-of-the-art approaches in the reported comparisons.
Takeaways & Limitations
The framework reconstructs detailed facial geometry from a single image without requiring external initialization or post-processing algorithms.
Takeaways & Limitations
Synthetic-data training can limit generalization to facial features absent from training, including beards, makeup, and glasses.
Abstract
from arXiv · showhide
Reconstructing the detailed geometric structure of a face from a given image is a key to many computer vision and graphics applications, such as motion capture and reenactment. The reconstruction task is challenging as human faces vary extensively when considering expressions, poses, textures, and intrinsic geometries. While many approaches tackle this complexity by using additional data to reconstruct the face of a single subject, extracting facial surface from a single image remains a difficult problem. As a result, single-image based methods can usually provide only a rough estimate of the facial geometry. In contrast, we propose to leverage the power of convolutional neural networks to produce a highly detailed face reconstruction from a single image. For this purpose, we introduce an end-to-end CNN framework which derives the shape in a coarse-to-fine fashion. The proposed architecture is composed of two main blocks, a network that recovers the coarse facial geometry (CoarseNet), followed by a CNN that refines the facial features of that geometry (FineNet). The proposed networks are connected by a novel layer which renders a depth image given a mesh in 3D. Unlike object recognition and detection problems, there are no suitable datasets for training CNNs to perform face geometry reconstruction. Therefore, our training regime begins with a supervised phase, based on synthetic images, followed by an unsupervised phase that uses only unconstrained facial images. The accuracy and robustness of the proposed model is demonstrated by both qualitative and quantitative evaluation tests.
1. Introduction
The paper addresses the difficulty of recovering detailed 3D facial geometry from one image by proposing an end-to-end coarse-to-fine CNN framework trained with synthetic and unlabeled facial images.
- Single-image facial geometry recovery is difficult because faces vary in expressions, poses, textures, materials, lighting, viewing direction, and intrinsic geometry.
- The proposed framework reconstructs a detailed facial surface by dividing the task between CoarseNet for coarse geometry and pose and FineNet for fine details.FineNet operates on depth maps and is not constrained by the morphable-model representation.
- The end-to-end design aims to provide detailed reconstruction without external initialization or post-processing algorithms.
- A novel rendering layer connects the 3DMM output to the refinement network by enabling back-propagation from a rendered depth map.
- Training bypasses manually labeled detailed-face data by combining synthetic supervised training with unsupervised training on unlabeled facial images.
2. Related Work
Prior single-image face-reconstruction methods use constrained morphable models, deformable templates, or data-driven regression, while the proposed approach combines elements of these strategies.
- Single-image face reconstruction is ill-posed because one image leaves intrinsic geometric ambiguities.
- 3DMM Methods: 3DMM methods constrain solutions to likely faces using a principal-components representation, simplifying reconstruction but relying on a restricted model space.
- Template-Based Methods: Template-based methods deform or align a reference face and refine it using appearance, depth, or shape-from-shading objectives.
- Data-Driven Methods: Data-driven methods regress from landmarks or image features, including networks that directly reconstruct geometry from images.
- The proposed solution combines a 3DMM, template-based refinement, and data-driven learning.
3. Coarse Geometry Reconstruction
The coarse reconstruction stage models facial shape and pose with a 3DMM, iteratively refines predictions using rendered feedback, and trains CoarseNet on synthetic examples.
- The first stage extracts coarse facial geometry and pose, integrating iterative 3DMM-based alignment and reconstruction.
- Modeling The Solution Space: Facial shape is represented with an average face, identity principal components, and expression blendshapes.The representation uses 200 identity coefficients and 84 expression coefficients.
- Modeling The Solution Space: A weak-perspective projection models image formation, while the pose uses 6 parameters and the combined geometry-and-pose representation has 290 parameters.
- Feedback Representation: Iterative feedback represents the current coarse geometry with PNCC and normal-map channels, with normal maps adding local geometric information.
- Acquiring The Data: Because large datasets of accurately scanned 3D faces are unavailable, training data are synthesized by rendering sampled geometry and pose with random appearance factors.
- The CoarseNet Architecture and Criterion: CoarseNet uses a ResNet architecture and separate geometry and pose losses, with geometry trained by GMSE and pose by a 6-parameter MSE.
- Iterative Refinement: The network starts from a centered mean face and updates geometry iteratively using masked inputs and newly rendered feedback until convergence.
4. The Coarse to Fine Approach
The approach transfers coarse 3DMM reconstruction into depth-map space, where FineNet recovers details using image-based shape-from-shading cues. A differentiable rendering layer and unsupervised criterion connect refinement back to the coarse model.
- Fine-detail refinement: Fine details such as wrinkles and dimples are recovered in depth-map space because the 3DMM representation cannot capture their diversity.FineNet modifies the coarse depth map using the original image.
- Rendering layer: The rendering layer converts CoarseNet’s geometry and pose representation into a 200 × 200 depth map for FineNet.It first constructs and positions the 3D mesh, then renders it with a z-buffer renderer.
- Rendering layer: During backpropagation, pixel gradients pass through matching mesh vertices and back to the morphable-model representation using barycentric-coordinate weighting.Hidden vertices receive no gradients because they do not appear in the output depth map.
- FineNet architecture: FineNet uses a hypercolumn architecture that concatenates feature responses from multiple convolution layers to combine structural and semantic information per pixel.The feature maps are interpolated to the input resolution, using learned upconvolution layers in the implementation.
- Unsupervised loss: Because detailed face geometry lacks suitable paired training data, FineNet is trained with an image-formation loss based on shape from shading.The loss connects the predicted depth map to the input intensity image through recovered albedo and lighting, with fidelity and smoothness regularization.
- Unsupervised loss: The shape-from-shading formulation constrains albedo to a low-dimensional 3DMM texture subspace, while an already trained CoarseNet supplies the essential coarse initialization.A fidelity criterion limits CoarseNet’s deviation during end-to-end fine-tuning.
5. Experiments
The experiments evaluate the proposed framework on 3D facial datasets and unconstrained images using qualitative and quantitative comparisons. Results indicate detailed reconstructions, robustness to pose and expression, and graceful scaling to larger inputs.
- The framework is compared with template-based, 3DMM-based, and data-driven methods, all of which require alignment information unlike the proposed method.
- Qualitative tests on 400 × 400 in-the-wild face images show fine details and greater robustness to expressions and poses than compared methods.The method also produces comparable high-quality geometry to a multiple-image reconstruction method using only one image.
- Quantitative evaluation uses roughly two thousand aligned facial images with ground-truth depth, comparing errors on pixels valid for every method.
- FineNet accepts varying input sizes, allowing more details to be extracted from high-quality images and scaling gracefully from 200 × 200 training inputs to 400 × 400 inputs.
6. Discussion
The discussion examines qualitative robustness and input scaling alongside limitations from synthetic-data training. The method handles pose and expression variation, but unusual facial features can challenge generalization.
- The method shows robustness to extreme orientations, including nearly 90° angles.
- Qualitative comparisons present reconstructions from two viewpoints and report robustness to pose and expression variations while preserving subtle facial details.
- FineNet outputs are compared for 200 × 200 and 400 × 400 inputs, with the latter using a larger input resolution.
- Synthetic-data training can limit generalization, and the network may fail on beards, makeup, and glasses absent from its training data.
- Unsupervised end-to-end training successfully trains FineNet but only slightly tunes CoarseNet, whose 3DMM representation mainly captures coarse geometry.
- The quantitative comparison reports depth-estimation errors for the different methods.
7. Conclusion
The paper concludes with an end-to-end two-stage system for detailed single-image face reconstruction, combining synthetic and unsupervised real-image training through differentiable rendering. Comparisons demonstrate that it outperforms recent state-of-the-art approaches.
- The framework combines rough geometry recovery with fine-detail reconstruction, training the two stages on synthetic and real facial images respectively.
- A differentiable rendering layer connects the coarse and fine reconstruction networks.
- Comparisons demonstrate that the proposed framework outperforms recent state-of-the-art approaches.
A. Supplementary Qualitative Results
Supplementary results provide additional qualitative comparisons and visualize the diversity of synthetic training examples. The comparisons emphasize alignment, facial detail, and expression robustness.
- The network infers face alignment without external information, producing alignment similar to the state-of-the-art alignment from.
- Additional comparisons report finer facial details than [34] and greater robustness to expressions than the template-based method of.
- Figures 12 and 13 present additional qualitative results.
- Synthetic examples vary in geometry, texture, illumination, and reflectance while using random backgrounds.
B.1. Generalizing from Synthetic Data
The experiments examine synthetic-data generalization and internal network behavior, including convergence, depth, and input-resolution choices. Synthetic training remains vulnerable to unseen facial details, although the proposed method handles some cases better than the comparison method.
- Generalizing from Synthetic Data: Synthetic training can fail on unseen details such as glasses or facial hair, although the proposed method produces more viable beard-case reconstructions than [34].Both methods show robustness to eyeglasses, but dominant beards can misalign the chin or mouth.
- Additional Experiments: The supplementary experiments include convergence analysis and comparisons of FineNet results across network depths.These analyses correspond to Figures 16 and 17.
- Convergence: CoarseNet converges after a few iterations, as measured by the average change in its output across iterations.Figure 16 plots average MSE as a function of iteration number.
- Network Depth: FineNet’s depth was evaluated using 1 to 4 VGG Face Net blocks, with more blocks increasing network size, training cost, and runtime complexity.The figure compares FineNet results produced with 1, 2, 3, and 4 blocks.
- Input Resolution: FineNet accepts different input sizes and can extract more details from higher-resolution inputs, whereas CoarseNet uses a fixed 200 × 200 input.CoarseNet recovers only coarse geometry and therefore does not benefit from higher resolution.
D. Supplementary Quantitative Analysis
The quantitative analysis uses absolute depth error heat maps to compare the proposed method with other reconstruction techniques. The analysis focuses on their typical spatial error distributions.
- Quantitative Analysis: Absolute depth error heat maps compare the proposed method with techniques [34].The heat maps show the typical error distribution for each method.