Source-linked AI summary
Unrestricted Facial Geometry Reconstruction Using Image-to-Image Translation
Matan Sela, Elad Richardson, Ron Kimmel
TL;DR
Existing single-image facial reconstruction methods restrict solutions to low-dimensional models, limiting geometric expressiveness. The paper instead predicts depth and dense facial correspondence in the image domain, then applies geometric deformation and refinement; qualitative and quantitative analyses report accurate, robust reconstruction, while strong occlusions and rotations remain challenging.
Problem
Existing face reconstruction methods commonly restrict solutions to low-dimensional subspaces, limiting their expressiveness for diverse facial geometry.
Method
An image-to-image network maps a facial image to aligned depth and dense facial correspondence maps, followed by iterative geometric deformation and image-guided refinement.
Results
Qualitative and quantitative evaluations report accurate and robust reconstruction of real faces from networks trained only on synthetic facial images.
Takeaways & Limitations
The pixel-based representation supports high-quality reconstructions beyond the limited scope of the training generative model and preserves geometry during synthetic-to-real image translation.
Takeaways & Limitations
Extreme occlusions absent from training data and strong rotations produce missing areas or errors in the representation and reconstruction.
Abstract
from arXiv · showhide
It has been recently shown that neural networks can recover the geometric structure of a face from a single given image. A common denominator of most existing face geometry reconstruction methods is the restriction of the solution space to some low-dimensional subspace. While such a model significantly simplifies the reconstruction problem, it is inherently limited in its expressiveness. As an alternative, we propose an Image-to-Image translation network that jointly maps the input image to a depth image and a facial correspondence map. This explicit pixel-based mapping can then be utilized to provide high quality reconstructions of diverse faces under extreme expressions, using a purely geometric refinement process. In the spirit of recent approaches, the network is trained only with synthetic data, and is then evaluated on in-the-wild facial images. Both qualitative and quantitative analyses demonstrate the accuracy and the robustness of our approach.
1. Introduction
The paper targets single-image facial geometry recovery across varied expressions and poses, addressing limits imposed by sparse landmarks, reference models, and low-dimensional morphable models. It proposes an unrestricted image-to-image representation combined with geometric deformation and refinement.
- Single-image facial geometry recovery is studied for faces under a wide range of expressions and poses.
- Three-dimensional morphable models provide efficient recovery but express only coarse geometries within a low-dimensional linear subspace.
- Landmark-only methods constrain reconstruction to sparse points, while reference models can fail on extreme expressions and non-frontal poses.
- The proposed network predicts an image-domain representation containing a depth image and a facial-template embedding from one input image.
- A purely geometric deformation and refinement procedure uses the learned representation to produce high-quality facial reconstructions.
- The network also translates synthetic facial images into realistic ones while preserving geometric structure.
2. Overview
The reconstruction pipeline maps a facial image to depth and dense correspondence outputs, then converts them into a complete, detailed facial mesh through geometric deformation and refinement.
- The network outputs an estimated depth map aligned with the input and a dense pixel-to-vertex map on a reference facial mesh.
- An iterative non-rigid three-dimensional deformation warps a template mesh to establish full vertex correspondence and complete occluded face regions.
- A final image-guided detail reconstruction recovers subtle facial geometry.
3. Learning the Geometric Representation
The method learns a pixel-based geometric representation from facial images using depth and dense correspondence maps, with synthetic training data and losses designed for detailed depth prediction.
- Geometric representation: The network represents each facial image with an estimated depth image and a correspondence map to a template facial mesh.The correspondence embedding is deformation agnostic and maps image pixels to template-model points.
- Training data: Training samples are generated by rendering randomly sampled morphable-model faces under varied illumination conditions and poses.Synthetic data provides known geometry labels and a deterministic image-to-label connection, although generalization to in-the-wild images can remain challenging.
- Network and training: The architecture follows an Image-to-Image translation framework using a U-net-style encoder-decoder with skip connections for pixel-wise prediction.The framework omits adversarial loss because the mapping is considered less ambiguous and distributional loss can introduce artifacts.
- Network and training: The loss combines pixel-wise L1 matching with a normals-based term to improve fine geometric structures in the predicted depth.The normals loss addresses the L1 loss limitation demonstrated by artifacts in the L1-only result; correspondence uses only L1 loss.
4. From Representations to a Mesh
The predicted depth and correspondence maps are converted into a complete, template-corresponding facial mesh through iterative non-rigid registration and image-guided fine-detail reconstruction.
- Mesh reconstruction: The pipeline converts the network’s depth and correspondence outputs into a 3D mesh with full vertex correspondence to a fixed-triangulation template.It combines iterative elastic deformation with a subsequent fine-detail recovery step.
- Iterative registration: Registration first meshes the depth map, computes an affine template alignment from correspondence, and then applies iterative non-rigid deformation.The deformation aligns the transformed template to the depth-derived mesh while warping only the template.
- Iterative registration: Nearest-neighbor matching in correspondence-embedding space lets one template face register to different identities with arbitrary expressions.Physically distant matches and pairs with disagreeing normal directions are discarded before deformation.
- Iterative registration: The deformation energy balances squared correspondence distances, tangent-plane distances, and mesh stiffness.When motion falls below a threshold, stiffness is reduced to permit greater deformation in the next iteration; termination occurs below a stiffness threshold.
- Fine detail reconstruction: The registered template supplies the overall facial structure, while fine-detail recovery uses high-frequency image texture to displace vertices along surface normals.A curvature-based regularizing displacement reduces the probability of triangle collisions during detail reconstruction.
- Fine detail reconstruction: The fine-detail stage produces the final surface by combining texture-guided displacement with regularization and mesh fairing.The process is illustrated as input image, registered shape, high-frequency texture component µ(v), and final shape.
5. Experiments
Experiments evaluate the network outputs, reconstruction pipeline, qualitative comparisons, and geometric-constraint application. The proposed representation supports depth and correspondence recovery and preserves facial structure during synthetic-to-real translation.
- Network evaluation: The network produces depth and correspondence outputs that are used throughout the reconstruction pipeline.
- Quantitative evaluation: The evaluation uses the first 200 BU-3DFE subjects with aligned ground-truth depth images, accounting for valid pixels and global depth scaling and translation.
- Reconstruction pipeline: The reconstruction stages proceed from the input and network reconstruction to template registration and final shape refinement.
- Qualitative evaluation: Qualitative comparisons assess input images alongside reconstructions from different methods, including zoomed views of selected subjects.
- Geometric constraint: The proposed network is also used as an additional geometric loss for translating synthetic facial images into realistic ones while preserving their underlying geometry.
6. Limitations
The model-free, high-dimensional representation generalizes to in-the-wild images but remains limited by domain deviation, occlusion, strong rotation, and registration speed.
- The solution space is represented in the high-dimensional image domain rather than restricted to a low-dimensional facial subspace.
- Facial images that differ substantially from the training domain can produce missing areas and errors in the representation maps.
- Extreme occlusions such as sunglasses, hands, and beards are difficult because they were absent from the training data.
- Strong rotations are not well handled, and template registration can take a few minutes because it is not optimized for speed.
7. Conclusion
The paper presents an unrestricted single-image facial reconstruction pipeline based on pixel-based geometric prediction and geometric refinement. It also uses the network as a loss for realistic image translation that preserves facial structure.
- The algorithm maps a facial image to a pixel-based geometric representation, then applies geometric deformation and refinement to reconstruct the face.
- The network is trained only on synthetic facial images yet reconstructs real faces and supports synthetic-to-real translation while preserving geometric structure.
Supplementary Material
The supplementary material describes network architecture, training resolution, representation-learning choices, scale prediction, and the iterative registration procedure.
- The architecture extends an Image-to-Image design with three additional 1 × 1 convolution layers for further refinement.
- Training uses 512×512-pixel input and output images, and the network did not successfully scale to unseen resolutions.
- Although one network can recover both depth and correspondence, experiments prefer separate networks with identical architectures for the two representations.
- The registration scale is predicted from per-pixel x, y, and z coordinates rather than estimated only from global facial features.
- The registration phase is an iterative deformation-based procedure that includes fine-detail reconstruction implementation.
B.1. Non-Rigid Registration
The method registers a template mesh to the network-generated target using dense correspondence, robust filtering, and iterative non-rigid deformation. The process produces a complete, smoother template preserving the recovered facial structure.
- Correspondence Initialization: The network’s correspondence map associates each template vertex with a target vertex using nearest neighbors in embedding space.Unlike Euclidean nearest-neighbor registration, embedding-space matching supports different identities and arbitrary expressions.
- Robust Matching: Physically distant matches and pairs with normal disagreement above 5 degrees are excluded before deformation.The thresholds are 1 millimeter for distance and 5 degrees for normal disagreement.
- Non-Rigid Deformation: The template is deformed by minimizing an energy combining correspondence distances, tangent-plane distances, and biharmonic smoothness.The point-to-point term has weight αp2point = 0.1, while biharmonic weights regularize the deformation.
- Adaptive Iteration: Outer iterations relax stiffness when motion becomes small and switch toward physical nearest-neighbor associations when match counts stabilize.Stiffness is reduced when motion falls below 0.1, and association updates depend on a match-count difference below 500.
- Output: The process terminates when αmemb falls below 10^6, yielding a fixed-triangulation template that is smoother and complete.The output retains the network-recovered overall facial structure while filling holes and completing occluded regions.
B.2. Fine Detail Reconstruction
Fine detail reconstruction extracts high-frequency image information and converts it into normal-direction vertex displacements. Regularization based on curvature helps limit geometric artifacts and triangle collisions.
- Texture Preparation: The reconstruction starts from a subdivided deformed template whose vertices inherit intensity values from nearest image pixels.Only the high spatial frequencies of the resulting texture are used for fine-detail recovery.
- Frequency Decomposition: The mesh-based filter computes μ(v) using the identity matrix, the cotangent-weight discrete Laplacian, and dt = 0.2.The scalar dt controls the filter’s cut-off frequency.
- Detail Displacement: Each vertex is displaced along its normal using a data-driven term proportional to local high-frequency texture variation.Neighboring vertices contribute through weighted 1-ring averaging, with additional spatial attenuation available in curved regions.
- Regularization: A curvature-proportional regularizing displacement acts as an explicit mesh-fairing step to reduce triangle intersections in high-curvature regions.The final modification combines the data-driven displacement with this regularization term.
C. Additional Experimental Results
Additional experiments show that the network generalizes across varied faces but remains vulnerable to occlusions, facial hair, and extreme poses. Correspondence errors are visualized through texture mapping.
- Network Outputs: Figure 14 displays network output representations for varied faces and includes failure cases in its last two rows.The results indicate broad generalization alongside identifiable failure cases.
- Failure Cases: The network may fail when images contain occlusions, facial hair, or extreme poses because it remains limited by synthetic training data.These limitations are also examined through correspondence-error visualizations.
- Correspondence Evaluation: Figure 15 places input images beside matching correspondence results, using texture mapping to make correspondence errors visible.The visualization uses the color value of the corresponding projected vertex.
- Additional Results: Figures 16 and 17 provide additional reconstruction results and error heat maps expressed as percentiles of ground-truth depth.Together, these figures extend qualitative reconstruction and depth-error inspection.