Source-linked AI summary
3D Face Reconstruction with Geometry Details from a Single Image
Luo Jiang, Juyong Zhang, Bailin Deng, Hao Li, Ligang Liu
TL;DR
Single-image 3D face reconstruction is challenging because projection discards 3D information, while existing methods face dataset or shape-from-shading limitations. The paper combines bilinear fitting, local corrective deformation, and shape-from-shading in a coarse-to-fine pipeline. Experiments report accurate reconstructions with recovered fine details and performance exceeding existing example-based and shape-from-shading methods.
Problem
Single-image 3D face reconstruction is challenging because camera projection loses information, while existing approaches can depend on datasets or prior geometry and illumination assumptions.
Method
A bilinear face model is fitted using landmarks, refined with local corrective deformation, and processed with shape-from-shading to recover fine geometry.
Results
The method outperforms existing example-based and shape-from-shading methods in reconstruction accuracy and geometric detail recovery.
Takeaways & Limitations
Combining low-dimensional face modeling with shape-from-shading enables more accurate and robust reconstruction with details such as wrinkles and teeth.
Takeaways & Limitations
Performance depends on whether the constructed face model covers the target’s overall shape; faces far outside FACEWAREHOUSE and BFM2009 may be reconstructed poorly.
Abstract
from arXiv · showhide
3D face reconstruction from a single image is a classical and challenging problem, with wide applications in many areas. Inspired by recent works in face animation from RGB-D or monocular video inputs, we develop a novel method for reconstructing 3D faces from unconstrained 2D images, using a coarse-to-fine optimization strategy. First, a smooth coarse 3D face is generated from an example-based bilinear face model, by aligning the projection of 3D face landmarks with 2D landmarks detected from the input image. Afterwards, using local corrective deformation fields, the coarse 3D face is refined using photometric consistency constraints, resulting in a medium face shape. Finally, a shape-from-shading method is applied on the medium face to recover fine geometric details. Our method outperforms state-of-the-art approaches in terms of accuracy and detail recovery, which is demonstrated in extensive experiments using real world models and publicly available datasets.
I. INTRODUCTION
Single-image 3D face reconstruction is important but difficult because camera projection loses information. The paper proposes a coarse-to-fine method combining example-based modeling, local deformation, and shape-from-shading to recover accurate faces with fine details.
- Camera projection loses information, making single-image 3D face reconstruction particularly challenging despite its applications in recognition and animation.
- Example-based methods depend on their datasets and limited model freedom, while shape-from-shading requires prior geometry or illumination assumptions.
- The method first fits a parametric face model derived from FACEWAREHOUSE and BFM2009 to capture the target’s overall shape.
- It then applies smooth deformation to capture medium-scale features and estimates lighting and reflectance from the enhanced face.
- Finally, illumination and the enhanced face produce a height-field surface that captures fine geometric details.
- The combined approach is reported to outperform existing example-based and shape-from-shading methods in reconstruction accuracy and detail recovery.
II. RELATED WORK
Prior work mainly uses low-dimensional example-based face models or shape-from-shading. These approaches offer complementary strengths but inherit dataset, ambiguity, and assumption-related limitations.
- Low-dimensional face representations reduce reconstruction to searching a compact parameter space learned from example faces.
- Example datasets strongly influence low-dimensional models, limiting reconstruction when expressions or identities differ substantially from the samples.
- Shape-from-shading recovers fine geometry from shading when illumination, projection, and reflectance are known, but remains ill-posed and ambiguous.
III. OVERVIEW
The pipeline reconstructs a face from one photograph in stages: landmark-guided coarse modeling, person-specific deformation, and subsequent fine-detail recovery.
- The overview describes the complete procedure as a coarse-to-fine reconstruction approach for a high-quality 3D face from one photograph.
- The pipeline begins by fitting a bilinear model built from FACEWAREHOUSE and BFM2009 using projected 3D landmarks and detected 2D landmarks.
- An additional deformation field enhances the coarse face with person-specific features that the bilinear model cannot represent.
IV. COARSE FACE MODELING
The coarse model combines a bilinear face representation with landmark-based fitting, iterative optimization, and view-dependent silhouette updates. This produces a plausible initial face while improving correspondence for non-frontal images, although accuracy degrades beyond large poses.
- Bilinear face model: A bilinear face model represents identity and expression variation using reduced identity and expression weights while retaining facial-expression variety.The model is derived from FACEWAREHOUSE and BFM2009 through 2-mode SVD reduction.
- Landmark fitting: The coarse face is fitted by aligning 3D landmark projections with 2D image landmarks while jointly optimizing shape and camera parameters.The fitting energy uses landmark correspondences and is minimized over identity, expression, rotation, and translation through coordinate descent.
- Optimization: Four coordinate-descent iterations are sufficient in experiments to reach a good coarse reconstruction.The optimization alternates between shape-related and camera-related parameters before updating expression parameters.
- Silhouette update: For non-frontal views, silhouette landmark vertices are updated from the current rotation using horizontal mesh lines and view-dependent vertex selection.Internal landmarks remain unchanged, while projected silhouette vertices are reassigned according to the rotated mesh and view direction.
- Silhouette update: Silhouette updating improves coarse-model accuracy at +30° and +45° yaw by avoiding erroneous silhouette correspondences that make the model too wide.At poses beyond 60° yaw, the approach does not work well unless invisible landmarks can be accurately detected.
V. MEDIUM FACE MODELING
The medium face model refines a coarse example-based face with smooth, low-dimensional local deformations while jointly estimating lighting and albedo. This improves medium-scale feature accuracy, but the chosen local frequency basis cannot recover all facial details or global-scale errors.
- Lighting and albedo estimation: Lighting and albedo are estimated from the enhanced face using a Lambertian spherical-harmonics model and a PCA reflectance model.The experiments retain 100 principal albedo components from BFM2009.
- Optimization: The deformation optimization is solved efficiently in the reduced subspace and iteratively updates image–mesh correspondences, lighting, albedo, and the face model.The refinement and estimation process is repeated twice in the experiments.
- Medium face modeling: Medium face modeling uses smooth local deformation fields to reduce shading discrepancy, with nine regions and five Laplacian eigenfunctions per region in the experiments.The deformation is optimized in a reduced subspace, penalizing large low-frequency deformations more strongly.
- Results: Medium face modeling improves medium-scale features around laugh lines and produces more accurate results numerically and visually than fine reconstruction without it.The comparison is shown in Fig. 7, while Fig. 8 illustrates the improvement around laugh lines.
- Limitations: The local basis cannot reconstruct features whose frequency bands have limited overlap with the selected eigenvectors, such as dimples, and cannot reduce global-scale reconstruction errors.Changing the coarse model's identity parameters exposes the dependence of later reconstruction on the initial coarse face.
VI. FINE FACE MODELING
Fine face modeling reconstructs a height-field surface by optimizing a detailed normal map from image shading, while regularizing it toward the medium face and integrability. Gradient-based shading consistency reduces illumination-noise effects and unnatural sharp creases.
- Fine face reconstruction: The final model converts an optimized normal map into a height-field surface, using the medium face and estimated lighting and albedo as inputs.The normal map is integrated to recover per-pixel depth values for the final face shape.
- Normal map optimization: Gradient-difference minimization attenuates cast-shadow influence while preserving facial features and produces smoother results with fewer unnatural creases than intensity-difference minimization.The improvement is illustrated around the eye, nose, and mouth in Fig. 9.
- Normal map optimization: The normal-map optimization combines shading consistency with deviation, smoothness, and integrability regularization to constrain the recovered geometry.The integrability condition ensures that the normal map corresponds to an existing height field.
- Results: The fine reconstruction captures more geometric details and reduces reconstruction error relative to the medium face model, while remaining sensitive to the initial coarse face identity.Figure 7 compares reconstructions from the original and modified coarse models.
VII. EXPERIMENTS
Experiments evaluate the method on controlled and unconstrained images across poses and expressions, with quantitative and visual comparisons against existing approaches. The method achieves lower frontal-neutral error, reconstructs varied poses and expressions, and performs competitively on unconstrained data while recovering detailed geometry.
- Frontal and neutral faces: 1.56 ± 0.24 3DRMSE versus 1.97 ± 0.35 for method demonstrates lower mean error on frontal neutral faces.The comparison used 55 subjects from the Bosphorus database.
- Near-frontal poses and expressions: Reconstructions remain consistent across Yaw +10°, +20°, and +30° poses and happy, surprise, and disgust expressions.Results for four subjects are visualized in Fig. 11, with small reconstruction errors reported for varied poses and expressions.
- Large poses: The method reconstructs large-pose images well when paired with landmark detection designed for poses such as 90°.Two large-pose examples are shown in Fig. 13.
- Unconstrained facial images: On MICC, the reconstruction error is close to method and lower than method on outdoor images with structured-light ground truth.The MICC data span varying resolutions, conditions, zoom levels, and controlled to uncontrolled environments.
- Unconstrained facial images: Visual comparison on unconstrained images shows more accurate overall shape and more recovered geometric details, including wrinkles and teeth, than SFS method.No ground-truth shapes were available for this comparison, so evaluation was visual.
VIII. DISCUSSION AND CONCLUSION
The method combines a bilinear face model, local corrective deformation, and shape-from-shading to reconstruct faces across poses and expressions with fine details. Its main limitation is dependence on whether the constructed face model covers the target’s overall shape, while lighting and albedo remain ambiguous.
- Discussion: Performance depends on whether the constructed face model covers the target’s overall shape; faces far from both source datasets may reconstruct poorly.The authors identify broader face datasets as a potential way to improve coverage.
- Discussion: Albedo-lighting ambiguity is not fully resolved, although the PCA albedo and second-order spherical-harmonics lighting model is considered sufficient for geometry recovery.The limitation follows from computing pixel values by multiplying albedo with lighting.
- Conclusion: The coarse-to-fine pipeline combines reliable large- and medium-scale shape estimation with shape-from-shading recovery of fine details such as wrinkles and teeth.The approach is presented as combining the benefits of low-dimensional face models and shape-from-shading for more accurate and robust reconstruction.