Source-linked AI summary
Towards High-fidelity Nonlinear 3D Face Morphable Model
Luan Tran, Feng Liu, Xiaoming Liu
TL;DR
High-fidelity nonlinear 3D face models are difficult to learn because strong regularization is needed to resolve ambiguities but can suppress facial detail. The paper learns shape and albedo proxies and uses a dual-pathway global-local network, achieving high-fidelity reconstruction and state-of-the-art performance by optimizing latent representations.
Problem
Strong regularization needed to resolve ambiguities in nonlinear 3DMM learning can prevent high-fidelity models from representing faces with detailed shape and albedo.
Method
The paper learns regularized shape and albedo proxies and uses a dual-pathway network balancing global structure with local facial-part detail.
Results
The model achieves high-fidelity 3D face reconstruction solely by optimizing latent representations.
Takeaways & Limitations
Improving both the nonlinear 3DMM objective and architecture supports reconstruction that captures mid- and high-level facial details through model fitting.
Abstract
from arXiv · showhide
Embedding 3D morphable basis functions into deep neural networks opens great potential for models with better representation power. However, to faithfully learn those models from an image collection, it requires strong regularization to overcome ambiguities involved in the learning process. This critically prevents us from learning high fidelity face models which are needed to represent face images in high level of details. To address this problem, this paper presents a novel approach to learn additional proxies as means to side-step strong regularizations, as well as, leverages to promote detailed shape/albedo. To ease the learning, we also propose to use a dual-pathway network, a carefully-designed architecture that brings a balance between global and local-based models. By improving the nonlinear 3D morphable model in both learning objective and network architecture, we present a model which is superior in capturing higher level of details than the linear or its precedent nonlinear counterparts. As a result, our model achieves state-of-the-art performance on 3D face reconstruction by solely optimizing latent representations.
1. Introduction
Monocular 3D face reconstruction is ill-posed because shape, albedo, and lighting are blended in one image. The paper addresses limitations of strongly regularized and global models with proxies and a dual-pathway architecture for higher-fidelity reconstruction.
- Monocular reconstruction blends shape, albedo, and lighting into each pixel, making inference from one photograph arduous and ill-posed.
- Strong priors can leave reconstructed faces missing important features, containing incorrect details, or poorly aligned with the input.
- Neural nonlinear 3DMMs improve representation power but still face conflicting regularization requirements for global shape and high-level detail.
- A dual-pathway network combines a global structure pathway with local facial-part pathways to balance robustness and flexibility.
- The model learns additional shape and albedo proxies so regularization can be applied without sacrificing detailed learning of either component.
- The proposed nonlinear 3DMM achieves high-fidelity reconstruction solely by optimizing latent representations.
2. Prior Work
Prior work includes linear 3DMMs built from 3D scans and newer nonlinear models using deep networks. Some nonlinear approaches learn 3DMMs from 2D image collections, while residual learning has also been used for iterative refinement.
- Linear 3DMMs model facial geometry from aligned 3D scans, with later extensions addressing alignment, expression, identity, and texture variation.
- Nonlinear face models use deep networks such as VAEs or learned neural bases to represent facial geometry, shape, and albedo.
- This work uses residual learning for a different purpose: overcoming conflicting objectives in learning 3D models.
3. Proposed Method
The method improves nonlinear 3DMM learning through proxy-and-residual representations and a dual-pathway decoder that balances global structure with local facial detail.
- Nonlinear 3DMM: 3DMM represents faces using shape geometry and albedo skin reflectance, then projects and renders them to synthesize images.The framework uses weak-perspective projection and spherical-harmonics illumination.
- Nonlinear 3DMM: Nonlinear mappings replace linear 3DMM bases with neural networks, increasing representation power but retaining conflicting regularization demands.Global shape requires strong constraints, whereas detailed recovery benefits from weaker regularization.
- Reconstruction Objectives: The reconstruction objective combines image-space and feature-space losses, with sparse landmark alignment as an auxiliary fitting task.The robust l2,1 image loss handles outliers, while perceptual loss compares activations from selected network layers.
- Proxy and Residual Learning: Proxy-and-residual learning applies strong regularization to proxy shape and albedo while weakly encouraging the actual components to remain close to their proxies.The actual shape and albedo receive no direct regularization beyond this weak proximity constraint.
- Proxy and Residual Learning: Pairing actual and proxy components creates multiple reconstruction images, using cross-pairings to promote detailed shape and albedo without sacrificing either component.The approach intentionally omits the reconstruction loss for the image rendered from both actual components to avoid in-between solutions.
- Proxy and Residual Learning: A soft symmetry loss constrains occluded regions by enforcing symmetry on depth residuals, while allowing the full shape to remain asymmetric.The visibility mask is approximated from estimated surface normals in UV space.
- Global-Local-Based Network Architecture: Dual-pathway decoders combine a global pathway for overall structure with four shallower local subnetworks for details around the eyes, nose, and mouth.Local subnetworks use fewer up-sampling steps and part-specific filters to represent high-frequency details efficiently.
4. Experimental Results
Experiments evaluate the framework’s design, representation power, reconstruction quality, identity preservation, and face-editing utility. Across these tests, the model recovers detailed shape and texture and supports latent-only reconstruction.
- The experiments assess framework design, model representation power, and applications to facial analysis.
- Ablation Study: Combining perceptual loss with l2,1 produces more realistic reconstructions than l2,1 alone, which yields blurry images.
- Ablation Study: Proxy-based pairing encourages shape and albedo to capture high-level details while residual soft symmetry preserves detail in occluded regions.
- Representation Power: The nonlinear model significantly outperforms the Basel Face Model for in-the-wild texture representation, while proxy training recovers more realistic high-frequency facial details.
- 3D Reconstruction: The model reconstructs detailed face shapes and compares against linear, nonlinear, and beyond-3DMM methods on CelebA.
- Representation Power: 77.2% of images have the smallest identity-space reconstruction difference for the proposed model among three compared models.
- Applications: Estimated shape, projection, and lighting enable UV-space texture editing for stickers, tattoos, and make-up.
5. Conclusions
The paper identifies strong regularization and global-only modeling as obstacles to high-fidelity nonlinear 3DMMs. It addresses both through improved learning objectives and architecture, aiming for detailed reconstruction from latent fitting alone.
- Strong regularization and global-based modeling are identified as roadblocks to high-fidelity 3DMMs.
- The proposed approach improves nonlinear 3DMM learning objectives and network architecture to capture mid- and high-level facial details.
- High-fidelity 3D face reconstruction can be achieved solely by fitting model representations.