Source-linked AI summary
Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network
Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, Xi Zhou
TL;DR
Monocular 3D face reconstruction and alignment remain difficult because existing approaches face limitations from large poses, occlusions, model-space constraints, or complex intermediate processing. The paper proposes PRN, which regresses a semantically meaningful UV position map with a weighted loss from one image. It reports more than 25% relative improvement over state-of-the-art methods on both tasks while processing an image in 9.8ms.
Problem
Existing alignment and reconstruction approaches face difficulties with large poses or occlusions and often rely on constrained model spaces, templates, or complex transformations.
Method
PRN regresses a UV position map encoding complete 3D facial coordinates and semantic correspondence from a single image, using a weighted loss focused on discriminative regions.
Results
More than 25% relative improvements over other state-of-the-art methods are reported on both 3D face reconstruction and dense face alignment.
Takeaways & Limitations
The method provides complete 3D structure and dense alignment in one lightweight, real-time-suitable pass without intermediate 3DMM or TPS parameters.
Takeaways & Limitations
Evaluation on AFLW2000-3D is bounded by controversy over the dataset's ground-truth annotations.
Abstract
from arXiv · showhide
We propose a straightforward method that simultaneously reconstructs the 3D facial structure and provides dense alignment. To achieve this, we design a 2D representation called UV position map which records the 3D shape of a complete face in UV space, then train a simple Convolutional Neural Network to regress it from a single 2D image. We also integrate a weight mask into the loss function during training to improve the performance of the network. Our method does not rely on any prior face model, and can reconstruct full facial geometry along with semantic meaning. Meanwhile, our network is very light-weighted and spends only 9.8ms to process an image, which is extremely faster than previous works. Experiments on multiple challenging datasets show that our method surpasses other state-of-the-art methods on both reconstruction and alignment tasks by a large margin.
1 Introduction
The paper introduces PRN, an end-to-end method that jointly reconstructs 3D facial shape and dense alignment from one image. Its UV position map and weighted loss support accurate, fast prediction without relying on a prior face model.
- PRN jointly predicts dense face alignment and 3D face shape directly from a single 2D facial image.
- The UV position map records complete 3D facial coordinates while preserving the semantic meaning of each UV location.
- 9.8ms per image enables the lightweight network to produce reconstruction and alignment results in one pass.
- The method is reported to be robust to poses, illuminations, and occlusions, with qualitative results showing alignment and reconstructed shapes.
- A weighted loss assigns greater emphasis to discriminative facial regions during position-map regression.
- More than 25% relative improvements over other state-of-the-art methods are reported on both 3D reconstruction and dense alignment tasks.
2 Related Works
Earlier reconstruction and alignment methods commonly rely on 3DMMs, facial templates, or intermediate transformations. PRN instead establishes dense correspondence directly from its regressed position map, avoiding those intermediate parameters and complex operations.
- 3DMM-based reconstruction constrains facial geometry to a model shape space and requires post-processing to generate a 3D mesh.
- Some non-3DMM methods still depend on a 3D facial template, while related operations include perspective projection and TPS transformation.
- 2D landmark alignment regresses only visible points, limiting face-shape description under large poses.
- PRN directly establishes dense correspondence between face regions and a 3D template after regressing the position map.
- By avoiding 3DMM coefficients and TPS warping parameters, PRN bypasses complex operations and can run very fast.
3 Proposed Method
The proposed method represents complete 3D facial geometry in UV space and uses a CNN to regress this representation from a single image. A weighted loss emphasizes informative facial regions while supporting joint reconstruction and dense alignment.
- 3.1 3D Face Representation: UV position maps record 3D positions of all facial points in UV space while preserving their semantic correspondence.They replace texture-map color channels with x, y, z coordinates and represent the full facial structure.
- Implementation Details: Training pairs are formed from 300W-LP images and corresponding 3D shapes recovered from annotated 3DMM parameters.The images are cropped and resized to 256 × 256 before training; augmentation includes geometric, color, and synthetic occlusion perturbations.
- 3.1 3D Face Representation: The representation enables a CNN to jointly predict complete 3D facial structure and dense alignment from unconstrained 2D images.Because the position map includes invisible facial regions, the method can predict a complete 3D face rather than only visible areas.
- 3.2 Network Architecture and Loss Function: PRN uses an encoder-decoder CNN with 10 residual blocks and 17 transposed convolution layers to map a 256 × 256 × 3 image to a 256 × 256 × 3 position map.The encoder reduces the input to 8 × 8 × 512 feature maps before decoding the predicted position map.
- 3.2 Network Architecture and Loss Function: The weighted loss assigns larger weights to central and landmark regions because facial features there are more discriminative than other regions.The 68 keypoints receive the highest weight, while neck points receive zero weight to reduce disturbance from less relevant or occluded regions.
- 3.2 Network Architecture and Loss Function: The experiments use a 16:4:3:0 weight ratio for landmarks, eye-nose-mouth regions, other face areas, and the neck, respectively.The weight mask has the same size as the position map and maintains pixel-to-pixel correspondence.
4 Experimental Results
Experiments evaluate PRN on multiple datasets for dense alignment and 3D face reconstruction, including challenging poses. The method achieves strong accuracy, robustness, and runtime efficiency, while the ablation study examines its weight-mask design.
- Experimental setup: PRN is evaluated for alignment and reconstruction on AFLW2000-3D, AFLW-LFPA, and Florence using quantitative and qualitative comparisons.AFLW2000-3D supports both tasks, AFLW-LFPA evaluates 3D alignment, and Florence evaluates reconstruction against recent methods.
- 3D Face Alignment: The method slightly outperforms 3D-FAN on 2D sparse alignment and shows a larger advantage when depth is included.Evaluation uses NME with bounding-box normalization on 68 landmarks from AFLW2000-3D.
- 3D Face Alignment: The method remains robust across yaw angles and datasets, with stable alignment performance despite changes in pose.Table 1 reports NME for small, medium, and large yaw angles on AFLW2000-3D and mean NME on AFLW2000-3D and AFLW-LFPA.
- 3D Face Alignment: More than 27% improvement over the best competing methods is reported for dense alignment on both 2D and 3D coordinates.The comparison uses around 45K points from the largest common face region on AFLW2000-3D.
- Runtime and ablation: The model processes both reconstruction and dense alignment in one pass in 9.8ms, while using 160MB compared with VRN's 1.5GB.Runtime measurements use cropped face images and an NVIDIA GeForce GTX 1080 GPU for the reported PRN timing.
5 Conclusion
The end-to-end method jointly addresses 3D face alignment and reconstruction by directly regressing complete 3D structure with semantic meaning from a single image. Results across three test datasets show robustness, significant improvements, faster execution, and real-time suitability.
- The method simultaneously solves 3D face alignment and 3D face reconstruction.
- Learning the position map directly regresses complete 3D facial structure with semantic meaning from a single image.
- Quantitative and qualitative results demonstrate robustness to poses, illuminations, and occlusions.
- Experiments on three test datasets show significant improvements over other methods.
- The method runs faster than other methods and is suitable for real-time usage.