Source-linked AI summary
Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression
Aaron S. Jackson, Adrian Bulat, Vasileios Argyriou, Georgios Tzimiropoulos
TL;DR
Single-image 3D face reconstruction remains difficult because existing methods rely on complex correspondence and optimization pipelines. The paper instead trains a CNN to directly regress a volumetric representation of full facial geometry from one image, and reports strong performance across varied poses, expressions, and occlusions, with large improvements over 3DMM-fitting methods on three datasets.
Problem
Single-image 3D face reconstruction remains far from solved, while existing approaches require complex pipelines and difficult optimization.
Method
A CNN directly regresses a spatially aligned volumetric representation of full 3D facial geometry from a single 2D image, bypassing 3DMM fitting.
Results
The proposed networks perform well across facial poses, expressions, and occlusions, with large performance improvements over 3DMM-fitting methods on three datasets.
Takeaways & Limitations
Direct volumetric CNN regression can reconstruct whole facial geometry from unconstrained single images, including non-visible facial parts.
Takeaways & Limitations
Compared with shape-from-shading methods, the approach cannot capture fine facial details; the authors primarily associate this limitation with the dataset.
Abstract
from arXiv · showhide
3D face reconstruction is a fundamental Computer Vision problem of extraordinary difficulty. Current systems often assume the availability of multiple facial images (sometimes from the same subject) as input, and must address a number of methodological challenges such as establishing dense correspondences across large facial poses, expressions, and non-uniform illumination. In general these methods require complex and inefficient pipelines for model building and fitting. In this work, we propose to address many of these limitations by training a Convolutional Neural Network (CNN) on an appropriate dataset consisting of 2D images and 3D facial models or scans. Our CNN works with just a single 2D facial image, does not require accurate alignment nor establishes dense correspondence between images, works for arbitrary facial poses and expressions, and can be used to reconstruct the whole 3D facial geometry (including the non-visible parts of the face) bypassing the construction (during training) and fitting (during testing) of a 3D Morphable Model. We achieve this via a simple CNN architecture that performs direct regression of a volumetric representation of the 3D facial geometry from a single 2D image. We also demonstrate how the related task of facial landmark localization can be incorporated into the proposed framework and help improve reconstruction quality, especially for the cases of large poses and facial expressions. Testing code will be made available online, along with pre-trained models http://aaronsplace.co.uk/papers/jackson2017recon
1 The University of Nottingham, UK 2 Kingston University, UK
Figure 1 presents results from the VRN-Guided method across a full range of facial poses, including large expressions.
- VRN-Guided produces reconstruction results across varied facial poses.
- The examples include large facial expressions.
- The figure illustrates the method’s coverage across pose and expression variation.
1. Introduction
The paper addresses single-image 3D face reconstruction by replacing complex correspondence and optimization pipelines with direct CNN regression from pixels to full facial geometry.
- Single-image 3D face reconstruction remains far from solved, despite its importance in Vision and Graphics research.
- Existing approaches often require complex pipelines and difficult non-convex optimization for model building and fitting.
- The CNN uses direct regression of a volumetric facial-geometry representation and bypasses 3DMM construction and fitting.
- The proposed method learns an end-to-end mapping from a single image to full 3D facial structure, including non-visible parts.
- Landmark localization is incorporated into the framework to improve reconstruction quality, especially under large poses and expressions.
2. Closely related work
The paper positions its method against optimization-heavy 3DMM, shape-from-shading, and CNN-based approaches, emphasizing direct volumetric prediction from unconstrained single images.
- Related single-image methods commonly use 3DMM fitting, while another approach reconstructs a neutral frontal face through joint reconstruction and alignment.
- The proposed method directly predicts a 3D volume rather than estimating 3DMM parameters or fitting a morphable model.
- Its voxel-level spatial prediction differs from CNNs that holistically regress 3DMM parameters.
- The method is designed for unconstrained web images spanning arbitrary poses, expressions, and occlusions.
- Unlike shape-from-shading methods, it cannot capture equally fine facial details, although the authors attribute this mainly to the training dataset.
3. Method
The method reformulates single-image 3D face reconstruction as direct CNN regression of an image-aligned volumetric facial representation. It uses stacked hourglass architectures, with optional landmark guidance or multitask learning, to recover a 3D mesh from predicted volumes.
- Data and representation: Each 3D scan is voxelised into a binary volume aligned with its 2D image, reducing vertex regression to 3D binary-volume segmentation.Voxels inside the facial scan receive 1 and background voxels receive 0.
- Data and representation: A 192 × 192 × 200 output volume provides the facial representation, and voxelisation at this resolution introduces negligible error relative to errors of a few millimetres.The predicted volume is converted into an outer facial mesh using its iso-surface; ICP can establish correspondence with a fixed mesh if needed.
- Volumetric regression networks: The VRN uses two stacked hourglass modules with skip connections and residual learning to produce spatially aligned voxel-level predictions.The encoder-decoder structure re-establishes spatial correspondence, while the second hourglass refines the first output.
- Prediction and training: At inference, soft sigmoid voxel predictions are preferred over hard binary predictions because they produce smoother facial results.The network is trained with sigmoid cross-entropy, and the soft volume is used for subsequent mesh extraction.
- Volumetric regression networks: VRN-Multitask adds a landmark-regression branch and a volumetric-regression branch after shared features, aiming to learn features suited to both tasks.The landmark branch predicts 68 iBUG landmarks as separate 2D Gaussian channels, while the other branch predicts the 3D face volume.
- Volumetric regression networks: VRN-Guided stacks an RGB image with 68 landmark heatmap channels, using ground-truth landmarks during training and a landmark-localisation network during testing.Each landmark channel contains a Gaussian centred on one facial landmark.
4. Results
Across three databases, the proposed Volumetric Regression Networks were evaluated against EOS and 3DDFA using NME, with results reported for unconstrained and rendered facial images. VRNs largely outperformed both state-of-the-art baselines, while landmark guidance gave the best performance at added computational cost.
- Volumetric Regression Networks largely outperform 3DDFA and EOS on AFLW2000-3D, BU-4DFE, and Florence.The comparisons use Normalised Mean Error, for which lower values indicate better reconstruction accuracy.
- All VRNs perform well across facial poses, expressions, and occlusions, with no significant performance discrepancies across datasets.AFLW2000-3D appears slightly more difficult than the other evaluated datasets.
- VRN-Guided is the best-performing VRN, but it adds computational complexity through a stacked hourglass landmark-localization network.
- VRN-Multitask does not consistently outperform plain VRN and performs worse on BU-4DFE, failing to justify its increased network complexity.
- Reconstruction accuracy was measured with NME, the average per-vertex Euclidean error normalized by the outer 3D interocular distance.The error is computed over the face region, covering approximately 19,000 vertices.
5. Importance of spatial alignment
The study compares spatially aligned reconstruction with a frontalized variant to assess whether alignment matters. The frontalized network produces reasonable faces but loses expression and identity-related shape variation, supporting the importance of spatial alignment.
- The proposed approach differs from prior fixed-orientation volume regression by taking a single image and regressing a spatially aligned volume.
- The frontalized VRN can produce a reasonable face, but captures only diminished expression and nearly identical shapes across faces.This behavior is shown visually in Fig. 9 and evaluated numerically in Fig. 7.
- The authors use the comparison to argue that spatial alignment is paramount for this form of 3D reconstruction.
6. Ablation studies
Ablation studies examine pose, expression, and landmark-guidance settings using VRN-Guided. Performance worsens with larger pose, varies little across expressions, and changes negligibly when guidance Gaussians are enlarged from σ = 1 to σ = 2.
- Effect of pose: As yaw increases on Florence renderings, reconstruction performance decreases because less of the face is visible.The authors nevertheless describe the error as very low for these poses.
- Effect of expression: Performance variation across BU-4DFE facial expressions is quite minor, including extreme acted expressions generally absent from training.
- Effect of Gaussian size for guidance: Increasing guidance Gaussian size from σ = 1 to σ = 2 causes only a negligible drop in 3D reconstruction performance.The result suggests guidance remains useful when the Gaussians have a sensible size.
7. Conclusions
The paper presents direct volumetric CNN regression for single-image 3D face reconstruction, reporting strong performance across poses, expressions, and occlusions. Compared with recent 3DMM-fitting methods, the networks show large performance improvements on three datasets.
- Direct volumetric CNN regression reconstructs 3D facial geometry from a single 2D image.
- The evaluated networks perform well across the full spectrum of facial poses and handle expressions and occlusions.
- Large performance improvements over recent 3DMM-fitting methods are reported on three different datasets.
- Future work includes improving reconstruction detail and establishing fixed correspondence from the mesh isosurface.