Disentangling Features in 3D Face Shapes for Joint Face Reconstruction and Recognition

1. Introduction

• Existing reconstruction methods rarely explicitly model identity-sensitive and identity-irrelevant features, while 3DMM fitting is constrained by its predetermined shape bases.The paper connects this limitation to poor capture of features unique to individuals.Source: §1 · pp. 1–2
• Reconstruction and recognition have different feature needs: reconstruction retains facial detail, while recognition favors identity-sensitive features.The paper proposes exploiting these objectives together to disentangle identity-relevant from identity-irrelevant shape information.Source: §1 · p. 2
• The method represents identity and residual shape components with separate latent variables and jointly trains an encoder-decoder for reconstruction and identification.Training data use 2D images with identity labels and corresponding 3D shapes obtained through adapted multi-image 3DMM fitting.Source: §1 · p. 2
• The authors report state-of-the-art 3D face reconstruction accuracy and improved face recognition accuracy from jointly learning discriminative shape features and reconstruction.They also report better capture of identity-sensitive and identity-irrelevant features than the limited 3DMM shape space.Source: §1 · p. 2

3D Face Reconstruction for Recognition

• Blanz and Vetter used fitted 3DMM parameters for recognition, while Tran et al. [34] pooled parameters across a subject’s images and used a CNN to regress them.Tran et al. experimentally reported improved discriminative power for the resulting parameters.Source: §2 · p. 2
• Liu et al. matched pose- and expression-normalized reconstructed shapes, while other researchers used reconstructed shapes to assist pose-robust feature extraction.Source: §2 · p. 2
• Existing 3D reconstruction methods generally do not explicitly consider recognition performance when learning to reconstruct faces.The reviewed methods in and [34] use identity-related strategies but learn the 2D-to-3D mapping without identity-label supervision.Source: §2 · p. 2

CNN-based 3D Face Reconstruction

• CNN-based reconstruction includes 3DMM-parameter and volumetric approaches; Jackson et al. [16] used volumetric coordinates, resulting in low-resolution shapes described as unfavorable for recognition.For Richardson et al. [28], the recognition benefit of visually plausible facial detail remained unclear.Source: §2 · p. 3

3.1. A Composite 3D Face Shape Model

• Each 3D face shape is modeled as a mean shape plus identity-sensitive and residual components, with the residual covering effects such as expression and temporary detail.Separate latent representations describe the two components, which are mapped to their corresponding shapes.Source: §3.1 · p. 3

3.2. An Encoder-Decoder Network

• The encoder extracts identity and residual representations from a 2D face image; the identity representation receives identification supervision while reconstruction loss guides the residual component.This jointly trains a 3D shape reconstructor and discriminative shape features.Source: §3.2 · pp. 3–4, Figure 3 · p. 4

3.3.1 Network Structure

• The SphereFace-based encoder uses parallel fully connected layers to produce 199-dimensional identity and 29-dimensional residual representations; separate decoder layers recover their shape components.The decoder output represents a 3D face point cloud with 29,495 vertices.Source: §3.3.1 · p. 4, Figure 4 · p. 4

3.3.2 Training Data

• Because a large-scale dataset with multiple images and corresponding 3D shapes per subject is unavailable, training data are constructed by fitting a 3DMM to multiple CASIA-WebFace images per subject.The fitting shares identity parameters across a subject’s images and estimates image-specific residual shape components.Source: §3.3.2 · pp. 4–5
• Multi-image 3DMM fitting minimizes landmark reprojection differences under a shared identity-parameter constraint, estimating projection and shape parameters in alternating steps.The optimization typically converges within seven iterations.Source: §3.3.2 · p. 5

3.3.3 Training Process

• Training proceeds through latent-representation pretraining, separate decoder training, and end-to-end joint training, which uses identification and reconstruction losses.Joint training begins with reconstruction-loss weight λR = 0.5, then changes it to 1.0 after the loss saturates; the joint phase takes about another 20 epochs.Source: §3.3.3 · p. 5

4.1. 3D Shape Reconstruction Accuracy

• The method achieves the best MICC reconstruction accuracy; on BU3DFE, it has lower RMSE across yaw angles and performs better than baselines across expressions.The method's RMSE does not increase substantially as yaw grows, and the experiments also compare reconstruction of expression-matched and identity shapes.Source: §4.1 · p. 6, Table 2 · p. 6, Table 1 · p. 6
• The MICC and BU3DFE examples illustrate reconstruction of expression-matched shapes and separate identity shapes, while the text reports disentangling identity and non-identity components.Source: §4.1 · pp. 6–7, Figure 5 · p. 6, Figure 8 · p. 7

4.2. Face Recognition Accuracy

• On LFW, TAR@1% FAR rises from 58.20% to 74.54% over the latest 3DMM-based method; on YTF, fusion with SphereFace raises accuracy from 94.78% to 95.18%.Source: §4.2 · pp. 7–8, Table 3 · p. 7

4.3. Computational Efficiency

• The method processes each image in 4.79 ms, reported as an order of magnitude faster than the baselines.The reported baseline runtimes exclude landmark detection for VRN, 3DDFA, and 3DMM-CNN.Source: §4.3 · p. 8, Table 4 · p. 8

4.4. Analysis and Discussion

• Unlike pre-trained identity bases that approximate conventional 3DMM bases, jointly learned identity bases all describe rich shape variations.Some jointly learned basis shapes do not resemble regular faces; the authors attribute this to the joint reconstruction and identification loss.Source: §4.4 · p. 8
• Jointly learned residual bases show more diverse deviation patterns than the symmetrical pre-trained residual bases, extending beyond expression changes.The authors interpret this as evidence that their method disentangles 3D face-shape features.Source: §4.4 · p. 8

• The method jointly learns discriminative shape features from a 2D face image while reconstructing its dense 3D face shape.Source: §5 · p. 8
• Training data are constructed with a multi-image 3DMM fitting method, and the network is trained with a joint reconstruction and identification loss.Source: §5 · p. 8
• The experiments show disentangled identity and non-identity features alongside state-of-the-art reconstruction accuracy and improved face-recognition accuracy.Source: §5 · p. 8

Supplementary Material

• The supplementary material includes IJB-A recognition results, a CNN-versus-3DMM phase-by-phase evaluation, and qualitative reconstruction results.The IJB-A evaluation covers verification and identification on a database with full pose variation.Source: Supplementary Material · p. 9, Recognition Results on IJB-A · p. 9

Recognition Results on IJB-A

• Shape-only features achieve the best reported IJB-A TAR-10%, TAR-1%, rank-1, and rank-5 recognition rates, while score-level fusion with DR-GAN further improves accuracy.The reported results are consistent with LFW and YTF findings and indicate that shape features complement texture features.Source: Recognition Results on IJB-A · p. 9
• Fusion of the proposed method with DR-GAN correctly recognizes example genuine and imposter pairs that DR-GAN misrecognizes.The examples include off-angle faces with large rotations, where complementary shape features support recognition.Source: Recognition Results on IJB-A · p. 9, Figure 10 · p. 9
• Adding identification loss in Phase III further reduces reconstruction errors and significantly improves recognition accuracy over Phase II.The authors interpret this as evidence of limited discrimination in 3DMM representations and the importance of CNN-based joint learning.Source: Recognition Results on IJB-A · p. 9

Qualitative Results

• Qualitative examples show identity shapes separating identity-sensitive from identity-irrelevant features while reconstructed faces capture shape deformation around the mouth.Blurry and very low-resolution images or video frames are identified as the main reasons for implausible reconstructions.Source: Qualitative Results · p. 9, Figure 12 · p. 9