PaperlayerView exact revision on arXiv

Source-linked AI summary

Reconstruction-Based Disentanglement for Pose-invariant Face Recognition

Xi Peng, Xiang Yu, Kihyuk Sohn, Dimitris Metaxas, Manmohan Chandraker

arXiv:1702.03041v2cs.CV

TL;DR

Large pose changes remain difficult for face recognition when pose variation is limited in training data. The paper synthesizes pose views, learns identity and non-identity features, and disentangles them through reconstruction; it reports improvements across datasets, particularly for large pose variations.

  • Problem

    Face recognition remains challenged by large pose variations that are relatively underrepresented in training data.

    Source: §1 · p. 1

  • Method

    The method synthesizes pose views from near-frontal faces, jointly learns identity and non-identity features, and uses feature reconstruction to disentangle them.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · pp. 4–5

  • Results

    Across MultiPIE, 300WLP and CFP, self- and cross-reconstruction regularization yields significant improvements with 2% to 12% gaps, and cross-database evaluation verifies generalization.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The authors report pose-invariant recognition without extensive pose coverage in training data and strong performance for large pose variations, including up to 90°.

    Source: §1 · p. 2

    The assumption that deep networks implicitly suppress non-identity factors after extensive identity-labeled training does not always hold when non-identity variation is extensive.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Deep neural networks (DNNs) trained on large-scale datasets have recently achieved impressive improvements in face recognition. But a persistent challenge remains to develop methods capable of handling large pose variations that are relatively underrepresented in training data. This paper presents a method for learning a feature representation that is invariant to pose, without requiring extensive pose coverage in training data. We first propose to generate non-frontal views from a single frontal face, in order to increase the diversity of training data while preserving accurate facial details that are critical for identity discrimination. Our next contribution is to seek a rich embedding that encodes identity features, as well as non-identity ones such as pose and landmark locations. Finally, we propose a new feature reconstruction metric learning to explicitly disentangle identity and pose, by demanding alignment between the feature reconstructions through various combinations of identity and pose features, which is obtained from two images of the same subject. Experiments on both controlled and in-the-wild face datasets, such as MultiPIE, 300WLP and the profile view database CFP, show that our method consistently outperforms the state-of-the-art, especially on images with large head pose variations. Detail results and resource are referred to https://sites.google.com/site/xipengcshomepage/iccv2017

1. Introduction

Large pose variation can separate same-identity faces in learned feature spaces, while collecting and labeling broad pose coverage is costly or unavailable. The paper proposes pose-variant data generation, rich feature embedding, and reconstruction-based disentanglement, reporting strong results especially at large poses.

1. Introduction

  • Large pose changes can map images of the same identity to distant feature-space regions, while conventional datasets have long-tailed pose distributions and broad coverage is costly or unavailable.View-specific methods may also require pose information or multiple poses at test time.Source: §1 · pp. 1–2
  • The method generates varied-pose faces from near-frontal examples, jointly learns identity, pose, and landmark features, then uses feature reconstruction to disentangle identity from non-identity information.The reconstruction approach combines a reference face's identity and non-identity features for self-reconstruction, and combines its non-identity feature with a pose-variant sample's identity feature for cross-reconstruction.Source: §1 · p. 2
  • The framework aims to achieve pose-invariant recognition without extensive pose coverage in training data or additional labeling expense for generated pose variations.The authors describe this as a reconstruction-based feature-learning contribution and report strong performance on controlled and uncontrolled datasets.Source: §1 · p. 2
  • The authors report strong improvements over state-of-the-art methods on controlled and in-the-wild datasets, with increasingly significant gains as pose variation grows and strong performance up to 90°.Ablations examine pose-variant generation, rich feature embedding, and feature-reconstruction disentanglement.Source: §1 · p. 2

2. Related Work

Related work spans 3D face synthesis, deep recognition, pose-invariant methods, and disentangled representations. The paper uses 3D synthesis to enrich training variation and feature-level reconstruction targets to separate identity from other factors.

2. Related Work

  • Face-synthesis work includes 3D morphable models, frontalization, and single-image reconstruction, whereas this paper uses 3D face models to enrich training-data diversity.Prior reconstruction and frontalization approaches are also used for recognition or face normalization.Source: §2 · p. 2
  • Deep recognition systems have achieved impressive performance, but very large training sets can be expensive to collect and may be unavailable in applications such as surveillance.FaceNet is described as training on 200 million images from 8 million subjects.Source: §2 · pp. 2–3
  • Pose-invariant recognition includes CCA, multiview methods, frontalization, progressive autoencoders, part matching, and 3D reconstruction, with cited limitations including strict pose pairing, training-data dependence, or possible synthesis noise.The paper instead targets pose-invariant features while preserving discrimination through deep training.Source: §2 · p. 3
  • Prior disentanglement methods use orthogonal or task-specific representations, graphics codes, or image-level cross-reconstruction; this paper uses designed feature embeddings as reconstruction targets to avoid reconstructing irrelevant variation.The captioned overview describes pose-variant generation, rich feature embedding, and disentanglement by reconstruction as successive stages.Source: §2 · p. 3, Figure 2 · p. 3

3. Proposed Method

The method combines generated pose variation, a richly supervised identity/non-identity embedding, and feature reconstruction to learn pose-invariant identity features. Its reconstruction objective uses same-identity images at different viewpoints to disentangle those features.

3.1. Pose-variant Face Generation

  • Near-frontal faces are reconstructed with a 3DMM, then rendered at yaw angles from −90° to 90° to add pose variation without hallucinating occluded textures.The paper motivates enriching intra-subject variation as useful training data and contrasts this with frontalizing non-frontal faces, which can hallucinate missing textures.Source: §3.1 · p. 4

3.2. Rich Feature Embedding

  • Because identity-only supervision may leave pose correlated with features, the method branches a rich embedding into identity and non-identity features supervised by identity, pose, and landmark labels.The non-identity branch also separates pose and landmark cues; the paper describes these as multi-source supervision for the rich embedding.Source: §3.2 · p. 4

3.3. Disentanglement by Feature Reconstruction

  • For same-identity images at different viewpoints, self- and cross-reconstruction combine identity and non-identity features to reconstruct near-frontal and non-frontal embeddings.The paper samples a near-frontal reference (≤5° yaw) and a non-frontal peer (>5°), then uses their features in the reconstruction tasks.Source: §3.3 · p. 5
  • Minimizing both reconstruction losses while retaining an identity cross-entropy constraint rebalances the features toward a normalized identity representation across poses.During this stage, the recognition network θr is fixed while θi, θn, and the reconstruction mapping are finetuned, preserving the previously learned rich embedding.Source: §3.3 · p. 5
  • Feature-level reconstruction targets pose invariance without image-level frontalization artifacts or the extensive preprocessing associated with image-level normalization.Unlike augmentation alone, reconstruction loss supervises feature decoupling; the cross-entropy constraint helps preserve identity discrimination across other training variations.Source: §3.3 · p. 5

4. Implementation Details

The implementation generates pose-varied faces, learns a rich embedding with identity and non-identity branches, and then fine-tunes the branches through reconstruction. The authors detail the network configurations, training data, and optimization choices for these stages.

Pose-variant face generation

  • The pose-generation network predicts 3DMM parameters using stride-2 convolutions and 66-dimensional outputs, initialized with ImageNet-pretrained weights rather than trained from scratch.The architecture is similar to VGG16 [35]; the 66 dimensions encode identity, expression, and pose, replacing the 235-dimensional representation used in [49].Source: §4 · p. 5
  • New viewpoints are sampled every 5° from near-frontal faces through left and right profiles to cover the full pose range.Source: §4 · pp. 5–6

Rich feature embedding

  • The 512-dimensional rich embedding branches into 256-dimensional identity and 128-dimensional non-identity features, with pose and landmark supervision on the latter.Pose and landmark outputs have 7 and 136 dimensions, respectively; training uses CASIA-WebFace, 300WLP, and MultiPIE, with the network trained from scratch.Source: §4 · p. 6

Disentanglement by reconstruction

  • For reconstruction-based disentanglement, the authors freeze the rich-embedding parameters and fine-tune identity and non-identity features through a two-layer mapping from 384 to 512 dimensions.They use early stopping, cross-entropy to preserve identity discrimination, and 5-fold cross-validation to select hyperparameters; triplet loss is described as a plug-in alternative.Source: §4 · p. 6

5. Experiments

The experiments evaluate pose-variant recognition on MultiPIE, 300WLP, and CFP, alongside module ablations and cross-database tests. Across datasets, the reported advantages are strongest under large pose variations.

5.1. Evaluation on MultiPIE

  • On MultiPIE, accuracy is 3–4% better than the best reported at 45°–90°, while the general methods’ >30% drop near 90° is reduced by >20%.The method also does not require pose information at test time, unlike MvDN.Source: Table 1 · p. 6, §5.1 · pp. 6–7

5.2. Evaluation on 300WLP

  • On 300WLP, accuracy is consistently better than the two general recognition frameworks, especially as pose approaches 90°.The test uses 500 subjects, with two frontal images per subject as gallery and the remaining 7,014 images as probes; poses are grouped into six intervals.Source: Table 2 · p. 6, §5.2 · p. 7

5. Experiments

  • Across MultiPIE, 300WLP, and CFP, the method improves over state-of-the-art comparisons, with particularly strong results at large pose angles.The reported comparisons include pose-invariant baselines on MultiPIE and general recognition methods on 300WLP and CFP.Source: §5.1 · p. 6, §5 · p. 6, §5.2 · p. 7

5.4. Control Experiments

  • Multi-source, multi-task training outperforms single-source training followed by fine-tuning; reconstruction regularization is more effective than direct identity-feature distance, especially at large poses.N-pair loss also boosts performance, while improvements from reconstruction over the N-pair baseline are described as marginal compared with those over MSMT.Source: Table 3 · p. 7, §5.4 · p. 8

5.3. Evaluation on CFP

  • On CFP, the method matches Chen et al. on Frontal-Frontal accuracy and is 1.8% better on Frontal-Profile.It is also 0.8% better than DR-GAN on frontal cases and slightly better on extreme-pose evaluation.Source: Table 4 · p. 7, §5.3 · p. 7

5.5. Cross Database Evaluation

  • Cross-database tests show 0.8% and 2.8% improvements over VGGFace on MultiPIE, and 6.6% and 7.2% on 300WLP, for the reported MSMT and MSMT+SR models.The authors attribute expected cross-database accuracy drops to the distribution gap between MultiPIE and 300WLP.Source: Table 5 · p. 8, §5.5 · p. 8

6. Conclusion

The paper combines pose-diverse face synthesis, rich embeddings, and self- and cross-reconstruction regularization to disentangle identity and pose. It reports gains across three datasets and cross-database generalization, while identifying database gaps as future work.

  • The method combines pose-diverse data synthesis, rich feature embedding, and self- and cross-reconstruction regularization for face recognition.Source: §6 · p. 8
  • The proposed regularization disentangles identity and pose, with reported improvements of 2% to 12% on MultiPIE, 300WLP, and CFP.Source: §6 · p. 8
  • Cross-database evaluation supports generalization across databases; future work targets the systematic gap among databases and further generalization.Source: §6 · p. 8

1. Summary of The Supplementary

The supplementary file adds implementation details and experimental results to improve reproducibility and validate the approach.

  • The supplementary file provides additional implementation details to improve reproducibility.Source: §1 · p. 11
  • It also presents further experimental results to validate the approach in different aspects.Source: §1 · p. 11
  • These details and results were omitted from the main submission due to space limitations.Source: §1 · p. 11

2. Additional Implementation Details

Implementation details describe pose-variant face generation, a branched rich embedding, and feature reconstruction to disentangle identity from pose.

Pose-variant face generation

  • The face-generation network predicts 3DMM identity, expression, and pose parameters from one face image, using 30 identity coefficients that preserve 90% of variations.Non-frontal textures are generated from frontal faces to mitigate self-occlusion hallucination and support high-fidelity reconstruction.Source: §2 · p. 11

Rich feature embedding

  • A 512-dimensional rich embedding branches into 256-dimensional identity and 128-dimensional non-identity features, with landmark and pose prediction from the latter.Source: §2 · p. 11

Disentanglement by feature reconstruction

  • For same-identity pairs with different viewpoints, two 512-unit fully connected layers reconstruct a 512-dimensional rich embedding from concatenated identity and non-identity features.Source: §2 · p. 11
  • Self- and cross-reconstruction losses pull identity features together, while cross-entropy on the near-frontal identity feature maintains discriminative power.Source: §2 · p. 11

3. Additional Experimental Results

Additional experiments examine recognition across protocols and datasets, feature-space visualizations, testing samples, and failure cases. Reported comparisons favor the proposed method, while the failure analysis identifies dataset-specific challenges.

3. Additional Experimental Results

  • The supplement presents further experiments to validate the approach from different perspectives.Source: §3 · p. 11

3.1. P1 and P2 protocol on MultiPIE

  • In cross-database testing, the method beats MvDN by 0.7% under MultiPIE P2 and consistently outperforms MSMT across protocols; its standard deviation across trials is small.The cross-database evaluation trains on CASIA and 300WLP and tests on MultiPIE under P1 or P2.Source: §3.1 · p. 11, Table 7 · p. 12

3.2. Control Experiments with P2 on MultiPIE

  • Under P2, all 0° images form the gallery; the results support the effectiveness of feature-reconstruction regularization for pose-invariant, discriminative representations.P2 is expected to perform better than P1 because it uses more reference images.Source: §3.2 · pp. 11–12, Table 8 · p. 12

3.3. Recognition Accuracy on LFW

  • On LFW, the method outperforms VGG-Face especially for non-frontal pose ranges above 30°.Table 9 reports pose-wise accuracy as correct-pair count over total pairs in each range.Source: §3.3 · p. 12, Table 9 · p. 13

3.4. Feature Embedding of MultiPIE

  • For MultiPIE, the proposed feature space clusters same-subject samples across pose and reduces overlap between identities compared with VGGFace.The t-SNE visualization uses 10 randomly selected test subjects at 0°, 30°, 60°, and 90° yaw.Source: §3.4 · p. 12, Figure 5 · p. 13

3.5. Feature Embedding of 300WLP

  • For 300WLP, the proposed embeddings cluster each subject across extensive pose variations, unlike VGGFace features that entangle pose and identity.Source: §3.5 · p. 12, Figure 6 · p. 13

3.6. Probe and Gallery Examples

  • MultiPIE testing uses one frontal gallery image per subject and probes spanning poses and expressions; 300WLP uses two randomly selected near-frontal gallery images and probes covering all poses.Source: §3.6 · p. 12, Figure 7 · p. 14

3.7. Failure cases in MultiPIE and 300WLP

  • Typical MultiPIE failures are associated mainly with exaggerated expressions, while 300WLP failures mostly involve large pose and illumination changes; most failed pairs look visually similar.Source: §3.7 · p. 12, Figure 8 · p. 15
Loading 1702.03041v2…