Source-linked AI summary
Towards Large-Pose Face Frontalization in the Wild
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, Manmohan Chandraker
TL;DR
Large pose variation, especially profile views, remains difficult for face recognition and frontalization in the wild. FF-GAN combines a deep 3DMM-conditioned GAN with identity regularization and specialized losses, and experiments report consistent gains across recognition, landmark localization, and 3D reconstruction.
Problem
Large-pose face views remain challenging because frontalization from a single 2D image is ambiguous, self-occluded, and associated with severe recognition accuracy drops.
Method
FF-GAN combines a deep 3DMM reconstructor, generator, discriminator, and face-recognition engine to synthesize identity-preserving frontal faces from non-frontal inputs.
Results
FF-GAN consistently improves performance across face recognition, landmark localization, and 3D reconstruction on faces in the wild, including pose ranges up to 90°.
Takeaways & Limitations
The framework supports high-quality, identity-preserving frontalization for challenging large-pose and extreme-profile face images.
Abstract
from arXiv · showhide
Despite recent advances in face recognition using deep learning, severe accuracy drops are observed for large pose variations in unconstrained environments. Learning pose-invariant features is one solution, but needs expensively labeled large-scale data and carefully designed feature learning algorithms. In this work, we focus on frontalizing faces in the wild under various head poses, including extreme profile views. We propose a novel deep 3D Morphable Model (3DMM) conditioned Face Frontalization Generative Adversarial Network (GAN), termed as FF-GAN, to generate neutral head pose face images. Our framework differs from both traditional GANs and 3DMM based modeling. Incorporating 3DMM into the GAN structure provides shape and appearance priors for fast convergence with less training data, while also supporting end-to-end training. The 3DMM-conditioned GAN employs not only the discriminator and generator loss but also a new masked symmetry loss to retain visual quality under occlusions, besides an identity loss to recover high frequency information. Experiments on face recognition, landmark localization and 3D reconstruction consistently show the advantage of our frontalization method on faces in the wild datasets.
1. Introduction
FF-GAN addresses severe face-recognition degradation under large pose variation by frontalizing in-the-wild faces, including profiles up to 90°. It combines a deep 3DMM-conditioned GAN with identity preservation and reports improvements across multiple tasks and datasets.
- Approach: FF-GAN combines a generator, discriminator, deep 3DMM reconstructor, and face-recognition CNN to synthesize realistic frontal faces that preserve identity.
- Motivation: Large-pose and profile face views cause severe accuracy drops even for state-of-the-art CNN-based face-recognition representations.
- Approach: FF-GAN frontalizes a single non-frontal in-the-wild face image, including profile views up to 90°, while targeting identity preservation and image quality.
- Approach: The framework incorporates deep 3DMM coefficients into an end-to-end GAN to provide shape and appearance priors for challenging pose variation.The 3DMM-enhanced input supplies a useful prior, while combining coefficients with the image retains global pose accuracy and local information.
- Results: The method reports consistent improvements across face recognition, landmark localization, and 3D reconstruction on several in-the-wild datasets.On LFW, recognition accuracy using frontalized outputs exceeds previous state of the art; Multi-PIE shows larger gains beyond 45° and at 75°–90°.
2. Related Work
Prior face frontalization methods use 3D face models, optimization, or deep networks, while pose-invariant recognition methods learn invariant features. FF-GAN combines deep 3DMM estimation, adversarial synthesis, and identity regularization in a unified framework.
- Model-based frontalization: 3D Morphable Models represent facial shape and appearance as PCA spaces and can generate images under novel viewpoints after fitting.
- Deep and optimization methods: Earlier frontalization approaches include low-rank optimization, recurrent view synthesis, concatenated rotation networks, and image-level reconstruction losses.
- FF-GAN architecture: Figure 2 separates FF-GAN into reconstruction, generation, discrimination, and recognition modules for coefficient estimation, frontal synthesis, realism decisions, and identity classification.
- Comparison with related work: Unlike DR-GAN’s explicit pose code, FF-GAN learns shape and appearance codes through a differentiable deep 3DMM network for end-to-end GAN optimization.The paper contrasts this with DR-GAN’s encoder-based identity preservation, which it associates with loss of spatial and high-frequency information important for image generation.
- Identity-preserving methods: FF-GAN uses a face-recognition engine as an additional discriminator-like regularizer that guides synthesis toward identity preservation.
3. Proposed Approach
FF-GAN combines a deep 3DMM reconstructor, fused encoder-decoder generator, discriminator, and recognition engine to frontalize extreme-pose faces while preserving realism and identity. Its losses jointly address reconstruction, smoothness, symmetry under visible regions, adversarial frontalization, and identity preservation.
- 3.1. Reconstruction Module: 3DMM priors reduce training complexity and improve empirical performance for extreme-pose frontalization with limited data.The prior provides shape and appearance regularization beyond the training data.
- 3.1. Reconstruction Module: The reconstructor R estimates 3DMM coefficients from an input image, including pose, identity, expression, and texture parameters.The coefficients define a unique 3D face, while a CNN based on CASIA-Net performs regression.
- 3.2. Generation Module: The generator fuses image and 3DMM features through an encoder-decoder network to synthesize a frontal face that preserves global pose accuracy and local detail.The model uses the input image and coefficients p to recover low- and high-frequency components.
- 3.2. Generation Module: L1 reconstruction, total variation, and masked symmetry losses preserve detail, reduce block artifacts, and compare flipped outputs only over visible face regions.The binary mask excludes invisible regions and background from the symmetry constraint.
- 3.3. Discrimination Module: The discriminator distinguishes generated images from real frontal faces, while adversarial training guides pose correction early and subtle frontal-face realism later.The discriminator supplies supervision beyond reconstruction and smoothness losses.
- 3.4. Identity-Preserving Module: The recognition engine regularizes generated images to preserve the input identity, addressing self-occlusion and the pixelwise losses’ limited discriminative power.When identity labels are unavailable, extracted identity features from generated and input images are matched.
4. Implementation Details
Implementation uses four jointly trained components, with the reconstructor and recognition engine prepared before end-to-end optimization. The generator and discriminator are trained from scratch together during adversarial training.
- 4. Implementation Details: FF-GAN comprises a deep 3DMM reconstructor, fused encoder-decoder generator, discriminator, and recognition engine for identity regularization.Detailed network structures are introduced in the appendix.
- 4. Implementation Details: The reconstructor and recognition engine are prepared before end-to-end joint training because the generator requires correct 3DMM coefficients.The overall network is difficult to initialize from scratch.
- 4. Implementation Details: The generator and discriminator are trained from scratch simultaneously so their adversarial objectives remain matched during training.The authors state that pre-trained generator and discriminator components do not contribute much to adversarial training.
5. Experiments
FF-GAN is evaluated across reconstruction, landmark localization, frontalization, and face recognition on several challenging face datasets. The results show realistic identity-preserving frontalization, competitive landmark accuracy, and consistent recognition gains, while ablations highlight the importance of the framework’s modules and losses.
- 3D Reconstruction and Landmark Localization: 6.01 normalized mean square error is achieved for landmark localization, compared with 5.42 for 3DDFA and 6.12 for SDM.The method also generates realistic frontal faces from extreme profile inputs.
- 3D Reconstruction and Landmark Localization: The reconstruction module preserves shape and identity information, while fusing 3DMM coefficients with the input restores local high-frequency appearance.Qualitative results report estimations that can outperform supplied 3DDFA ground truth in preserving input identity and appearance.
- Face Recognition: On IJB-A, verification improves by 6.46% at FAR 0.01 and 11.13% at FAR 0.001, while identification improves by 4.95% at Rank-1 and 1.66% at Rank-5.The fused metric combines original-image and generated-image pair distances, weighting generated pairs by discriminator confidence.
- Face Recognition: On Multi-PIE, FF-GAN is competitive near-frontal poses and shows significant advantages over prior methods at poses of 30° or larger, including 90° profiles.Fused results outperform original images especially for large-pose faces, although synthetic images alone can perform worse than originals.
- Qualitative Results: Qualitative results remain realistic and identity-preserved under medium-to-large pose, race, age, expression, and lighting variation.Visual results are reported on unseen images from Multi-PIE, AFLW, and IJB-A.
- Ablation Study on Multi-PIE: The full FF-GAN outperforms incomplete variants; removing recognition, discrimination, reconstruction, or identity-loss components degrades identity, sharpness, geometry, or lighting preservation.Without the recognizer, identity is harder to preserve at large poses; without the discriminator, outputs become blurry; without the reconstructor, artifacts appear.
6. Conclusions
FF-GAN frontalizes faces across all pose ranges, including extreme profile views, using 3DMM priors, identity regularization, and new losses. Extensive experiments suggest improved recognition and applicability to 3D face reconstruction.
- 6. Conclusions: FF-GAN frontalizes faces across all pose ranges, including extreme profile views up to 90° in challenging in-the-wild databases.The framework uses a 3DMM reconstruction module as a shape and appearance prior to guide face rotation.
- 6. Conclusions: The recognition engine regularizes generated images to preserve identity, while new losses and training procedures target high-quality frontalized images.
- 6. Conclusions: Extensive experiments suggest that FF-GAN can boost face recognition performance and support 3D face reconstruction.