PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis

Rui Huang, Shu Zhang, Tianyu Li, Ran He

arXiv:1704.04086v2cs.CV

TL;DR

Large-pose face recognition remains difficult because frontal-view recovery from profile images is ill-posed and existing invariant-feature approaches have limited effectiveness. TP-GAN addresses this with global and local synthesis pathways plus adversarial, symmetry, and identity-preserving losses. It produces photorealistic, identity-preserving frontal faces and outperforms state-of-the-art recognition results under large poses.

  • Problem

    Large-pose face recognition remains challenging, while recovering a frontal view from an incompletely observed profile is ill-posed and has multiple possible solutions.

  • Method

    TP-GAN combines global-structure and local-texture pathways with adversarial, symmetry, and identity-preserving losses for frontal-view synthesis.

  • Results

    TP-GAN consistently achieves the best recognition performance across tested angles and improves performance by about 20% over c-CNN Forest on large-pose cases.

  • Takeaways & Limitations

    The synthesized identity-preserving images can be used directly for downstream face analysis, including face recognition and attribution estimation.

  • Takeaways & Limitations

    Feature-based approaches remain limited for large poses by a tradeoff between invariance and discriminability.

Abstract

from arXiv · show

Photorealistic frontal view synthesis from a single face image has a wide range of applications in the field of face recognition. Although data-driven deep learning methods have been proposed to address this problem by seeking solutions from ample face data, this problem is still challenging because it is intrinsically ill-posed. This paper proposes a Two-Pathway Generative Adversarial Network (TP-GAN) for photorealistic frontal view synthesis by simultaneously perceiving global structures and local details. Four landmark located patch networks are proposed to attend to local textures in addition to the commonly used global encoder-decoder network. Except for the novel architecture, we make this ill-posed problem well constrained by introducing a combination of adversarial loss, symmetry loss and identity preserving loss. The combined loss function leverages both frontal face distribution and pre-trained discriminative deep face models to guide an identity preserving inference of frontal views from profiles. Different from previous deep learning methods that mainly rely on intermediate features for recognition, our method directly leverages the synthesized identity preserving image for downstream tasks like face recognition and attribution estimation. Experimental results demonstrate that our method not only presents compelling perceptual results but also outperforms state-of-the-art results on large pose face recognition.

1. Introduction

Frontal view synthesis addresses pose variation in face recognition but remains difficult because profile images omit information and unconstrained recovery is ill-posed. TP-GAN combines global and local perception with adversarial, symmetry, and identity-preserving constraints to generate frontal faces for recognition.

  • Pose variation remains a bottleneck for real-world face recognition despite progress on benchmark datasets.
  • Large-pose recognition methods either learn pose-invariant features or synthesize frontal faces before recognition.
  • Feature-based approaches face a tradeoff between invariance and discriminability, limiting their effectiveness for large poses.
  • Frontal recovery from an incompletely observed profile is ill-posed, and pairwise supervision alone can produce blurry results.
  • TP-GAN uses global structure and local texture pathways alongside adversarial, symmetry, and identity-preserving losses for photorealistic, identity-preserving synthesis.
  • The synthesized frontal faces improve the method’s potential for downstream face-analysis tasks, including recognition.

2. Related Work

Frontal view synthesis, or face normalization, has evolved from geometric and statistical methods to CNN-based representation learning and GAN-based image generation. Prior methods often use intermediate features for recognition rather than directly exploiting synthesized faces.

  • Frontal view synthesis, also called face normalization, is challenging because the problem is ill-posed.
  • Traditional approaches use 2D or 3D texture warping, statistical modeling, or constrained low-rank minimization for normalization.
  • Recent CNN methods jointly learn representations and view synthesis, including identity-preserving rotation and disentangled identity-pose modeling.
  • Most previous synthesis methods rely on intermediate features for face recognition because directly using synthesized images is difficult.
  • GANs estimate target image distributions and encourage photorealistic outputs with plausible high-frequency details.

3. Approach

TP-GAN learns frontal-view synthesis with a two-pathway CNN and a weighted synthesis objective for paired profile-frontal training data. The objective combines synthesis losses with classification supervision to constrain the generated image.

  • The task is to recover a photorealistic, identity-preserving frontal image from a differently posed profile image using paired identities during training.
  • TP-GAN models synthesis with a two-pathway CNN whose global and local pathways separately process face structure and local texture.
  • The global pathway uses an encoder-decoder design, with its bottleneck also associated with classification through cross-entropy loss.
  • Training minimizes a synthesis loss together with cross-entropy loss over N paired training examples.
  • The synthesis loss is a weighted sum of individual losses that jointly constrain generated images to the desired manifold.

3.1. Network Architecture

TP-GAN uses separate global and local pathways to synthesize frontal faces, then fuses their representations while adversarial training encourages outputs to match the frontal-face distribution.

  • Two-Pathway Generator: TP-GAN combines one global network for facial structure with four landmark-centered patch networks for local textures.The local patches correspond to the left eye, right eye, nose, and mouth landmarks.
  • Two-Pathway Generator: The two pathways reflect a human-like process that first infers global structure and then fills in local facial details.Their feature maps are fused before generating the final synthesis.
  • Local Pathways: Each landmark patch pathway uses an encoder-decoder without a fully connected bottleneck and learns filters for rotating its centered patch to frontal view.After rotation, the corresponding facial landmark remains centered.
  • Feature Fusion: Local feature tensors are placed at template landmark locations, fused into one tensor, and combined with the global feature tensor using max-out fusion.This fusion strategy is introduced to reduce stitching artifacts.
  • Adversarial Training: An adversarial discriminator distinguishes real frontal faces from generated faces and pushes generator outputs toward the frontal-face manifold.The discriminator produces a 2 × 2 probability map so different face regions can receive focused judgments.

3.2. Synthesis Loss Function

The synthesis objective combines pixel consistency, symmetry, adversarial, identity-preserving, and total-variation terms to constrain frontal-view generation. These losses jointly promote content consistency, realistic facial structure, identity retention, and reduced artifacts.

  • Overall Objective: The synthesis loss is a weighted sum of four individual loss functions, with total variation additionally reducing spike artifacts.The paper describes the combined objective as jointly constraining synthesized images.
  • Pixel-wise Loss: Multi-location pixel-wise L1 loss enforces image-content consistency at global, landmark-patch, fused, and multi-scale outputs.The loss supports deep supervision across the network.
  • Symmetry Loss: Symmetry loss compares synthesized facial regions in pixel and Laplacian spaces to alleviate self-occlusion during large-pose synthesis.The Laplacian-space comparison is described as robust to illumination changes, and the loss is applied to the occluded side.
  • Adversarial Loss: Adversarial loss pushes synthesized images toward the manifold of frontal faces and helps prevent blur while producing visually pleasing results.The discriminator distinguishes real frontal images from synthesized frontal images.
  • Identity-preserving Loss: Identity-preserving loss uses activations from the last two Light CNN layers to minimize distance between predicted and ground-truth deep features.Because Light CNN was pretrained to classify tens of thousands of identities, its features capture prominent identity-discriminative face structure.
  • Loss Interaction: Combining identity-preserving and adversarial losses retains identity while keeping optimization within the manifold of natural face images.Using identity-preserving loss alone can produce artifacts, whereas the combination produces photorealistic images.

4. Experiments

TP-GAN is evaluated for photorealistic, identity-preserving frontalization across extreme poses, challenging attributes, cross-dataset generalization, recognition, gender classification, and architecture components. Results show strong synthesis quality, improved downstream recognition, and contributions from both the two-pathway design and combined losses.

  • Face Synthesis: TP-GAN preserves observed attributes such as eyeglasses and hairstyle while hallucinating occluded ears, cheeks, and forehead consistently with identity.The model produces photorealistic synthesis alongside identity preservation in challenging situations.
  • Face Synthesis: TP-GAN’s mean synthesized faces preserve more texture detail and exhibit less blur, indicating stable geometry across multiple poses without 3D geometry knowledge.The geometry is inferred through data-driven learning rather than explicit 3D modeling.
  • Face Synthesis: TP-GAN generalizes to LFW faces despite being trained solely on Multi-PIE, producing finer details and better global shapes than state-of-the-art methods.The resulting color tone remains similar to Multi-PIE images.
  • Identity Preserving Property: TP-GAN achieves the best recognition performance across all angles in Setting 1, with about a 20% boost over c-CNN Forest on large-pose cases.Recognition uses Light-CNN features and cosine distance with Rank-1 accuracy.
  • Identity Preserving Property: Synthesized frontal images improve gender classification over original profile images, while other compared models perform worse than the baseline.The authors attribute the comparison setup to normalized views and warn that pixel-space manipulation can lose prominent facial features.
  • Identity Preserving Property: TP-GAN-generated feature representations separate identities more clearly than original profile features, especially for large-pose faces.The comparison visualizes 256-dimensional Light-CNN features with t-SNE; 90° profile images are not separable before synthesis.
  • Algorithmic Analysis: The two-pathway architecture and identity-preserving loss contribute most to recognition gains, while adversarial and symmetry losses provide additional improvements.Removing identity preservation or the local pathway seriously changes appearance; removing adversarial loss causes blur, and removing symmetry loss can create unnatural asymmetry.

5. Conclusion

The paper presents TP-GAN, a global and local perception GAN for single-image frontal view synthesis, combining dual pathways with adversarial, symmetry, and identity-preserving losses. The method produces visually appealing identity-preserving images and outperforms state-of-the-art results on large-pose face recognition.

  • TP-GAN models global structure and local texture through separate pathways for frontal view synthesis from a single image.The global pathway models out-of-plane rotation, while the local pathway models nonlinear local-texture transformation.
  • Adversarial, symmetry, and identity-preserving losses constrain the ill-posed synthesis problem using frontal-face distribution and facial symmetry priors.The identity-preserving loss also targets recognition usability, while symmetry loss addresses self-occlusion in large-pose cases.
  • The synthesized images are visually appealing and readily applicable to accurate face recognition.
  • TP-GAN outperforms state-of-the-art results on large-pose face recognition.

6. Supplementary Material

The supplementary material details TP-GAN’s architecture, feature fusion, semantic-region activations, illumination robustness, and additional implementation and recognition results. These analyses describe how global and local features are integrated and show consistent synthesis across challenging illumination conditions.

  • Robustness: Under extreme illumination, synthesized exposure is consistent with or moderately better than the input or ground-truth frontal face, while skin tone and details remain consistent.The method also automatically adjusts exposure and white balance.
  • Architecture: The decoder upsamples and reconstructs features from the identity vector and noise, with extra inputs processed through residual blocks before concatenation.
  • Architecture: The local pathway uses separate down-sampling and up-sampling processes for landmark-centered eye, nose, and mouth patches.Patch sizes differ by facial region, with eye patches set to 40×40 and nose and mouth patches using region-specific dimensions.
  • Architecture: The global encoder and decoder structures are documented in Tables 5 and 6, while Table 7 documents the local pathway.
  • Implementation: The model uses weight-sharing convolutions to reduce parameters and avoid expensive per-pixel computation, while its early supervised decoder enables end-to-end high-resolution generation.
  • Feature analysis: Global up-sampled features Cg and fused local features Cl are concatenated and integrated by subsequent convolutional layers before final output.
  • Feature analysis: Certain skip layers automatically activate for non-face and hair regions, potentially simplifying frontal-face composition without supervision.
  • Recognition: With external face alignment under extreme poses, TP-GAN achieves Rank-1 recognition rates of 87.63(±60°), 76.69(±75°), and 62.43(±90°).
Loading 1704.04086v2…