PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning

Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, Xin Tong

arXiv:2004.11660v2cs.CV

TL;DR

DiscoFaceGAN addresses the limited evidence for de novo face synthesis with semantically disentangled latent factors. It combines 3D priors and imitative learning with contrastive comparisons to constrain domain-gap-induced freedom. The resulting model supports highly disentangled, precisely controllable generation and real-image embedding, with scope bounded by the paper's stated assumptions and future-work framing.

  • Problem

    De novo face synthesis with multiple disentangled latent spaces remains insufficiently investigated, despite its value for constrained generation and real-image embedding.

  • Method

    DiscoFaceGAN maps latent variables to 3D face-model coefficients, uses imitative losses with analytic rendering, and adds contrastive losses comparing generated image pairs.

  • Results

    The model produces highly disentangled latent variables with accurate control of face generation and can embed real images into the disentangled latent space.

  • Takeaways & Limitations

    The imitative-contrastive scheme provides a way to connect physical face properties with controllable deep image synthesis and latent-space analysis.

Abstract

from arXiv · show

We propose DiscoFaceGAN, an approach for face image generation of virtual people with disentangled, precisely-controllable latent representations for identity of non-existing people, expression, pose, and illumination. We embed 3D priors into adversarial learning and train the network to imitate the image formation of an analytic 3D face deformation and rendering process. To deal with the generation freedom induced by the domain gap between real and rendered faces, we further introduce contrastive learning to promote disentanglement by comparing pairs of generated images. Experiments show that through our imitative-contrastive learning, the factor variations are very well disentangled and the properties of a generated face can be precisely controlled. We also analyze the learned latent space and present several meaningful properties supporting factor disentanglement. Our method can also be used to embed real images into the disentangled latent space. We hope our method could provide new understandings of the relationship between physical properties and deep image synthesis.

1. Introduction

DiscoFaceGAN addresses the limited investigation of de novo face synthesis with disentangled latent spaces by combining 3D priors, imitative learning, and contrastive learning. It produces controllable representations for identity, expression, pose, illumination, and noise, while also supporting real-image embedding.

  • Motivation: Disentangled latent representations are desirable for constrained face generation and embedding real images, but unsupervised approaches do not guarantee semantically meaningful factors.The paper targets independent representations for multiple facial properties rather than generation quality alone.
  • Approach: The method maps latent variables to parametric face-model coefficients and incorporates 3D Morphable Face Model priors and analytic rendering into adversarial learning.Imitative losses train the generator to follow explainable rendered face formation.
  • Approach: Contrastive losses compare generated image pairs to enforce independent influence from each latent variable and are crucial for complete disentanglement.The losses penalize appearance differences unrelated to the varied latent variable.
  • Results: The model achieves highly disentangled latent variables and accurately controlled generation, while supporting real-image embedding into the disentangled latent space.The analysis also reports meaningful properties of the learned StyleGAN latent space.
  • Contributions: The paper proposes precise control of pose, expression, and illumination through an imitative-contrastive scheme leveraging 3D priors.It also analyzes the latent space and projects real images into it for analysis and decomposition.

2. Related Work

Prior work spans disentangled representation learning, conditional face synthesis, GAN-based face embedding and editing, and 3D-prior integration. DiscoFaceGAN is positioned within these directions while emphasizing generative modeling of interpretable facial factors.

  • Disentangled representation learning: Disentangled representation methods include InfoGAN and related approaches, but InfoGAN can be unstable and does not guarantee semantically meaningful latent variables.InfoGAN-CR adds a discriminator to identify latent codes under traversal.
  • Conditional GAN for face synthesis: Conditional GANs commonly combine random noise with preset factors and auxiliary classifiers, but they do not generatively model the conditional factors.These methods are widely used for identity-preserving face synthesis.
  • Face image embedding and editing with GANs: GAN-based face manipulation methods typically use encoder-decoder or generator-discriminator paradigms to embed images into disentangled representations.The paper likewise applies its method to face-image embedding.
  • 3D prior for GANs: Many face-synthesis methods incorporate 3D priors, especially 3D Morphable Models, for tasks including frontal synthesis, translation, re-animation, and reconstruction.The cited approaches use 3D information in several distinct generation and manipulation settings.

3. Approach

The approach combines interpretable 3D face parameters, imitative losses, and contrastive losses within a GAN to disentangle identity, expression, illumination, pose, and noise. Contrastive comparisons constrain domain-gap-induced freedom, enabling accurate control and strong disentanglement in generated faces.

  • 3. Approach: The generator models five independent factors: identity, expression, illumination, pose, and random noise, with 3D priors incorporated into an imitative-contrastive scheme.Latent variables are mapped to interpretable parameters for generation and rendering.
  • Imitative Learning: Four VAEs map standard-normal latent variables to identity, expression, illumination, and pose coefficients extracted from real images.The VAE encoders are discarded after training, retaining decoders for latent-to-parameter mapping.
  • Imitative Learning: The generator and analytic renderer receive the same interpretable parameters to produce a generated face and its rendered counterpart.Using interpretable parameters simplifies face-property control.
  • Imitative Learning: Imitative losses align generated and rendered faces through identity features, facial landmarks, illumination coefficients, skin color, and other 3DMM-derived properties.These losses encourage generated images to follow the properties characterized by the corresponding latent variables.
  • Domain gap: The rendered-real domain gap preserves realism but can cause unwanted changes in identity, background, expression, hair, or other factors.The generated image differs from its rendered counterpart by a free, partly uncontrollable residual.
  • Contrastive Learning: Contrastive learning varies one latent variable while holding the others fixed, then penalizes differences unrelated to that variable.Expression-invariant comparisons use rendered-face flow fields to warp images, while illumination-invariant losses preserve semantic structure and identity.
  • Generation Results: Generated samples show highly disentangled identity, expression, pose, and illumination, with precise control of expression, illumination, and pose.Real-image properties can also be combined with random identities to generate new identities matching a reference image's expression, lighting, and pose.
  • Contrastive Learning: Contrastive losses are essential for enhanced disentanglement and also improve imitation and face-property control by suppressing pairwise misalignment noise.Pose variations were found to be sufficiently disentangled without an additional pose-specific contrastive loss.

4. Experiments

Experiments evaluate DiscoFaceGAN’s disentanglement, controllability, reference-based generation, ablations, and image quality. Results show precise factor control and improved disentanglement from contrastive losses, with a quality trade-off in FID but comparable PPL.

  • 4.1. Generation Results: The model generates high-fidelity faces with diverse identities, poses, illumination, and expressions while preserving other factors when one factor varies.Expression, illumination, and pose can be precisely controlled through their parametric model coefficients.
  • 4.1. Generation Results: Reference-based generation combines expression, lighting, and pose extracted from real images with randomly generated identities.This produces new identities that mimic the properties of a real reference image.
  • 4.2. Ablation Study: Removing imitation losses prevents reasonable face generation, whereas removing contrastive losses prevents full disentanglement of different factors.Contrastive losses also improve the preciseness of imitation and generation control.
  • 4.3. Quantitative Evaluation: The disentanglement score measures whether varying one latent factor changes its corresponding property while keeping other generated properties stable.The evaluation generates 10K images from 1K fixed sets of other variables and 10 samples of the varied variable, then re-estimates factor variances.
  • 4.3. Quantitative Evaluation: Imitative learning yields high factor disentanglement, while contrastive learning further improves expression, illumination, and pose disentanglement.Identity disentanglement scores decrease with contrastive learning because identity changes make 3D reconstruction slightly unstable.
  • 4.3. Quantitative Evaluation: FID increases with the added losses, while PPL remains comparable to adversarial-loss-only training.FID and PPL are computed from 50K and 100K randomly generated images, respectively.

5. Latent Space Analysis and Embedding

The learned latent space exhibits factor-specific directions that remain stable across other factors and supports real-image embedding with controllable editing. Experiments demonstrate pose, illumination, and related face-property manipulation while preserving important image characteristics.

  • 5.1. Analysis of Latent Space: Varying a λ-space factor produces a stable W-space direction across different settings of the other factors.Property 1 concerns fixed start and end values; Property 2 indicates that the direction depends only on their difference.
  • 5.1. Analysis of Latent Space: Cosine similarities close to 1 empirically support consistent W-space directions for factor changes with fixed endpoints or offsets.The experiment samples 50 value pairs and 100 settings of the remaining factors for each pair.
  • 5.2. Real Image Embedding and Editing: Real images are embedded into W+ for higher-fidelity reconstruction, then edited using offsets induced by changing a selected λ-space variable.W and W+ are not geometrically interpretable directly, so the learned W-space properties provide the editing mechanism.
  • 5.2. Real Image Embedding and Editing: The method freely rotates real input faces by desired pose angles and preserves identity-bearing characteristics, hair structure, and illumination during frontalization.Frontalization results are evaluated on LFW against several previous methods.
  • 5.2. Real Image Embedding and Editing: Real-image relighting varies lighting direction and intensity and produces reasonable lighting transfer even under extreme indoor lighting outside the training data.The lighting-transfer evaluation uses MultiPIE images and compares against prior methods.

6. Conclusion and Future Work

DiscoFaceGAN combines 3D-prior imitation with contrastive losses to learn disentangled, controllable face representations. Experiments on virtual synthesis and face embedding support the efficacy of this scheme, while future applications remain prospective.

  • 6. Conclusion and Future Work: DiscoFaceGAN incorporates 3D priors into adversarial learning and uses contrastive losses to explicitly enforce disentanglement despite the rendered-real domain gap.The network is trained to imitate rendered 3D faces while contrastive learning handles generation freedom associated with the domain gap.
  • 6. Conclusion and Future Work: Extensive experiments on disentangled virtual face synthesis and face-image embedding demonstrate the efficacy of the imitative-contrastive learning scheme.The conclusion reports efficacy across both generation and embedding settings.
  • 6. Conclusion and Future Work: Accurately controlled virtual identity faces are proposed for future vision and graphics applications, including possible forgery detection and anti-spoofing.These applications are described as future exploration or possible uses rather than demonstrated results.

(Supplementary Material)

The supplementary material section identifies the authors and their institutional affiliations and provides the corresponding Microsoft Research Asia contact information.

  • (Supplementary Material): Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong are listed as the paper’s authors.
  • (Supplementary Material): The authors are affiliated with Tsinghua University and Microsoft Research Asia.
  • (Supplementary Material): The listed contact addresses use the Microsoft Research Asia email domain.

I. More Implementation Details

The implementation uses VAEs to parameterize face factors, StyleGAN-based training on FFHQ, progressive resolution growth, and staged imitative and contrastive losses. Training reaches 256 × 256 resolution, with reported loss weights not carefully tuned.

  • I. More Implementation Details: Separate VAEs parameterize identity, expression, illumination, and pose in λ space using three hidden layers with dimensions 512, 256, 128, and 32.The VAEs use ReLU activations and share the same structure across the four factors.
  • I. More Implementation Details: The z-space dimensions for the four factors are 128, 32, 16, and 3, with corresponding λ-space dimensions 160, 64, 27, and 3; additional noise has dimension 32.
  • I. More Implementation Details: Training follows StyleGAN on FFHQ, removes latent normalization and style mixing, and stops at 256 × 256 resolution due to time constraints.
  • I. More Implementation Details: The reported loss weights and other hyperparameters were not carefully tuned.

II. More Generation Results

DiscoFaceGAN generates diverse faces while disentangling identity, expression, pose, and illumination, enabling independent interpolation and controlled expression transfer. An attribute-preserving truncation trick improves quality while retaining non-identity attributes.

  • More Generation Results: Identity, expression, pose, and illumination vary highly independently, while expression, illumination, and pose remain precisely controllable through parametric coefficients.The model generates diverse identities and preserves unrelated attributes when one factor changes.
  • Latent Space Interpolation: Each latent factor can be interpolated independently, producing smooth changes in its corresponding attribute while preserving attributes tied to other factors.The interpolation results support continuous, factor-specific control in the learned latent space.
  • Attribute-Preserving Truncation Trick: The attribute-preserving truncation trick improves generation quality while maintaining pose, expression, and illumination.Unlike the original truncation trick, the proposed variant targets identity quality without changing these attributes.
  • Real Image Expression Transfer: Expression transfer successfully applies desired expressions to different subjects under varied poses and lighting conditions.This demonstrates controllable manipulation of real-image attributes across subjects.

VI. Analysis of Image Generation

The paper uses DiscoFaceGAN’s disentangled controls to inspect how StyleGAN’s stages contribute to pose, expression, lighting, and identity generation. Stage-specific latent replacement reveals the impact of each resolution stage on a selected factor.

  • Analysis of Image Generation: DiscoFaceGAN enables stage-by-stage analysis of StyleGAN’s image-generation process for pose, expression, lighting, and identity.The analysis leverages flexible control over disentangled factor variations.
  • Analysis of Image Generation: The heatmaps visualize color differences between generated images and the original source image for 256 × 256-resolution StyleGAN outputs.These visualizations accompany the stage-specific generation analysis.
  • Analysis of Image Generation: Replacing the W-space vector at two AdaIN layers in each generation stage isolates that stage’s impact on a factor of interest.The replacement vector comes from another latent code differing from the source in one factor.

VII. Limitations

The model’s generation quality degrades for extreme pose and lighting outside the real-image training distribution, and 3DMM limitations prevent control of detailed expressions and eye gaze.

  • VII. Limitations: Generation quality degrades under extreme pose and lighting because these inputs fall outside the real-image training distribution.Addressing this out-of-domain issue would require training images with a wider distribution beyond FFHQ.
  • VII. Limitations: The method cannot control detailed facial expressions and eye gaze because of the limited ability of 3DMM.This limitation concerns fine-grained expression and gaze control rather than the broader controllability demonstrated elsewhere.
Loading 2004.11660v2…