Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning

1. Introduction

• The method targets a gap in de novo face synthesis: without prior or weak supervision, disentangled latent variables are not guaranteed to represent semantically meaningful factors.Disentangled representations are useful for constrained generation and for embedding real images into a learned feature space.Source: §1 · pp. 1–2
• DiscoFaceGAN learns independent latent variables for identity, expression, pose, and lighting by translating them into parametric-model coefficients and training with 3D priors and imitative rendering losses.The imitative losses encourage generated face properties to match the analytic rendering process; the domain gap between rendered and real faces otherwise leaves uncontrollable generation freedom.Source: §1 · p. 2
• Contrastive losses compare generated-image pairs sharing latent variables and penalize appearance differences, encouraging each variable to influence the output independently.The paper reports these losses are crucial for complete latent-variable disentanglement.Source: §1 · p. 2
• The authors report highly disentangled latent variables and precise control of face generation, and demonstrate embedding real images into the learned space for analysis and decomposition.The StyleGAN-based method generates faces that do not correspond to real people and can be extended to other GAN models.Source: §1 · p. 2
• The paper analyzes properties of the disentangled StyleGAN latent space and presents them as evidence supporting factor disentanglement.Source: §1 · p. 2

Disentangled representation learning

• Prior face disentanglement spans bilinear and RBM models, InfoGAN [6], pairwise or 3D GAN approaches [29] [11] [32], and VAE methods [26] [48] [5] [25]; InfoGAN can be unstable [48] and does not guarantee meaningful latent variables [30] [7].InfoGAN-CR [29] identifies latent codes under traversal, while SD-GAN [11] uses image pairs to separate identity and appearance.Source: §2 · p. 2

Conditional GAN for face synthesis

• Conditional GANs use noise and preset factors as constraints, with an auxiliary classifier or feature extractor restoring those factors, but they do not model the factors generatively.The authors state that their method can also handle face-generation tasks previously addressed with CGAN frameworks.Source: §2 · p. 2

Face image embedding and editing with GANs

• GAN face manipulation methods commonly use an encoder-decoder/generator-discriminator paradigm to embed images into latent representations of facial properties; DiscoFaceGAN can also embed face images in its disentangled space.Source: §2 · p. 2

3D prior for GANs

• Unlike prior face-synthesis methods that often leverage 3DMMs, DiscoFaceGAN uses 3DMM only as a training prior and requires neither a 3DMM nor rendering after training.Prior approaches include 3DMM-based frontal-face synthesis, cycle translation between rendered and real faces, re-animation, and portrait reconstruction or pose manipulation.Source: §2 · p. 2

3. Approach

• The method combines adversarial generation with 3D priors and contrastive learning to disentangle identity, expression, illumination, pose, and other image properties.The generator receives separate latent variables for the five factors, including random noise for properties such as background.Source: §3 · p. 3

3.1. Imitative Learning

• Interpretable 3DMM coefficients for identity, expression, illumination, and pose condition both the generator and an analytic renderer.Four VAE decoders map sampled normal latent variables to these coefficients; using the interpretable coefficients as generator input makes face properties easier to control.Source: §3.1 · p. 3
• Imitative losses align generated faces with rendered faces using identity features, facial landmarks, illumination coefficients, and average skin color.The identity-feature loss allows a margin to accommodate the domain gap between rendered and real faces; landmarks supervise expression and pose.Source: §3.1 · pp. 3–4
• The domain gap must remain large enough to preserve realism and avoid confinement to the 3DMM identity subspace, but it can cause unwanted changes across factors.Because generated images can deviate from rendered faces in identity and non-face content, the resulting variation is partly uncontrollable; contrastive learning constrains these deviations.Source: §3.1 · p. 4, §3.2 · p. 5

3.2. Contrastive Learning

• Contrastive learning compares generated pairs that differ in one latent factor and penalizes changes attributable to the other factors.For expression-invariant comparison, a flow field derived from rendered faces with differing expressions warps one generated image toward the other; illumination-invariant losses compare face structure and identity features. The method reports satisfactory disentanglement with expression and illumination losses, without an additional pose loss.Source: §3.2 · p. 4
• Pairwise comparisons constrain otherwise uncontrollable deviations from rendered faces, enhancing disentanglement and, empirically, improving imitation and face-property control.Misalignment between generated and rendered pose or expression increases the contrastive loss, helping suppress imitation noise.Source: §3.2 · p. 5

4. Experiments

• Training uses StyleGAN on FFHQ, first with adversarial and imitation losses through 15M real images, then with contrastive losses through 20M total.Training uses 256 × 256 resolution and removes latent-input normalization and style mixing.Source: §4 · p. 5

4.1. Generation Results

• Generated faces show diverse identities, poses, illumination, and expressions, with factor variations largely disentangled and expression, illumination, and pose precisely controllable.Varying one factor can preserve the others; control uses the corresponding parametric model coefficients.Source: §4.1 · p. 6
• Reference-based generation combines a real image’s expression, lighting, and pose with randomly generated identity variables to create new identities.Figure 5 illustrates this procedure.Source: §4.1 · p. 6, Figure 5 · p. 6

4.2. Ablation Study

• Without imitation losses, the network fails to generate reasonable faces; without contrastive losses, factor disentanglement and precise imitation are weaker.Contrastive losses help prevent expression or lighting changes from affecting identity-related traits or hair structure, and improve control such as mouth-closing status.Source: §4.2 · p. 6, Figure 6 · p. 6

4.3. Quantitative Evaluation

• The disentanglement score measures whether changing one latent factor changes its corresponding image property while other estimated properties remain stable.The evaluation generates 10,000 images, re-estimates factors with 3D reconstruction (identity with face recognition), and normalizes variances using FFHQ.Source: §4.3 · pp. 6–7
• Imitative learning yields high disentanglement; contrastive learning further improves expression, illumination, and pose scores but lowers identity DS; FID increases while PPL remains comparable to adversarial-only training.The authors attribute the identity-score decrease to instability in 3D reconstruction when identity changes.Source: Table 1 · p. 6, §4.3 · p. 7

5.1. Analysis of Latent Space

• For the first four λ-space factors, W-space change direction is independent of other factors and depends only on the change between the factor values.The properties hold both for fixed start and end values and for any fixed offset.Source: §5.1 · p. 7
• Mean pairwise cosine similarities for the tested directions were close to 1, supporting high consistency across sampled factor values and other factors.The evaluation also presents statistics from a model trained without the imitative-contrastive losses; Table 2 reports cosine similarities for fixed endpoints and fixed offsets.Source: §5.1 · p. 7, Table 2 · p. 7

5.2. Real Image Embedding and Editing

• Real images are embedded into W+ with optimization for higher-fidelity reconstruction, then edited using a W-space offset computed from the embedded code and a chosen factor change.The target factor’s start and end values can be set flexibly; the offset can be computed using any values for the other latent variables.Source: §5.2 · p. 8
• Pose editing rotates input faces by desired angles and supports frontalization while preserving identity-bearing characteristics and contextual information such as hair structure and illumination.Figure 7 presents pose manipulation and frontalization results on LFW alongside prior methods.Source: §5.2 · p. 8, Figure 7 · p. 8
• Relighting varies illumination direction and intensity, producing reasonable lighting directions aligned with references even for an extreme indoor-lighting case that may lie outside the training data.Figure 8 shows real-image light editing and a challenging lighting-transfer task on MultiPIE images.Source: §5.2 · p. 8, Figure 8 · p. 8

6. Conclusion and Future Work

• DiscoFaceGAN integrates 3D priors into adversarial learning to imitate rendered faces, using contrastive losses to address the real–rendered domain gap and explicitly enforce disentanglement.Experiments on virtual face synthesis and face image embedding demonstrate the efficacy of the imitation-contrastive learning scheme.Source: §6 · p. 8
• Future work will explore vision and graphics applications of generated virtual identity faces with accurately controlled properties.Source: §6 · p. 8
• The method may also support forgery image detection and anti-spoofing by analyzing real and fake images in the disentangled space.Source: §6 · p. 8

I. More Implementation Details

• Each λ-space VAE for identity, expression, illumination, and pose has three hidden layers, listed dimensions 512, 256, 128, and 32, and ReLU activations.The four factor VAEs share this structure.Source: §I · p. 11
• The z-space dimensions for z1–z4 are 128, 32, 16, and 3; corresponding λ-space dimensions are 160, 64, 27, and 3, with z5 set to 32.Source: §I · p. 11
• StyleGAN training on FFHQ follows the original procedure, omitting input-latent normalization and style mixing, and reaches 256 × 256 resolution due to time constraints.The λ-space VAEs follow the first stage of the schedule in.Source: §I · p. 11
• At resolutions ≤32 × 32, training uses face-region average L1 pixel loss weighted 20; from 64 × 64, it switches to imitative losses, then adds contrastive losses after 15M real images.The adversarial loss weight remains 1; training continues to 20M real images, with contrastive-loss weight set to 1000 and imitative-loss weight reduced to 100. Hyperparameters were not carefully tuned.Source: §I · p. 11

II. More Generation Results

• Randomly generated faces vary widely in identity, pose, illumination, and expression.The examples are shown in Figures V and VI.Source: §II · p. 11
• Identity, expression, pose, and illumination variations are highly disentangled from one another.This is reported for the additional generation results.Source: §II · p. 11
• Parametric model coefficients precisely control expression, illumination, and pose.Source: §II · p. 11

III. Latent Space Interpolation

• Each factor can be interpolated independently in the λ space.The paper presents these interpolation results in Figure VII.Source: §III · p. 11
• Changing one factor produces continuous and smooth changes in its corresponding generated-image attributes.Source: §III · p. 11
• During interpolation, attributes related to other factors remain unchanged.Source: §III · p. 11

IV. Attribute-Preserving Truncation Trick

• The standard StyleGAN truncation moves latent code w toward the empirical W-space center, but may change all attributes controlled by factors in λ space.It uses w′ = w + (1 − ψ)(w̄ − w), where ψ < 1.Source: §IV · p. 11
• Attribute-preserving truncation aims to improve generated identity quality while keeping expression, illumination, and pose unchanged.The method is based on latent-space properties described in Section 5.1 of the main paper.Source: §IV · p. 11
• Figure I shows improved generation quality while preserving pose, expression, and illumination; the original truncation trick [3] does not preserve these attributes.Source: Figure I · p. 12

V. Real Image Editing

• The paper presents real-image pose and lighting modification results in Section 5.2 of the main paper.Source: §V · p. 12
• In an expression-transfer task, the method successfully transfers desired expressions to different subjects.Source: §V · p. 12
• Expression transfer succeeds under varied pose and lighting conditions.Source: §V · p. 12

• The method’s disentangled factor controls enable an analysis of StyleGAN’s image-generation process.Source: §VI · p. 12
• At each generation stage, the analysis replaces the corresponding W-space vector with one from a sample differing in a single factor, so image changes reflect that stage’s impact.The replacement is applied to the two AdaIN layers at each spatial resolution.Source: §VI · p. 13
• The stage-by-stage visualizations examine impacts on pose, expression, lighting, and identity; heatmaps show color differences from the original source image.Source: §VI · p. 13, Figure III · p. 13

VII. Limitations

• Generated-face quality decreases when pose and lighting inputs fall outside the real-image training distribution.The paper identifies this as a common out-of-domain issue.Source: §VII · p. 13, Figure IV · p. 13
• The model cannot control detailed facial expressions or eye gaze because of the limited capabilities of the 3DMM.Source: §VII · p. 13
• Addressing the out-of-distribution quality issue would require training images with a wider range than FFHQ provides.Source: §VII · p. 13