InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs

Yujun Shen; Ceyuan Yang; Xiaoou Tang; Bolei Zhou

Source-linked AI summary

InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs

Yujun Shen, Ceyuan Yang, Xiaoou Tang, Bolei Zhou

arXiv:2005.09635v2May 18, 2020cs.CVcs.LGeess.IV

TL;DR

InterFaceGAN addresses limited understanding of how GAN latent representations encode facial attributes and how that knowledge can be reused for editing. It identifies linear semantic subspaces, analyzes their correlations, and enables realistic, more precise manipulation without retraining, including extensions to real faces. The study supports a disentangled and controllable face representation while noting limits beyond faces and in StyleGAN boundary accuracy.

Problem
How GANs organize facial attributes in latent representations and whether pretrained models can support controllable synthesized and real-face editing remain insufficiently understood.
Method
InterFaceGAN bridges latent and semantic spaces with semantic classifiers, identifies linear semantic subspaces, and disentangles correlated attributes through subspace projection.
Results
InterFaceGAN realistically manipulates gender, age, expression, eyeglasses, pose, and artifacts, evaluates semantic and identity effects, and extends editing to real faces.
Takeaways & Limitations
GAN face synthesis spontaneously yields semantic knowledge that can be reused for precise, disentangled, and controllable face editing.
Takeaways & Limitations
The approach is presented as a pilot study for faces, and StyleGAN boundary correlations may be inaccurate because W space has a more complicated distribution than a Gaussian distribution.

Abstract

from arXiv · show

Although Generative Adversarial Networks (GANs) have made significant progress in face synthesis, there lacks enough understanding of what GANs have learned in the latent representation to map a random code to a photo-realistic image. In this work, we propose a framework called InterFaceGAN to interpret the disentangled face representation learned by the state-of-the-art GAN models and study the properties of the facial semantics encoded in the latent space. We first find that GANs learn various semantics in some linear subspaces of the latent space. After identifying these subspaces, we can realistically manipulate the corresponding facial attributes without retraining the model. We then conduct a detailed study on the correlation between different semantics and manage to better disentangle them via subspace projection, resulting in more precise control of the attribute manipulation. Besides manipulating the gender, age, expression, and presence of eyeglasses, we can even alter the face pose and fix the artifacts accidentally made by GANs. Furthermore, we perform an in-depth face identity analysis and a layer-wise analysis to evaluate the editing results quantitatively. Finally, we apply our approach to real face editing by employing GAN inversion approaches and explicitly training feed-forward models based on the synthetic data established by InterFaceGAN. Extensive experimental results suggest that learning to synthesize faces spontaneously brings a disentangled and controllable face representation.

1 INTRODUCTION

InterFaceGAN interprets how GANs encode facial semantics in latent representations and reuses that knowledge for controllable face editing without retraining. It further analyzes editing precision, semantic disentanglement, identity effects, layer-wise representations, and real-image manipulation.

GANs produce realistic faces, but how latent representations organize facial attributes and support controllable editing remains insufficiently explored.
InterFaceGAN bridges latent and semantic spaces using off-the-shelf classifiers, finding that true-or-false attributes align with linear latent subspaces.
Linearly varying latent codes manipulates gender, age, expression, eyeglasses, and pose, while conditional manipulation reduces attribute entanglement for more precise control.
The study quantitatively evaluates semantic changes, analyzes StyleGAN representations layer by layer, and examines how manipulation affects face identity.
InterFaceGAN extends to real face editing through GAN inversion and feed-forward models trained on synthetic before-and-after manipulation pairs.
The expanded study adds StyleGAN analysis, comparisons with PGGAN, quantitative editing evaluation, identity analysis, and synthetic-data feed-forward editing.

2 RELATED WORK

Related work covers GAN applications, latent-space semantic structure, semantic face editing, and GAN inversion. InterFaceGAN focuses on interpreting and reusing semantic knowledge learned by pretrained face generators.

GANs have been applied to image editing, super-resolution, inpainting, video synthesis, and other image-generation tasks.
Prior studies observed specialized generator units and vector arithmetic, while InterFaceGAN analyzes how face semantics are encoded in latent representations.
Semantic face editing seeks to change a target facial attribute while preserving other information, often requiring designed losses, labels, or specialized architectures.
GAN inversion reverses image-to-latent mapping so pretrained generators can be used with real images; InterFaceGAN uses inversion to reuse latent semantic knowledge for editing.

3 FRAMEWORK OF INTERFACEGAN

InterFaceGAN connects latent codes to semantic scores and models binary facial attributes with linear subspaces. It uses subspace projection to disentangle correlated attributes and edit synthesized faces without retraining.

Latent-Space Semantics: InterFaceGAN bridges latent and semantic spaces by scoring generated images and analyzing the mapping s = fS(g(z)).The generator maps latent space Z to image space X, while an attribute scoring function maps images to semantic space S.
Latent-Space Semantics: Binary facial attributes are modeled as hyperplanes whose normal vectors define semantic directions in the latent space.The assumption is that an attribute remains stable on one side of a boundary and reverses after crossing it.
Multiple Semantics: Different semantics are disentangled when their boundary normals are orthogonal; non-orthogonality measures semantic entanglement.Under the linear model, the semantic-score covariance is diagonal only when the boundary directions are mutually orthogonal.
Face Editing: Editing adds a scaled semantic direction to the latent code, while conditional editing removes projections onto directions corresponding to attributes that should remain fixed.The conditional direction is independent of the conditioned semantic subspaces; shared subspaces are difficult to isolate.
Implementation Details: The framework analyzes pose, smile, age, gender, and eyeglasses using auxiliary attribute predictions from synthesized faces.The attribute model is trained with CelebA annotations and also predicts facial landmarks used to compute yaw pose.

4 INTERPRETING FACE REPRESENTATION

InterFaceGAN interprets face representations in PGGAN and StyleGAN, contrasting a traditional latent input with StyleGAN’s mapped, per-layer latent representation. The analysis uses classification accuracy on latent separation boundaries.

Model Comparison: The study applies InterFaceGAN to PGGAN and StyleGAN, both producing 1024 × 1024 faces, while differing in how latent codes enter the generator.PGGAN feeds the code into the first convolutional layer, whereas StyleGAN maps Z to W and injects the mapped code into all convolutional layers.
Separability Analysis: Figure 3 reports classification accuracy for latent separation boundaries across facial attributes and models.The comparison concerns PGGAN and StyleGAN latent representations, including StyleGAN’s Z and W spaces.

4.1 Separability of Latent Space

Linear boundaries separate binary facial attributes in both PGGAN and StyleGAN latent spaces, with StyleGAN’s W space showing the strongest separability. Moving too far from a boundary can distort the original face.

Separability: Over 95% validation accuracy and over 75% entire-set accuracy were achieved by all PGGAN linear boundaries.The validation set contains 3K positive and 3K negative samples, while the entire set excludes the 20K high-confidence candidates.
Separability: StyleGAN’s W space can better fit the underlying real attribute distribution because it is not constrained to a normal distribution like Z space.The generator receives w rather than z, encouraging semantic organization in W space.
Separability: Both Z-space and W-space boundaries separate the validation set well, but W space shows much stronger separability than Z space.StyleGAN W-space accuracy also exceeds PGGAN Z-space accuracy.
Distance Effect: Samples near a separation boundary preserve the original face better than samples moved extremely far along the boundary normal.Large movements cause severe appearance changes and can produce extreme cases.

4.2 Semantics in Latent Space for Face Manipulation

InterFaceGAN finds that facial semantics can be manipulated by moving latent codes along identified directions, while manipulation quality depends on distance and latent space. Results extend from PGGAN to StyleGAN, with StyleGAN’s W space showing stronger robustness for long-distance edits.

Single-attribute manipulation: InterFaceGAN manipulates five facial attributes in positive and negative directions, including pose and eyeglasses, even when training data contain few extreme examples.The results suggest GANs encode interpretable semantics rather than producing images randomly.
Distance effect: Near-boundary edits preserve the original face, but moving latent codes too far causes severe appearance changes and identity loss.Such extreme samples are manually constructed and are unlikely to be directly drawn from a standard normal distribution.
Artifact correction: Moving latent codes toward a positive “quality” direction gradually corrects artifacts in deficient PGGAN syntheses.The quality direction is learned from 4K manually labelled deficient syntheses using a linear SVM.
StyleGAN: StyleGAN supports attribute editing in both Z and W spaces, while its more diverse training data enable higher-quality profile faces and younger faces than PGGAN.The comparison covers pose and age manipulations in the two latent spaces.
StyleGAN: W space is more robust than Z space for long-distance manipulation because Z-space edits can alter correlated attributes farther from the separation boundary.Near the boundary, the two spaces have similar effects; farther away, pose editing in Z space can change skin color, whereas W space is more stable.

4.3 Disentanglement Analysis and Conditional Manipulation

InterFaceGAN measures semantic correlations in latent representations and uses subspace projection to reduce attribute entanglement during conditional manipulation. Across PGGAN and StyleGAN, conditional editing improves control, while StyleGAN’s W space is generally more disentangled than Z space.

Disentanglement Analysis: Attribute correlations in synthesized PGGAN data closely match real-data correlations, indicating that GANs encode observed semantic dependencies.Pose and smile are nearly independent, whereas gender, age, and eyeglasses are highly correlated in both distributions.
Disentanglement Analysis: Boundary correlations similarly reflect semantic entanglement, so moving along one non-orthogonal direction can alter other attributes.Conditional manipulation projects directions to reduce these cross-attribute effects.
Conditional Manipulation: Conditional PGGAN editing preserves gender during age manipulation and age during eyeglasses manipulation more precisely than unconditional editing.The same principle extends to multiple conditions by forcing the target direction orthogonal to conditioning directions.
StyleGAN: StyleGAN’s W space is more disentangled than Z space, although its boundary correlations no longer fully align with real-data semantic distributions.The authors suggest that nonlinear classifiers might identify more accurate boundaries.
StyleGAN: For long-distance age editing, W space better preserves unrelated features, while Z-space subspace projection reduces residual age–eyeglasses entanglement.Near the boundary, both spaces produce similar manipulation results.

5 QUANTITATIVE ANALYSIS ON MANIPULATION

The quantitative analyses test whether semantic editing changes target attributes, how effects spread across attributes, where semantics arise across StyleGAN layers, and how identity is preserved. Results confirm target-score control, expose entanglement, and show that identity preservation depends on the edited attribute and latent space.

Re-scoring Analysis: InterFaceGAN increases the intended attribute scores, but manipulating one attribute can also change scores for correlated attributes.For PGGAN, age manipulation also increases gender scores, while eyeglasses manipulation substantially increases age scores.
Layer-wise Analysis: StyleGAN semantics are distributed across layers: pose uses 00-01, smile 02-05, age 02-07, gender 02-03, and eyeglasses 00-03.Editing layers 08-17 barely affects these attributes because they mainly control texture and background.
Identity Analysis: Gender affects identity most and smile least, while pose and eyeglasses also produce substantial identity changes.Identity discrepancy is measured using 256-dimensional recognition features and cosine distance.
Identity Analysis: StyleGAN W space best preserves identity among the evaluated latent spaces, consistent with its greater disentanglement.The authors relate this to identity being more complex than the other analyzed semantics.

6 REAL IMAGE MANIPULATION

InterFaceGAN extends semantic editing to real faces through GAN inversion or feed-forward models trained on synthetic before-and-after pairs. The two approaches differ in reconstruction, identity preservation, inference speed, and the complexity of attributes they can transfer.

Approaches: Real-face editing uses either GAN inversion followed by latent manipulation or feed-forward pixel-to-pixel models trained on InterFaceGAN-generated pairs.The inversion route includes optimization-based and encoder-based approaches.
Combining GAN Inversion with InterFaceGAN: GAN inversion is difficult because a finite-dimensional latent code cannot perfectly recover every real image when the GAN misses modes and diversity.Optimization minimizes reconstruction error with a fixed generator, whereas encoder-based inversion learns an inverse mapping.
Paired Synthetic Data: Synthetic-data-trained pix2pixHD models successfully manipulate gender, eyeglasses, and smile, while weight interpolation enables gradual editing.These models offer fast inference and preserve identity better than the encoder-based inversion approach.
Paired Synthetic Data: Pix2pixHD transfers easy-to-map semantics most reliably; smile editing is weaker and pose rotation is not applicable because both require substantial movement.The authors identify this limited attribute-transfer scope as the approach’s primary limitation.

7 DISCUSSION AND CONCLUSION

InterFaceGAN interprets facial semantics learned by GANs and uses them for realistic, more precisely controlled editing, including real-image manipulation. The discussion identifies broader generalization and more expressive manipulation as open directions.

Discussion: The linear manipulation assumption may fail for long-distance edits, motivating more adaptive nonlinear models.The method also depends on off-the-shelf classifiers, which limits the semantics that can be discovered.
Discussion: Extending interpretation beyond faces requires methods that can represent scene-level semantics, spatial layouts, and individual objects.The authors present this as a future direction for generic object and scene generation.
Conclusion: InterFaceGAN uses latent semantic knowledge and conditional manipulation to edit facial attributes realistically with more precise control.Further experiments support its application to real-image manipulation.

APPENDIX A PROOF

The appendix proves concentration properties for high-dimensional spherical and Gaussian distributions using symmetry, surface-area comparisons, and radial integration. These results establish that random points and Gaussian mass concentrate near equatorial slabs or thin annuli.

Property 2: Property 2 bounds the probability that a unit-normal hyperplane coordinate satisfies |n^Tz| ≤ 2α by a term proportional to αe^(-α^2/2), for α ≥ 1 and d ≥ 4.The proof reduces n to the first coordinate vector by rotational symmetry, so the target becomes a bound on |z1|.
Proof strategy: The proof represents the relevant Gaussian event as a subset of the unit sphere and compares the surface areas of corresponding spherical regions.It introduces U and K as spherical caps or hemispherical regions and uses surface-area notation to establish the comparison.
Proof strategy: The Gaussian probability calculation conditions on the radius R = ||z||2 and integrates over its cumulative distribution and density.This radial decomposition connects the spherical geometry argument to the Gaussian probability bound.
Theorem 2: The Gaussian Annulus Theorem states that all but at most 3e^(-cβ^2) of unit-variance spherical Gaussian mass lies within an annulus around radius √d.Here c is a fixed positive constant, and the theorem applies for β within the stated dimensional range.

APPENDIX B REAL IMAGE MANIPULATION

The appendix extends InterFaceGAN to real face editing by combining it with optimization-based and encoder-based GAN inversion. Optimization-based inversion better recovers input images and preserves identity, while both approaches support faithful semantic editing.

Real image manipulation: Figure 18 compares optimization-based and encoder-based GAN inversion for real face manipulation.The comparison integrates both inversion approaches with InterFaceGAN.
Real image manipulation: Optimization-based inversion better recovers the input images and therefore better preserves identity information than encoder-based inversion.Both approaches nevertheless retain interpretable latent semantics capable of faithfully editing real faces.