Source-linked AI summary
Interpreting the Latent Space of GANs for Semantic Face Editing
Yujun Shen, Jinjin Gu, Xiaoou Tang, Bolei Zhou
TL;DR
GANs offer photorealistic synthesis, but how latent codes organize facial semantics remains poorly understood. InterFaceGAN interprets those semantics with latent-space directions and projection-based manipulation, finding disentangled and controllable representations that support synthetic and real-face editing. Its interpretation relies on assumptions about binary-attribute separation and has limits for extreme manipulations.
Problem
How GAN latent codes connect to facial semantics such as gender and age, including their entanglement, remains insufficiently understood.
Method
InterFaceGAN identifies semantic directions in well-trained GAN latent spaces and uses projection-based conditional manipulation to edit facial attributes.
Results
InterFaceGAN finds latent representations that become disentangled after linear transformations and supports editing attributes of generated and real faces across multiple GAN models.
Takeaways & Limitations
The framework enables precise facial-attribute control with fixed GANs, including conditional editing and real-image manipulation.
Takeaways & Limitations
The method assumes each binary attribute has a separating latent hyperplane, and extreme directional edits can cease to resemble the original face.
Abstract
from arXiv · showhide
Despite the recent advance of Generative Adversarial Networks (GANs) in high-fidelity image synthesis, there lacks enough understanding of how GANs are able to map a latent code sampled from a random distribution to a photo-realistic image. Previous work assumes the latent space learned by GANs follows a distributed representation but observes the vector arithmetic phenomenon. In this work, we propose a novel framework, called InterFaceGAN, for semantic face editing by interpreting the latent semantics learned by GANs. In this framework, we conduct a detailed study on how different semantics are encoded in the latent space of GANs for face synthesis. We find that the latent code of well-trained generative models actually learns a disentangled representation after linear transformations. We explore the disentanglement between various semantics and manage to decouple some entangled semantics with subspace projection, leading to more precise control of facial attributes. Besides manipulating gender, age, expression, and the presence of eyeglasses, we can even vary the face pose as well as fix the artifacts accidentally generated by GAN models. The proposed method is further applied to achieve real image manipulation when combined with GAN inversion methods or some encoder-involved models. Extensive results suggest that learning to synthesize faces spontaneously brings a disentangled and controllable facial attribute representation.
1. Introduction
GANs can synthesize photorealistic faces from latent codes, but the organization of facial semantics in those codes and their usefulness for editing remain insufficiently understood. InterFaceGAN interprets these semantics and uses their latent structure for controllable face manipulation.
- GANs learn a nonlinear mapping from randomly sampled latent codes to photorealistic images.
- How latent codes determine facial attributes such as gender and age, and how those attributes become entangled, remains uncertain.
- Existing work largely improves synthesis quality while comparatively few studies examine what GANs learn in latent space.
- InterFaceGAN identifies latent semantics in well-trained face models and uses them for semantic face editing.
- The framework studies latent-space semantics alongside prior work on GAN synthesis and latent inversion for real image processing.
2. Framework of InterFaceGAN
InterFaceGAN models semantic attributes with latent-space hyperplanes and linear scores, then edits latent codes along semantic directions. For coupled attributes, projection-based conditional manipulation orthogonalizes directions to preserve the conditioned semantics.
- 2.1. Semantics in the Latent Space: The framework connects a deterministic GAN generator with a semantic scoring function, mapping latent codes z to semantic scores s.
- 2.1. Semantics in the Latent Space: A binary semantic is modeled by a latent-space hyperplane whose normal vector defines the direction associated with that attribute.
- 2.1. Semantics in the Latent Space: Semantic scores vary linearly with signed hyperplane distance, with λ measuring how rapidly the semantic changes along that direction.
- 2.1. Semantics in the Latent Space: Multiple semantics are represented by separation boundaries and linear coefficients; their covariance reveals disentanglement when the boundary normals are orthogonal.
- 2.2. Manipulation in the Latent Space: Single-attribute editing updates the latent code as z_edit = z + αn, where positive and negative α move the semantic score in opposite directions.
- 2.2. Manipulation in the Latent Space: Conditional manipulation subtracts a direction’s projection onto conditioned directions, changing one attribute while leaving the conditioned attribute unaffected.
3. Experiments
Experiments on PGGAN and StyleGAN test latent-space separation, attribute manipulation, disentanglement, and real-image editing. Linear boundaries support semantic control, while manipulation distance and training-data correlations delimit fidelity and independence.
- Experimental setup: Experiments evaluate InterFaceGAN on PGGAN for latent-space interpretation and on StyleGAN for style-based representations, with additional real-image manipulation.The evaluation covers latent-space separation, manipulation, disentanglement, and inversion-based editing.
- 3.1. Latent Space Separation: Over 95% validation accuracy and over 75% accuracy on the entire sample set support linear hyperplanes separating binary facial attributes.Five linear SVMs are trained for pose, smile, age, gender, and eyeglasses.
- 3.1. Latent Space Separation: Samples ranked by distance to each decision boundary show distinguishable positive and negative attribute groups, with extreme cases constructed by moving latent codes far along the normal direction.The figure contrasts near-boundary samples with extremely distant samples for each attribute.
- 3.2. Latent Space Manipulation: Moving latent codes along learned directions changes pose and four facial attributes in both positive and negative directions, including continuous pose changes and generated extreme poses or eyeglasses.The manipulation results indicate that the GAN encodes interpretable semantic directions despite limited training examples for extreme cases.
- 3.2. Latent Space Manipulation: Near-boundary editing preserves identity better, whereas moving beyond a certain distance causes severe appearance changes and eventually produces extreme cases.The reported threshold for the gender example is 5.0.
- 3.2. Latent Space Manipulation: A learned positive quality direction corrects some GAN artifacts, while conditional manipulation removes components of correlated attributes during independent editing.The method uses a linear SVM for artifact separation and projection subtraction for conditional attribute control.
- 3.3. Disentanglement: Pose and smile are nearly orthogonal to other attributes, whereas gender, age, and eyeglasses are highly correlated in the latent space and reflect correlations in CelebA-HQ.The observed correlations are also captured by the GAN during image generation.
- 3.4. Results on StyleGAN: StyleGAN’s W space is more disentangled than Z space and performs better for attribute editing, although it can encode correlated age-and-eyeglasses styles that limit projection-based decorrelation.Conditional manipulation decorrelates these attributes in Z space but cannot be applied to W space in the reported case.
4. Conclusion
InterFaceGAN interprets latent semantics and uses conditional manipulation to control facial attributes with fixed GANs, including real-image editing.
- InterFaceGAN interprets GAN latent semantics and enables precise facial-attribute control with fixed GAN models.
A. Overview
The appendix documents implementation details, proof material, and supplementary continuous attribute-editing results.
- The appendix introduces implementation details for InterFaceGAN.
- It provides a detailed proof of Property 2 from the main paper.
- A supplementary video presents continuous attribute-editing results.
B. Implementation Details
The implementation analyzes five facial attributes using an auxiliary predictor, large-scale GAN sampling, score-based candidate selection, and linear SVM verification.
- The analysis covers pose, smile, age, gender, and eyeglasses as five key facial attributes.Positive directions correspond to turning right, laughing, getting old, changing to male, and wearing eyeglasses.
- An auxiliary ResNet-50 model trained on CelebA annotations predicts attributes and facial landmarks for synthesized images.The landmarks are used to compute yaw pose, treated as a binary left-or-right attribute.
- 500K randomly sampled latent codes are used to reduce sampling randomness and obtain enough rare wearing-glasses examples.
- For each attribute, the 10K highest- and 10K lowest-scoring synthesized samples are selected as candidate extremes.This selection reduces ambiguity from imperfect attribute predictions.
- The remaining 30% of samples verify linear-classifier behavior using 512-dimensional latent codes and binary predictor-derived labels.
C. Proof
The proof section establishes geometric and probabilistic properties of high-dimensional spherical and Gaussian distributions used in the paper’s analysis.
- Property 2 analyzes probability mass near the equator of a high-dimensional Gaussian distribution.
- The proof reduces the geometric claim to comparing surface areas of subsets of the unit sphere.
- Theorem 1 bounds the probability that a coordinate of a unit spherical vector remains within α√(d−2) of zero.The stated bound applies for α ≥ 1 and d ≥ 4.
- The appendix completes the geometric bound using surface-area inequalities and probability estimates.
- The Gaussian Annulus Theorem states that almost all mass of a d-dimensional spherical Gaussian lies within a narrow annulus around its typical radius.The exception is bounded by 3e^(−cβ^2) under the stated β condition.