Source-linked AI summary
GANSpace: Discovering Interpretable GAN Controls
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, Sylvain Paris
TL;DR
GANs provide limited direct control over image content, and supervised methods require costly labels for each new control. GANSpace uses PCA and layer-wise perturbations to discover interpretable directions in existing GANs without retraining. Across BigGAN and StyleGAN-family models, it reports controls over geometry, viewpoint, lighting, facial attributes, and other image properties, while also exposing model entanglements and disallowed combinations.
Problem
Existing GANs offer little direct control over image content, while supervised control methods require expensive manual supervision for each new control.
Method
GANSpace applies PCA in latent or feature space, transfers BigGAN feature directions to latent space by regression, and applies directions layer-wise to pretrained GANs.
Results
The method yields interpretable controls across BigGAN512-deep and multiple StyleGAN and StyleGAN2 models, spanning pose, shape, lighting, facial attributes, and landscape attributes.
Takeaways & Limitations
Layer-wise PCA edits provide a simple way to analyze and control existing GAN representations without training a new model for each task.
Takeaways & Limitations
Some edits vary across starting images and some attribute combinations are disallowed by the model, reflecting built-in priors or biases.
Abstract
from arXiv · showhide
This paper describes a simple technique to analyze Generative Adversarial Networks (GANs) and create interpretable controls for image synthesis, such as change of viewpoint, aging, lighting, and time of day. We identify important latent directions based on Principal Components Analysis (PCA) applied either in latent space or feature space. Then, we show that a large number of interpretable controls can be defined by layer-wise perturbation along the principal directions. Moreover, we show that BigGAN can be controlled with layer-wise inputs in a StyleGAN-like manner. We show results on different GANs trained on various datasets, and demonstrate good qualitative matches to edit directions found through earlier supervised approaches.
1 Introduction
GANSpace identifies interpretable controls in existing GANs without post hoc supervision or expensive optimization. It combines PCA-based directions with layer-wise edits to expose controls ranging from pose and shape to lighting and facial attributes.
- Motivation: Existing GANs generate high-quality images but offer little direct content control, while supervised control methods require expensive labels for each new direction.Spatial-control methods also require users to provide painted label or edge maps.
- Approach: GANSpace discovers controls by browsing concepts learned by the model rather than learning a representation for each predefined concept.Useful directions require only an optional one-time user-labeling effort.
- Approach: PCA identifies important directions in StyleGAN latent space and BigGAN feature space, with BigGAN directions transferred back to latent space by regression.For BigGAN, the method computes PCA on intermediate feature tensors and aligns the resulting basis with latent directions using least squares.
- Layer-wise controls: The method applies to pretrained GANs and extends BigGAN with StyleGAN-like layer-wise style mixing without retraining.The techniques are described as algorithmically simple and intended to augment existing GANs.
- Results: Experiments report powerful controls across BigGAN512-deep and multiple StyleGAN and StyleGAN2 models, covering high-level geometry and nuanced appearance attributes.The paper also presents the edits as a way to visualize how the GAN operates through its “EiGANspace.”
- Layer-wise controls: Layer-wise perturbations decompose PCA directions into more interpretable edits, including purer head rotation, lighting, hair color, and hairstyle controls.StyleGAN notation specifies the component and the layer range receiving the edit.
3 Findings and Results
Across pretrained GANs, PCA reveals organized, interpretable variation: early components primarily govern geometry and viewpoint, while later components control appearance and detail. Layer-wise edits and comparisons show useful unsupervised controls, but entanglements, biases, and data-dependent limitations remain.
- PCA properties: Large-scale geometry and viewpoint changes are concentrated in the first 20 principal components, while later components alter appearance, backgrounds, and details.For StyleGAN2 on FFHQ, geometric changes are limited to rotations in the first three components because the aligned training set provides no translations.
- PCA properties: The first 100 StyleGAN2 components describe overall image appearance, whereas the remaining 412 dimensions produce subtle but perceptible appearance changes.The latent distribution has nearly independent principal coordinates with non-Gaussian unimodal distributions.
- PCA properties: BigGAN components appear class-independent, although later components can receive different interpretations across classes.Global motion components such as zoom were consistent across tested classes, while a blue shift could represent winter for one class and nighttime for another.
- Model entanglements and disallowed combinations: Selective layer-wise editing exposes interpretable controls, but learned correlations and priors can produce entangled effects or disallowed combinations.Examples include sporty cars with open-road backgrounds, dog rotation with mouth opening, and edits that do not affect certain faces or combinations.
- Comparisons: PCA-derived edits can resemble supervised directions without target transformations or supervised learning, though some edits are more entangled or introduce inconsistencies.The comparison includes visually close matches, variants of the same effect, and cases where both approaches remain highly entangled.
- Comparisons: PCA separates pose from major appearance variation, unlike random directions, which mix pose and appearance without a meaningful ordering.Fixing early principal coordinates preserves pose while varying appearance; fixing later coordinates preserves appearance while varying camera and orientation.
4 Discussion
The paper demonstrates simple controls for existing GANs and identifies future opportunities to analyze their representations and discover richer editing techniques.
- The method creates images with existing GANs instead of training a new model for each task.
- Future work could apply unsupervised methods beyond PCA to discover richer controls in image representations.
- The approach may support supervised edit training by narrowing the search space.
Broader Impact
The method inherits biases from its input GAN while also helping reveal biases that are otherwise difficult to identify; image-synthesis benefits and dangers remain relevant.
- As an image-synthesis tool, the method shares the potential benefits and dangers discussed for other image-synthesis tools.
- Because it takes an existing GAN as input without image training, the method inherits that GAN’s biases.
- The input GAN has limited ability to place makeup on male-presenting faces.
- The method can help discover biases that would otherwise be hard to identify.
1 Examples of Principal Components and Layerwise Edits
Supplementary examples show interpretable edits across multiple pretrained GANs, with practical details on PCA computation, model sources, slider ranges, truncation, and software availability.
- Interpretable edits are shown for many different models.
- The first 20 principal components are visualized for StyleGAN2 FFHQ, Cars, Cats, and BigGAN512-deep Husky.
- Incremental PCA uses N = 106 samples and takes around 1.5 hours on BigGAN512-deep versus 2 minutes on StyleGAN and StyleGAN2.
- The first ten or so components generally work over [−2...2], while face roundness can work over [−20...20] with truncation parameter 0.7.
- The GUI sliders use standard-deviation units, and later components support wider value ranges than earlier ones.
- The method uses interpolation to the mean for truncation, and its code is hosted online.
4 How many components are needed?
PCA concentrates much of the latent variation in the first components: 100 capture 85% of variance, while projecting onto them preserves nearly all overall face details.
- 85% of FFHQ latent variance is captured by the first 100 PCA dimensions, 92.5% by 200, and 98.5% by 400.
- Projecting onto the first K components replaces w with VK(VT_K(w − µ)) + µ.
- The first 100 components capture nearly all overall face details, while the remaining 412 make small shape and tone adjustments.
5 What is p(w)?
PCA analysis finds that StyleGAN2’s latent distribution p(w) has a relatively simple structure: principal coordinates are nearly independent and individually close to unimodal Gaussian-like distributions. This structure may support lower-dimensional representations of StyleGAN models.
- Distribution shape: StyleGANv2 PCA reveals that p(w) has a rather simple form across all analyzed models.The authors state that this analysis describes the shape of p(w) thoroughly and could support dimensionality reduction.
- Analysis procedure: The analyzed latent samples are obtained by projecting N = 10^6 samples wi ∼ p(w) into PCA coordinates with xi = V^T(wi − µ).V is a full-rank 512 × 512 PCA matrix for the StyleGAN2 models.
- Independence: The principal coordinates are empirically very nearly independent, allowing the distribution to be studied component by component.PCA decorrelates variables but does not guarantee independence; the empirical analysis assesses independence using mutual information.
- Marginal distributions: Individual principal-coordinate histograms are remarkably unimodal and visually Gaussian, although log-domain plots reveal some asymmetries.The observed distributions do not show heavy tails.
- Modeling implication: The authors propose modeling p(w) with a multivariate distribution built from suitable one-dimensional marginals, slightly distorted from a Gaussian.This representation would use substantially fewer parameters than StyleGAN’s M(z) representation.
6 BigGAN Principal Directions are Class-agnostic
BigGAN’s PCA components appear to transfer across classes, indicating that the discovered principal directions are not limited to the class used to compute them.
- Class transfer: Figure S10 illustrates transferring BigGAN edits between classes and supports the observation that PCA components are similar across different classes.The components appear to be shared across the analyzed BigGAN classes.
7 Entanglements and Disallowed Combinations
Most edits behave predictably across starting images, but some attribute edits depend on the input face and some edit combinations are unavailable to the model.
- Generalization: Most edits work predictably across different starting images, including a head-rotation edit that accurately rotates heads in the authors’ tests.The paper also documents input-dependent behavior for several facial edits.
- Input dependence: Baldness, makeup, white hair, and wrinkles can produce different results when the same edit is applied to different starting faces.These cases may reveal priors or biases learned by the GAN.
- Disallowed combinations: One combination of independently operating edits is not allowed by the model, exemplified by the absence of a “masculine little boy.”Figure S15 combines edits corresponding roughly to masculinity and age.
8 Comparisons
The supplementary comparisons visualize PCA directions against random directions and earlier supervised edits across several GANs and datasets. They also examine layerwise style mixing, dimensionality, cross-class transfer, input dependence, and content–style separation.
- Supervised comparisons: Figures S16–S18 compare edits discovered by this method with edits discovered by supervised methods [20] and [9].The comparisons cover BigGAN512-deep, StyleGAN FFHQ, StyleGAN Cars, and StyleGAN CelebaHQ.
- PCA versus random directions: Figures S2–S5 visualize the first 20 PCA directions alongside 20 scaled isotropic Gaussian directions for StyleGAN2 and BigGAN models.The models include FFHQ, Cars, Cats, and BigGAN512-deep Husky.
- Layerwise control: BigGAN exhibits StyleGAN-like style mixing even without explicit training for it, as latent inputs are swapped at different layers.Figure S6 visualizes swaps beginning at different layers.
- Distribution diagnostics: Figure S7 reports StyleGANv2 FFHQ principal-component variances, while Figure S8 visualizes samples projected onto 0, 1, 5, 10, 20, 100, and 512 PCA dimensions.Figure S9 displays marginal distributions for selected principal coordinates in ordinary and log domains.
- Cross-class edits: Figure S10 shows global components controlling translation and zoom, while later components applied selectively control textures such as clouds or nighttime illumination.The displayed components were computed from the husky class and transferred across BigGAN classes.
- Combined edits: Figure S15 visualizes combined edits over Δx0 ∈ [−3, 3] and x18, with axes corresponding roughly to masculinity and age.The components operate independently except for a combination the model does not produce.
- Representation structure: Figures S19–S22 show that the PCA basis separates content and style in ways not present in random bases, while early components can mix style changes with geometry.The latter effect is illustrated by identity changes in a spatially aligned dataset.