Source-linked AI summary
Closed-Form Factorization of Latent Semantics in GANs
Yujun Shen, Bolei Zhou
TL;DR
Existing GAN latent-semantic discovery methods rely on synthesized samples, predefined attributes, and labels or classifiers. SeFa addresses this gap with a closed-form factorization of pretrained generator weights, identifying latent directions directly from the generation mechanism. The method finds versatile semantics across GAN models and datasets and supports image manipulation without training or sampling.
Problem
Discovering meaningful GAN latent directions for image editing is difficult because latent spaces are high-dimensional, while existing approaches require predefined classifiers or labeled synthesized samples.
Method
SeFa discovers latent semantic directions in a closed form by examining the first latent-space projection and decomposing pretrained generator weights.
Results
SeFa identifies versatile semantic directions across PGGAN, StyleGAN, BigGAN, and StyleGAN2 models trained on diverse datasets, with results comparable to or more versatile than supervised approaches.
Takeaways & Limitations
SeFa enables unsupervised latent-semantic discovery and image manipulation across multiple GAN architectures and datasets without training or sampling.
Takeaways & Limitations
Comparisons with Info-PGGAN use different samples because Info-PGGAN requires model retraining and therefore produces a different model from the officially released one.
Abstract
from arXiv · showhide
A rich set of interpretable dimensions has been shown to emerge in the latent space of the Generative Adversarial Networks (GANs) trained for synthesizing images. In order to identify such latent dimensions for image editing, previous methods typically annotate a collection of synthesized samples and train linear classifiers in the latent space. However, they require a clear definition of the target attribute as well as the corresponding manual annotations, limiting their applications in practice. In this work, we examine the internal representation learned by GANs to reveal the underlying variation factors in an unsupervised manner. In particular, we take a closer look into the generation mechanism of GANs and further propose a closed-form factorization algorithm for latent semantic discovery by directly decomposing the pre-trained weights. With a lightning-fast implementation, our approach is capable of not only finding semantically meaningful dimensions comparably to the state-of-the-art supervised methods, but also resulting in far more versatile concepts across multiple GAN models trained on a wide range of datasets.
1. Introduction
GANs encode interpretable attributes in their latent spaces, but discovering useful semantic directions is difficult and existing supervised approaches require labels or classifiers. SeFa instead discovers directions directly from pretrained generator weights and works across GAN models and datasets.
- GAN latent spaces encode interpretable attributes that can support image and scene editing.Examples include gender in face synthesis and lighting conditions in scene synthesis.
- Finding meaningful latent directions is challenging because GAN latent spaces are high-dimensional and contain diverse image semantics.
- Existing supervised methods sample latent codes, synthesize images, obtain labels, and train latent-space classifiers.Labels may come from pretrained attribute predictors or simple image statistics such as object position and color tone.
- These supervised pipelines depend on available target classifiers or labels, while sampling is time-consuming and unstable.
- SeFa discovers latent semantic directions without training or sampling by analyzing the GAN generation mechanism and decomposing pretrained weights.The method examines the first projection step acting directly on the latent space.
- SeFa is demonstrated on PGGAN, StyleGAN, BigGAN, and StyleGAN2 trained on different datasets, including manipulation such as object rotation without 3D models or pose labels.
2. Method
SeFa discovers interpretable GAN latent directions by decomposing the generator’s learned weights rather than using sampled images, labels, or additional training. It identifies directions associated with large projected variations and applies across several GAN architectures.
- Unsupervised Semantic Factorization: SeFa discovers latent semantic directions by decomposing the weights of a pre-trained GAN generator in closed form.The method is unsupervised and independent of data sampling and model training.
- Generation Mechanism of GANs: GAN generation proceeds through layer-wise projections, so SeFa examines the first affine transformation from latent code z to projected code y.This transformation uses weight matrix A and bias b, with y = Az + b.
- Manipulation Model in GAN Latent Space: Image editing uses a latent direction n by replacing z with z + αn, where α controls manipulation intensity.After the first projection, this changes the projected code by adding αAn.
- Unsupervised Semantic Factorization: The optimization selects directions n that maximize ||An||2^2, favoring latent movements that produce large variations after projection.Directions projected to zero remain unchanged in the output synthesis.
- Unsupervised Semantic Factorization: The top-k semantics are the eigenvectors of A^T A associated with its k largest eigenvalues.This gives SeFa a closed-form solution for the most important latent directions.
- Implementation on GAN Models: SeFa adapts its analyzed transformation to PGGAN, StyleGAN, and BigGAN, including latent-to-feature-map, latent-to-style-code, and combined mappings.For StyleGAN, weight parameters from selected layers can be concatenated to interpret all or a subset of layers.
3. Experiments
Experiments evaluate SeFa across GAN architectures, datasets, layers, and comparisons with supervised and unsupervised alternatives. The results show interpretable, generalizable directions, efficient discovery, and competitive or improved editing behavior.
- Results on Diverse Models and Datasets: SeFa is evaluated on StyleGAN, BigGAN, and StyleGAN2 models trained across faces, anime, objects, scenes, streetscapes, and ImageNet.The experiments cover multiple architectures and dataset domains.
- Results on Diverse Models and Datasets: Layer-wise StyleGAN analysis reveals a semantic hierarchy: bottom layers control car rotation, middle layers shape, and top layers color.A user study reports that SeFa finds human-understandable concepts even from particular layers.
- Results on Diverse Models and Datasets: SeFa requires less than 1 second for one model and supports interactive editing after decomposing model weights.An interface enables human-model interaction and semantic annotation.
- Results on Diverse Models and Datasets: SeFa discovers BigGAN directions that manipulate images from different ImageNet categories, supporting its generalization across conditional synthesis outputs.The evaluated BigGAN is conditionally trained on ImageNet.
- Comparison with Supervised Approach: SeFa achieves similar qualitative editing performance to InterFaceGAN for pose, gender, eyeglasses, and expression while avoiding data sampling and model training.The comparison uses PGGAN and StyleGAN target models.
- Comparison with Supervised Approach: SeFa identifies hair color, hair style, brightness, and non-binary hair-style variations that supervised baselines cannot readily capture.The limitation is attributed to unavailable predictors and binary-attribute objectives.
- Comparison with Unsupervised Baselines: SeFa outperforms GANSpace on attribute re-scoring and user study despite close FID scores, and produces more accurate semantics than Info-PGGAN.GANSpace and SeFa share the generator and manipulation model underlying their close FID scores.
- Real Image Editing: Real-image editing with SeFa removes eyeglasses and changes face pose, indicating that discovered directions generalize to inverted real images.GAN inversion projects the target image into StyleGAN's latent space.
4. Conclusion
The paper proposes a closed-form solution for factorizing latent semantics learned by GANs and demonstrates its ability to identify versatile semantics across different GAN models in an unsupervised manner.
- SeFa provides a closed-form solution for factorizing latent semantics learned by GANs.
- The algorithm identifies versatile semantics from different types of GAN models without supervision.