Source-linked AI summary
Unsupervised Discovery of Interpretable Directions in the GAN Latent Space
Andrey Voynov, Artem Babenko
TL;DR
Existing GAN latent-direction discovery relies on supervision or is restricted to simple transformations. This paper proposes a completely unsupervised method for pretrained generators, discovering interpretable and non-trivial directions such as background removal, which supports competitive weakly-supervised saliency detection.
Problem
Prior latent-direction discovery requires human labels, pretrained supervised models, or self-supervision limited to simple augmentation transformations.
Method
The paper proposes a completely unsupervised, model-agnostic approach to discover diverse interpretable directions in a pretrained GAN generator.
Results
The method reveals previously unobserved or supervision-intensive interpretable directions, including background removal, and achieves competitive performance for weakly-supervised saliency detection.
Takeaways & Limitations
Discovered latent directions can provide practical benefits beyond interpretation, including synthetic-data generation for weakly-supervised saliency detection.
Takeaways & Limitations
The saliency-detection application still requires image-level ILSVRC class labels, though it does not require pixel-level or dataset-specific supervision.
Abstract
from arXiv · showhide
The latent spaces of GAN models often have semantically meaningful directions. Moving in these directions corresponds to human-interpretable image transformations, such as zooming or recoloring, enabling a more controllable generation process. However, the discovery of such directions is currently performed in a supervised manner, requiring human labels, pretrained models, or some form of self-supervision. These requirements severely restrict a range of directions existing approaches can discover. In this paper, we introduce an unsupervised method to identify interpretable directions in the latent space of a pretrained GAN model. By a simple model-agnostic procedure, we find directions corresponding to sensible semantic manipulations without any form of (self-)supervision. Furthermore, we reveal several non-trivial findings, which would be difficult to obtain by existing methods, e.g., a direction corresponding to background removal. As an immediate practical benefit of our work, we show how to exploit this finding to achieve competitive performance for weakly-supervised saliency detection.
1. Introduction
GAN latent spaces contain semantically meaningful directions, but prior discovery methods require supervision. The paper introduces a completely unsupervised, model-agnostic approach that discovers interpretable and practically useful directions, including background removal.
- Motivation: GANs can encode human-interpretable transformations in their latent spaces, making generation more controllable.Examples include domain-agnostic transformations such as zooming or translation.
- Research gap: Prior methods require human labels, pretrained supervised models, or self-supervision limited to augmentation-like transformations.These requirements can be expensive or impossible to obtain and restrict the directions discoverable by existing approaches.
- Approach: The proposed approach discovers diverse interpretable directions in a pretrained generator without supervision or costly generator retraining.It seeks directions whose transformations are easy to distinguish, encouraging independent factors of variation.
- Findings: For several generators, the method identifies many human-interpretable directions and reveals non-trivial manipulations difficult for existing techniques to obtain.The paper specifically highlights a direction corresponding to background removal.
- Application: The background-removal direction is exploited to generate high-quality synthetic data for weakly-supervised saliency detection with competitive performance.This is presented as an immediate practical benefit of the discovered directions.
2. Related work
Prior work studies interpretable latent directions through supervision, disentangled-generator training, or Jacobian-based analysis. The paper contrasts these approaches with a lightweight method that operates on pretrained generators and can uncover surprising manipulations.
- Interpretable directions: Supervised methods use human-provided labels or pretrained attribute classifiers to identify latent directions.They construct a separating hyperplane from pseudo-labels and use its normal as the direction.
- Interpretable directions: Unlike prior approaches, the proposed unsupervised method often identifies surprising directions corresponding to non-trivial image manipulations.Existing approaches generally discover only directions researchers already expect to identify.
- Disentanglement learning: Disentangled generative models can reduce generation quality and diversity, whereas this method does not affect the pretrained generator distribution.The paper notes that competitiveness with state-of-the-art generators remains unclear for disentangled models.
- Jacobian decomposition: Jacobian-based curves can capture interpretable transformations but may entangle effects and require expensive, per-latent-code iterative computation.The proposed alternative identifies a set of directions at once with a lightweight approach.
3. Method
The method jointly learns latent directions and a reconstructor so that distinct image transformations are easy to distinguish while shifts remain continuous. It uses a pretrained, fixed generator and constrains the direction matrix to avoid degenerate high-norm solutions.
- Motivation: Random latent directions can alter several variation factors simultaneously, motivating directions whose image transformations do not interfere.The method targets directions that are easier to distinguish and ideally affect a single factor of variation.
- Learning protocol: A training sample pairs G(z) with G(z + A(εe_k)), where A's k-th column defines the shifted latent direction.The latent code z is sampled from N(0, I), while ε specifies a signed shift magnitude.
- Learning protocol: The reconstructor R predicts both the direction index k and the signed shift magnitude ε from the generated image pair.Joint optimization makes transformations easier to classify, encouraging directions that are disentangled and often human-interpretable.
- Optimization objective: The objective combines classification and regression losses, with λ=0.25 in all experiments.Classification uses cross-entropy, while regression uses mean absolute error.
- Optimization objective: The regression term enforces continuous effects along directions and prevents abrupt mappings that collapse images to a fixed output.Without this protection, the learned direction can map all images to the same image.
- Practical details: General linear A was impractical because optimization frequently produced columns with very high l2-norms.The classification loss can favor large constant shifts because they make generated samples nearly identical.
- Practical details: Unit-norm and orthonormal parameterizations produced similar interpretable directions, but unit-norm columns were more expressive overall.Orthonormal columns nevertheless found more interesting directions on some datasets; the experiments used different choices by dataset.
4. Experiments
The approach is evaluated without external supervision across four datasets and generator settings. Because interpretability and disentanglement are difficult to measure directly, the experiments introduce quantitative measures including reconstructor classification accuracy.
- Experimental setting: All experiments operate in a completely unsupervised manner without exploiting external supervision.
- Datasets and generator models: The evaluation covers MNIST, AnimeFaces, CelebA-HQ, and BigGAN trained on ILSVRC.The corresponding generators include Spectral Norm GAN, ProgGAN, and BigGAN.
- Optimization: The experiments use Adam to jointly learn A and R, with learning rate 0.0001 and dataset-dependent batch sizes and training steps.ProgGAN uses 2 · 10^5 gradient steps, while the other experiments use 10^5 steps.
- Evaluation metrics: Interpretability and disentanglement are evaluated with two proposed measures because they are challenging to measure directly.
- Evaluation metrics: High reconstructor classification accuracy indicates that directions are easy to distinguish and their image transformations influence different variation factors.RCA evaluates the reconstructor's direction-index classification task.
2. Individual interpretability (mean-opinion-score, MOS).
The method evaluates individual latent directions with human assessors, marking directions interpretable when they act consistently across latent codes and affect a single variation factor. Learned directions are reported as easy to interpret, while optimization concentrates transformations and BigGAN directions span geometry, texture, and color.
- Evaluation protocol: Eleven human assessors evaluate whether each direction operates consistently across latent codes and affects a single variation factor.A direction receives an interpretability mark of 1 only when it meets both requirements; MOS averages marks across assessors and directions.
- Qualitative results: Learned directions transform all samples in the same manner for any latent code and are easy to interpret.
- Evolution of directions: Optimization causes a direction that initially affects several factors to focus on digit thickness.The comparison uses snapshots from the identity initialization through the final direction matrix.
- Direction parameterization: Orthonormal A outperforms unit-norm columns on one dataset, but provides no practical benefit for MNIST or CelebA.The authors conjecture that orthonormality may regularize direction diversity, without advocating it universally.
- BigGAN direction types: BigGAN assessors categorize interpretable directions as geometry, texture, or color transformations.Examples include zoom, shift, rotation, background blur, adding grass, sharpness, lighting, and saturation.
- BigGAN direction types: Interpretable coordinate directions mostly involve color or texture, whereas interpretable random directions mostly affect geometry through zooming.
5. Weakly-supervised saliency detection
The discovered BigGAN background-removal direction generates synthetic saliency masks from shifted images, enabling segmentation training with image-level class supervision. On ECSSD, the resulting method achieves competitive weakly-supervised performance.
- Synthetic data generation: The background-removal direction hbg generates synthetic data for weakly-supervised saliency detection.
- Synthetic mask construction: Shifting BigGAN samples by hbg leaves foreground objects unchanged while turning background pixels white for threshold-based labeling.The threshold is set to θ = 0.95 when intensities lie in [0, 1].
- Weak supervision pipeline: The pipeline selects frequent ImageNet classes using top-5 predictions from ResNet-18, generates pseudo-labeled samples, filters mask areas, and trains a segmentation model.It uses the top 25% most frequent classes, excludes masks below 0.05 or above 0.5 of image area, and applies the trained model to real data.
- Supervision: The saliency approach requires image-level ILSVRC class labels but no pixel-level or dataset-specific supervision.
- Results: The method achieves MAE equal to 0.099 on ECSSD, described as competitive among methods using the same image-level ILSVRC supervision.
6. Ablation
The ablations examine how the number of latent directions and the shift-loss multiplier affect disentanglement and interpretability. Small K weakens disentanglement, while extreme loss scaling degrades direction quality.
- Small K values are inferior because easier classification does not enforce directions to be disentangled.
- The K ablations are evaluated on MNIST and ILSVRC using individual interpretability, MOS, RCA, and human-counted interpretable directions.
- Higher K does not harm interpretability but often produces duplicate directions.
- Extreme shift-loss scaling factors degrade quality: λ = 0 produces collapse directions, while high λ makes directions mostly similar.
7. Conclusion
The paper presents a completely unsupervised method for discovering interpretable directions in pretrained GAN latent spaces. It finds previously unobserved or supervision-intensive directions and applies background removal to weakly supervised saliency detection.
- The method is completely unsupervised and universally applicable to any pretrained generator.
- Across several standard datasets, it reveals interpretable directions that were previously unobserved or required expensive supervision.
- A revealed direction generates high-quality synthetic data for weakly supervised saliency detection.
8. Supplementary material
The supplementary material details orthogonal direction parametrization, DVN computation, evaluations, implementation settings, and additional observations about discovered directions.
- 8.1. Orthogonal parametrisation: Skew-symmetric matrices are mapped through the matrix exponential to orthogonal matrices with positive determinant for latent basis parametrization.
- 8.2. Direction Variation Naturalness: DVN evaluates whether a generated-image split transfers consistently to real images by retraining a classifier and measuring its accuracy on generated data.
- 8.2. Direction Variation Naturalness: The supplementary figures illustrate generated and real-image splits, DVN computation, and image variations associated with different latent directions.
- 8.2. Direction Variation Naturalness: DVN is averaged across directions and also reported for the top 50 directions, with comparisons against random and coordinate directions.
- 8.2. Direction Variation Naturalness: The DVN classifier uses a LeNet-like model trained with cross-entropy for 100 Adam steps, batch size 32, and learning rate 0.001.
- 8.3. Global effects: Global latent-shift effects are analyzed using FID between real images and shifted BigGAN distributions at different shift magnitudes.
- 8.4. Alternative disentanglement metrics: The paper introduces three quantitative measures and avoids the β-VAE metric because it requires an additional encoder and may be unreliable for large K.
- 8.5. Other details: The background-removal direction can blank BigGAN images lacking explicit foreground objects, including the coral reef example.