Source-linked AI summary
Learning Latent Subspaces in Variational Autoencoders
Jack Klys, Jake Snell, Richard Zemel
TL;DR
The paper addresses the difficulty of interpreting and controlling VAE latent representations, especially for label-correlated features. It proposes CSVAE, which uses mutual-information minimization to place such features in low-dimensional label-associated subspaces. On Toronto Faces and CelebA, CSVAE captures richer intra-class variation and supports more faithful, easily manipulable attribute transfer than baseline models.
Problem
Unsupervised VAE representations are difficult to interpret and control, while isolating labelled features remains difficult.
Method
CSVAE uses mutual-information minimization to separate label-correlated information into low-dimensional latent subspaces associated with each label.
Results
CSVAE captures richer intra-class variation and performs more faithful attribute transfer than baseline models on TFD and CelebA.
Takeaways & Limitations
The learned subspaces can be inspected and independently manipulated for generating or modifying label-associated attributes.
Takeaways & Limitations
The evaluation can reward images with poor quality when a classifier detects the desired attribute, so the authors fix p_j = 1 for some baselines.
Abstract
from arXiv · showhide
Variational autoencoders (VAEs) are widely used deep generative models capable of learning unsupervised latent representations of data. Such representations are often difficult to interpret or control. We consider the problem of unsupervised learning of features correlated to specific labels in a dataset. We propose a VAE-based generative model which we show is capable of extracting features correlated to binary labels in the data and structuring it in a latent subspace which is easy to interpret. Our model, the Conditional Subspace VAE (CSVAE), uses mutual information minimization to learn a low-dimensional latent subspace associated with each label that can easily be inspected and independently manipulated. We demonstrate the utility of the learned representations for attribute manipulation tasks on both the Toronto Face and CelebA datasets.
1 Introduction
The paper targets interpretable, controllable latent representations by learning label-correlated features in a structured subspace. CSVAE uses mutual-information minimization to separate label-related information and demonstrates richer attribute manipulation on face datasets.
- 1 Introduction: CSVAE learns a latent space that separates label-correlated information into a predefined, low-dimensional subspace.The model requires mutual information between z and y to be zero, making the subspace easier to analyze and manipulate.
- 1 Introduction: The model is designed to learn higher-dimensional features correlated with binary labels and represent them in an interpretable, manipulable subspace.These aims support generating or modifying data according to class-specific properties.
- 1 Introduction: CSVAE is evaluated against conditional VAE and information-minimizing VAE baselines on the Toronto Faces and CelebA datasets.The comparison uses quantitative and qualitative evaluation.
- 1 Introduction: CSVAE captures more intra-class variation and learns a richer, easily manipulable latent subspace for attribute style transfer than the baselines.The paper reports these capabilities from quantitative and qualitative evaluations.
2 Related Work
Related work addresses latent-space structuring and controllable data manipulation. CSVAE differs by discovering label-correlated structure and enabling attribute modification from a single example rather than interpolation between existing images.
- Incorporating structure into representations: Prior methods structure representations by using labels as latent variables, batching by labels, or discriminator networks, but do not provide the same label-correlated feature discovery.The paper distinguishes these approaches from its model's interpretable structure.
- Mutual information minimization: Mutual-information minimization is a central design element, while related work applies adversarial prediction or distribution-matching methods to enforce latent independence.The paper notes that the effects of maximum mean discrepancy in this model were not explored.
- Data manipulation and generation: Existing image attribute-transfer models can transfer attributes between source and target images through interpolation, whereas CSVAE modifies an attribute of a single image after training.The supplied passage is truncated after introducing this contrast.
3 Background
VAEs learn latent representations through an approximate posterior, while conditional VAEs additionally condition representation and generation on labels. Information factorization removes label-correlated information from the latent representation but forces reconstruction through a one-dimensional label variable.
- 3.1 Variational Autoencoder (VAE): A VAE uses an approximate posterior qφ(z|x) as an encoder mapping data to a lower-dimensional latent representation.The VAE is trained to maximize a lower bound on marginal log-likelihood.
- 3.2 Conditional VAE (CondVAE): A conditional VAE conditions the latent representation z and generative model on the label variable y, enabling controlled manipulation by changing y before decoding.Its objective includes reconstruction and KL-divergence terms conditioned on y.
- 3.3 Conditional VAE with Information Factorization (CondVAE-info): CondVAE-info adds an adversarial predictor that tries to recover y from z while the encoder minimizes prediction accuracy, removing label-correlated information from z.The decoder must then use the one-dimensional y variable to reconstruct the data.
- 3.3 Conditional VAE with Information Factorization (CondVAE-info): The paper describes this information-factorization design as suboptimal because reconstruction must rely on the one-dimensional label variable.CondVAE-info is compared with the CSVAE architecture in the subsequent model development.
4 Model
CSVAE structures the latent representation as Z × W, assigning label-correlated information to low-dimensional subspaces while discouraging label information in Z. Its jointly trained objective supports interpretable attribute visualization and manipulation.
- Conditional Subspace VAE: CSVAE learns a latent space Z × W that assigns information correlated with each binary label to a corresponding subspace W_i.The model aims to encode all information related to feature i in W_i.
- Objective: The objective combines a variational lower bound with mutual-information minimization between Z and Y, using adversarial predictors to discourage label information in Z.The predictor q_δ(y|z) estimates labels from z while q_φ(z|x) is trained to prevent prediction.
- Empirical Illustration: On the swiss roll, data overlaps in Z but separates by label in W, while CelebA and TFD manipulations produce greater attribute variety than baseline models.The reported examples cover glasses, facial hair, and facial expressions.
- Implementation: CSVAE uses label-conditioned priors and Gaussian approximate posteriors for z and w, with p_θ(x|w,z) decoding the concatenated latent representation.The implementation represents relevant distributions with Gaussian multilayer perceptrons and categorical label prediction.
- Implementation: The experiments set every W_i to R2 with fixed Gaussian priors that place active attributes near one location and inactive attributes at the origin.The authors note that alternate priors are possible but were not explored.
- Attribute Manipulation: Attribute manipulation encodes an input into Z, selects a point in W, and decodes the concatenated vector, allowing independent exploration of attribute styles.When W_i is higher dimensional, the selected point provides additional freedom for manipulation; searches use grids or principal components.
5 Experiments
Experiments show that CSVAE structures label-related variation in latent subspaces and supports diverse, independently manipulated attribute styles across toy and face datasets. Quantitative evaluations report stronger attribute classification and favorable image changes than the comparison models.
- Toy Data: Swiss Roll: In the Swiss Roll experiment, data overlaps in Z but separates by label in W, with label-1 points mapping near the origin.The model uses Z = R2 and W = R2 for this visualization.
- Qualitative Evaluation: CSVAE generates a larger variety of manipulated attributes than baseline models, including diverse glasses, facial hair, smiles, and disgust expressions.The comparisons cover CelebA and the Toronto Faces Dataset.
- Qualitative Evaluation: On the joint CelebA-GlassesFacialHair dataset, CSVAE learns and simultaneously manipulates styles for non-mutually-exclusive binary attributes.Figure 5 shows input, reconstruction, and combinations of representative glasses and facial-hair styles.
- Qualitative Evaluation: CSVAE preserves glasses and facial-hair styles across many identities, indicating that attribute information is disentangled from the Z subspace.This behavior is shown in the style-transfer evaluation across identities.
- Quantitative Evaluation: CSVAE obtains higher attribute and expression classification accuracy than the other models across the evaluated datasets.Table 1 reports the classifier-based evaluation, with CSVAE showing the best performance.
- Quantitative Evaluation: For the image-change metric, CSVAE shows a large improvement over CondVAE and CondVAE-info while making the largest change to the original image.The corresponding values are reported in Table 2.
6 Conclusion
The CSVAE captures intra-class variation through class-associated latent subspaces and supports attribute transfer more faithfully than baseline models on TFD and CelebA.
- CSVAE captures a range of class-associated variations in qualitative experiments on TFD and CelebA.
- CSVAE performs attribute transfer more faithfully than baseline models in quantitative evaluation.
- Future work will extend CSVAE to semi-supervised learning when some attribute labels are missing.
7 Appendix
The appendix documents the shared experimental implementation, weighted loss configuration, hyperparameter selection, and qualitative and quantitative evaluation materials.
- The models use shared architectures and hyperparameters, implemented in PyTorch with convolutional layers and ReLU activations.
- The training setup uses 2300 Adam epochs, minibatches of 64, and scheduled learning-rate decay.
- The objective combines reconstruction and divergence terms with weighted coefficients β1 through β5.
- Hyperparameters and stopping time are selected using quantitative and qualitative validation analyses across models trained for 100, 300, and 500 epochs.
- The appendix presents classifier-accuracy tables for expression transfer and CelebA glasses and facial-hair transfer, alongside qualitative manipulation figures.
- Figure 11 visualizes the model’s test-set output distributions for anger, disgust, fear, happiness, sadness, and surprise.