Source-linked AI summary
Disentangling by Factorising
Hyunjik Kim, Andriy Mnih
TL;DR
The paper addresses unsupervised disentanglement when labels are costly, inconsistent, or incomplete. It proposes FactorVAE, which encourages factorial representations through a Total Correlation penalty, and reports better disentanglement than β-VAE at the same reconstruction quality while introducing a more robust metric. The paper also identifies scope limitations involving independent factors and discrete factors.
Problem
Unsupervised disentanglement is needed because labels are costly, inconsistent, and may omit difficult-to-identify factors, while β-VAE trades reconstruction quality for disentanglement.
Method
FactorVAE encourages the representation distribution to be factorial by penalising Total Correlation and estimating it with a discriminator.
Results
FactorVAE achieves better disentanglement scores than β-VAE on 2D Shapes and 3D Shapes for the same reconstruction quality, and the proposed metric avoids the former metric's failure mode.
Takeaways & Limitations
The paper provides a method and evaluation framework for comparing disentanglement while controlling reconstruction quality.
Takeaways & Limitations
The independent-factor definition is simplistic for data with correlated or hierarchical factors, and jointly modelling discrete and continuous factors needs further research.
Abstract
from arXiv · showhide
We define and address the problem of unsupervised learning of disentangled representations on data generated from independent factors of variation. We propose FactorVAE, a method that disentangles by encouraging the distribution of representations to be factorial and hence independent across the dimensions. We show that it improves upon $β$-VAE by providing a better trade-off between disentanglement and reconstruction quality. Moreover, we highlight the problems of a commonly used disentanglement metric and introduce a new metric that does not suffer from them.
1. Introduction
The paper targets unsupervised learning of disentangled representations because labels are costly or incomplete, while existing β-VAE methods trade reconstruction quality for disentanglement. FactorVAE instead seeks better disentanglement at the same reconstruction quality and introduces a more robust evaluation metric.
- Motivation: Unsupervised disentangling is desirable because human labels are costly, inconsistent, and may omit factors that are difficult to identify.
- Prior work: β-VAE is an effective unsupervised method, but improving disentanglement requires sacrificing reconstruction quality relative to VAE.
- Contributions: FactorVAE encourages the representation distribution to be factorial, aiming for higher disentanglement than β-VAE at the same reconstruction quality.
- Contributions: The paper identifies weaknesses in a commonly used disentanglement metric and proposes a new metric that addresses them.
- Contributions: The study also quantitatively compares FactorVAE and β-VAE with an InfoGAN-based WGAN-GP counterpart.
2. Trade-off between Disentanglement and Reconstruction in β-VAE
The β-VAE objective creates a disentanglement–reconstruction trade-off because its KL penalty combines independence-promoting regularization with a penalty on information stored in the representation. Increasing β improves disentanglement but can degrade reconstruction enough to prevent recovery of the true factors.
- VAE framework: The VAE models observations with a real-valued latent vector and uses a factorised standard Gaussian prior.
- VAE framework: A disentangled representation assigns each latent dimension to one independent underlying factor, motivating a factorial aggregate representation distribution.
- β-VAE objective: For β ≥ 1, the β-VAE objective is a variational lower bound whose terms represent reconstruction error and a complexity penalty.
- β-VAE objective: The KL term separates into mutual information and a marginal-distribution divergence, with the latter encouraging independence in latent dimensions.
- Trade-off: Increasing β penalises both terms more, improving disentanglement while reducing reconstruction quality.
- Trade-off: Severe information loss in the latents can make recovering the true factors impossible, so the best-disentangling β can reconstruct worse than a VAE.
3. Total Correlation Penalty and FactorVAE
FactorVAE augments the VAE objective with a direct penalty on dependence among code dimensions. It estimates this penalty through a discriminator distinguishing the aggregate code distribution from the product of its marginals.
- Motivation: Penalising mutual information more than a VAE does is not necessarily needed for disentangling, motivating a direct independence penalty instead.
- FactorVAE objective: FactorVAE adds a term that encourages the code distribution to be factorial, targeting independence without directly increasing the mutual-information penalty.
- Total Correlation Penalty: Total Correlation is KL(q(z)||∏_j q(z_j)), measuring dependence among the latent variables.
- Estimating Total Correlation: The product of marginals is approximated by randomly permuting each latent dimension across a sufficiently large batch.
- Discriminator optimization: A discriminator distinguishes samples from q(z) and the product of marginals to estimate the density ratio needed for Total Correlation.
- Caveat: Low Total Correlation alone is insufficient for meaningful disentangling because q(z)=p(z) can have TC=0 while carrying no information about the data.
4. A New Metric for Disentanglement
The paper identifies weaknesses in the existing supervised disentanglement metric and proposes a deterministic alternative based on normalized conditional variances. The new metric avoids the earlier metric’s failure mode and optimization sensitivity.
- The paper defines disentanglement as changing one representation dimension corresponding to exactly one factor of variation.
- The paper notes that its definition is best suited to synthetic data with independent factors and that reliable unsupervised evaluation remains an open direction.
- The Higgins et al. metric trains a linear classifier to infer which factor was fixed from pairwise representation differences.
- 100% accuracy can occur when only K −1 of K factors are disentangled, because the classifier detects that the remaining factor has all other values non-zero.
- The proposed metric normalizes representation dimensions, computes conditional variances, and classifies each factor by the dimension with the lowest variance.
- Because its inputs and outputs are discrete, the new metric uses a deterministic majority-vote classifier without optimization hyperparameters.
5. Related Work
Related work includes unsupervised and supervised approaches that disentangle representations through predictability penalties, information-theoretic objectives, and divergence-based methods. The paper also situates FactorVAE among methods using discriminators and connects disentanglement to invariance and equivariance.
- Adversarial Autoencoder uses a density-ratio trick to match the aggregated posterior to a prior, but its objective is not a lower bound on marginal log likelihood.
- Earlier unsupervised methods penalized predictability between latent dimensions or used Boltzmann Machines to disentangle factors.
- Several methods quantify disentanglement by predicting ground-truth factors from latent codes, using linear maps or Lasso regression weights.
- Invariance removes changes caused by task-irrelevant nuisance factors, whereas equivariance requires stable and predictable changes under altered factors.
- A later work proposed a minibatch-based alternative for estimating Total Correlation and introduced an information-theoretic disentanglement metric.
6. Experiments
Experiments compare FactorVAE with β-VAE and InfoWGAN-GP across datasets, using reconstruction quality, disentanglement metrics, latent traversals, and Total Correlation diagnostics. FactorVAE generally improves the disentanglement–reconstruction trade-off, while performance remains limited for some discrete factors and InfoWGAN-GP settings.
- 2D Shapes: FactorVAE’s best 2D Shapes score is 0.889, compared with 0.814 for β-VAE.These are the best-scoring models shown in the latent-traversal comparison.
- 2D Shapes: FactorVAE achieves higher disentanglement than β-VAE at similar reconstruction quality on 2D Shapes.At reconstruction error around 45, FactorVAE reaches about 0.82 versus about 0.73 for β-VAE.
- 2D Shapes: Both models capture position and scale more reliably than orientation and shape, with neither robustly capturing the discrete shape factor on 2D Shapes.β-VAE struggles especially with orientation and shape.
- Total Correlation: The discriminator underestimates true Total Correlation, but true Total Correlation decreases during training and with larger γ.The authors report that the resulting gradients are sufficient to encourage independence in the code distribution.
- InfoWGAN-GP: InfoWGAN-GP obtains low disentanglement scores on 2D Shapes and mainly learns scale while placing positional information in the discrete code.Some latent traversals produce blank reconstructions, indicating poor generalization across the prior domain.
- 3D Shapes: On 3D Shapes, FactorVAE has lower reconstruction error than β-VAE despite similar top mean disentanglement scores: 3515 versus 3570.The reported settings are γ = 36 for FactorVAE and β = 24 for β-VAE.
7. Conclusion and Discussion
FactorVAE improves disentanglement over β-VAE at matched reconstruction quality and introduces a simpler, hyperparameter-free metric. The paper also identifies scope limitations and proposes future work on discrete and mixed latent factors.
- The proposed metric is conceptually simpler, hyperparameter-free, and avoids failure modes identified in the Higgins et al. metric.
- Low Total Correlation is necessary but not sufficient for disentangling independent factors, because collapsed latent dimensions can yield TC = 0 without disentanglement.
- The metric requires generating samples with one factor fixed and is unsuitable when factors are nonindependent.
- Future work targets discrete latent variables and combinations of discrete and continuous factors.
B. Details for the Disentanglement Metrics
The paper analyzes sensitivity and failure modes of disentanglement metrics, then specifies a majority-vote alternative and explains why batch density estimates are unreliable for Total Correlation.
- Metric sensitivity: The Higgins et al. metric is highly sensitive to Adagrad iterations and continually improves with more iterations.The analysis suggests using less noisy multiclass logistic-regression solvers than gradient-descent methods.
- Proposed metric: The proposed metric prunes latent dimensions collapsed to the prior before classification and is much less sensitive to hyperparameters.The reported settings are L = 100 and 800 majority-vote evaluations.
- Proposed metric: The majority-vote classifier assigns each latent dimension the factor with the largest count among observed dimension-factor pairs.For dimension j, C(j) = arg max_k V_jk.
- Proposed metric: Random-classifier accuracy is 1/K and does not depend on latent dimensionality D.
- Batch TC estimation: Batch estimates of q(z) can produce negative Total Correlation estimates, even with batch size 10,000, so this approach does not work.The problem arises because finite batches poorly approximate q(z) in higher dimensions as posterior variances shrink.
E. Log Marginal Likelihood and Samples
On 3D Shapes, FactorVAE obtains a better log marginal likelihood than β-VAE despite producing less coherent-looking samples, illustrating a trade-off between these evaluation views.
- The reported likelihood comparison uses the IWAE bound with 5000 particles because exact log marginal likelihood is intractable.
- On 3D Shapes, FactorVAE has better log marginal likelihood than β-VAE despite less coherent-looking samples.β-VAE with β = 32 has -3534, while FactorVAE with γ = 6 has -3520.
- For similarly high disentanglement, the paper recommends choosing the model with the lowest β or γ.
F. Losses and Experiments for other related Methods
The paper compares FactorVAE with related divergence- and mutual-information-based methods, finding lower disentanglement for AAE and a FactorVAE variant, while noting comparison constraints for other methods.
- AAE and FactorVAE variant: FactorVAE samples from the latent marginals that differ clearly from the prior, despite the prior-based alternative objective.
- AAE and FactorVAE variant: AAE and a FactorVAE variant achieve somewhat lower disentanglement than FactorVAE on 2D Shapes.
- AAE and FactorVAE variant: The paper attributes this difference to directly encouraging q(z) to be factorised rather than matching an arbitrarily chosen factorised prior p(z) = N(0, I).
- Comparison scope: The paper could not reproduce and compare Information Dropout and DIP-VAE because experimental details and publicly available code were insufficient.
- InfoGAN and InfoWGAN-GP: InfoGAN training has known instability issues, motivating evaluation of the more stable WGAN-GP objective in InfoWGAN-GP.
H. Empirical Study of InfoGAN and InfoWGAN-GP
Experiments found that InfoGAN suffers from discriminator degeneracy, inconsistent factor assignment, noise entanglement, and sensitivity to the number of latent codes. InfoWGAN-GP stabilizes training but leaves these disentanglement weaknesses largely unresolved.
- Training stability: InfoGAN runs eventually produced degenerate discriminators that classified all inputs as real, making early stopping crucial for the reported results.Reducing the discriminator learning rate delayed but did not prevent this behavior.
- Disentanglement behavior: Latent traversals contradicted the intended code semantics: continuous codes changed digit class, while categorical traversals contained overlapping digits.The same inconsistency was observed for InfoWGAN-GP.
- Disentanglement behavior: Varying noise latents often changed digit class, indicating that semantically meaningful information was not cleanly separated from incompressible noise.This was assessed by reconstructing an image while fixing its inferred code and varying the noise latent.
- Hyperparameter sensitivity: The number of latent codes is an important InfoGAN hyperparameter: adding a third continuous code caused more digit-class information to enter continuous traversals.The experiments used architectures with two or four continuous codes and categorical and noise variables.
- Hyperparameter sensitivity: Using 20 rather than 2 categorical categories reduced semantically meaningful information in noise variables, but digit-class information remained in continuous latents.The categorical-code and reconstruction comparisons are shown in Figures 28–31.
- InfoWGAN-GP: InfoWGAN-GP solved the training degeneracy and improved stability, but did not resolve the observed disentanglement problems.These remaining weaknesses include code-count sensitivity, semantic information in noise variables, and incomplete generalization across p(z).
- InfoWGAN-GP: A larger InfoWGAN-GP architecture did not improve disentanglement scores, although its latent traversals looked slightly more realistic.The larger architecture was evaluated on 2D Shapes using ten random seeds per hyperparameter setting.
I. Further Experimental Results
Further experiments examine FactorVAE’s discriminator behavior and latent marginals. They show that increasing γ lowers discriminator accuracy and that FactorVAE can factorize q(z) without forcing each marginal to match the Gaussian prior.
- FactorVAE discriminator: Higher γ values led to lower FactorVAE discriminator accuracy, consistent with bringing q(z) and the product of its marginals closer together.The discriminator distinguishes samples from these two distributions.
- Latent marginals: FactorVAE marginals q(z_j) were substantially different from the Gaussian prior across the tested β and γ settings.The comparison used histograms of q(z_j) with N(0, 1) overlaid.
- Latent marginals: The results indicate that FactorVAE can focus on factorizing q(z) rather than pushing its individual marginals toward the prior.This distinction was offered as a reason for differing results between FactorVAE and an alternative objective.