PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangling by Factorising

Hyunjik Kim, Andriy Mnih

arXiv:1802.05983v3stat.MLcs.LG

TL;DR

The paper addresses unsupervised disentanglement when labels are costly, inconsistent, or incomplete. It proposes FactorVAE, which encourages factorial representations through a Total Correlation penalty, and reports better disentanglement than β-VAE at the same reconstruction quality while introducing a more robust metric. The paper also identifies scope limitations involving independent factors and discrete factors.

  • Problem

    Unsupervised disentanglement is needed because labels are costly, inconsistent, and may omit difficult-to-identify factors, while β-VAE trades reconstruction quality for disentanglement.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    FactorVAE encourages the representation distribution to be factorial by penalising Total Correlation and estimating it with a discriminator.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    FactorVAE achieves better disentanglement scores than β-VAE on 2D Shapes and 3D Shapes for the same reconstruction quality, and the proposed metric avoids the former metric's failure mode.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The paper provides a method and evaluation framework for comparing disentanglement while controlling reconstruction quality.

    Source: §1 · p. 2, §7 · p. 8

    The independent-factor definition is simplistic for data with correlated or hierarchical factors, and jointly modelling discrete and continuous factors needs further research.

    Source: §4 · p. 4, §6 · p. 6

Abstract

from arXiv · show

We define and address the problem of unsupervised learning of disentangled representations on data generated from independent factors of variation. We propose FactorVAE, a method that disentangles by encouraging the distribution of representations to be factorial and hence independent across the dimensions. We show that it improves upon $β$-VAE by providing a better trade-off between disentanglement and reconstruction quality. Moreover, we highlight the problems of a commonly used disentanglement metric and introduce a new metric that does not suffer from them.

1. Introduction

The paper targets unsupervised learning of disentangled representations because labels are costly or incomplete, while existing β-VAE methods trade reconstruction quality for disentanglement. FactorVAE instead seeks better disentanglement at the same reconstruction quality and introduces a more robust evaluation metric.

Motivation

  • Unsupervised disentangling is desirable because human labels are costly, inconsistent, and may omit factors that are difficult to identify.Source: §1 · p. 1

Prior work

  • β-VAE is an effective unsupervised method, but improving disentanglement requires sacrificing reconstruction quality relative to VAE.Source: §1 · p. 1

Contributions

  • FactorVAE encourages the representation distribution to be factorial, aiming for higher disentanglement than β-VAE at the same reconstruction quality.Source: §1 · p. 2
  • The paper identifies weaknesses in a commonly used disentanglement metric and proposes a new metric that addresses them.Source: §1 · p. 2
  • The study also quantitatively compares FactorVAE and β-VAE with an InfoGAN-based WGAN-GP counterpart.Source: §1 · p. 2

2. Trade-off between Disentanglement and Reconstruction in β-VAE

The β-VAE objective creates a disentanglement–reconstruction trade-off because its KL penalty combines independence-promoting regularization with a penalty on information stored in the representation. Increasing β improves disentanglement but can degrade reconstruction enough to prevent recovery of the true factors.

VAE framework

  • The VAE models observations with a real-valued latent vector and uses a factorised standard Gaussian prior.Source: §2 · p. 2
  • A disentangled representation assigns each latent dimension to one independent underlying factor, motivating a factorial aggregate representation distribution.Source: §2 · p. 2

β-VAE objective

  • For β ≥ 1, the β-VAE objective is a variational lower bound whose terms represent reconstruction error and a complexity penalty.Source: §2 · p. 2
  • The KL term separates into mutual information and a marginal-distribution divergence, with the latter encouraging independence in latent dimensions.Source: §2 · p. 2

Trade-off

  • Increasing β penalises both terms more, improving disentanglement while reducing reconstruction quality.Source: §2 · p. 2
  • Severe information loss in the latents can make recovering the true factors impossible, so the best-disentangling β can reconstruct worse than a VAE.Source: §2 · p. 2

3. Total Correlation Penalty and FactorVAE

FactorVAE augments the VAE objective with a direct penalty on dependence among code dimensions. It estimates this penalty through a discriminator distinguishing the aggregate code distribution from the product of its marginals.

Motivation

  • Penalising mutual information more than a VAE does is not necessarily needed for disentangling, motivating a direct independence penalty instead.Source: §3 · pp. 2–3

FactorVAE objective

  • FactorVAE adds a term that encourages the code distribution to be factorial, targeting independence without directly increasing the mutual-information penalty.Source: §3 · pp. 2–3

Total Correlation Penalty

  • Total Correlation is KL(q(z)||∏_j q(z_j)), measuring dependence among the latent variables.Source: §3 · p. 3

Estimating Total Correlation

  • The product of marginals is approximated by randomly permuting each latent dimension across a sufficiently large batch.Source: §3 · p. 3

Discriminator optimization

  • A discriminator distinguishes samples from q(z) and the product of marginals to estimate the density ratio needed for Total Correlation.Source: §3 · p. 3

Caveat

  • Low Total Correlation alone is insufficient for meaningful disentangling because q(z)=p(z) can have TC=0 while carrying no information about the data.Source: §3 · p. 3

4. A New Metric for Disentanglement

The paper identifies weaknesses in the existing supervised disentanglement metric and proposes a deterministic alternative based on normalized conditional variances. The new metric avoids the earlier metric’s failure mode and optimization sensitivity.

  • The paper defines disentanglement as changing one representation dimension corresponding to exactly one factor of variation.Source: §4 · p. 3
  • The paper notes that its definition is best suited to synthetic data with independent factors and that reliable unsupervised evaluation remains an open direction.Source: §4 · pp. 4–5
  • The Higgins et al. metric trains a linear classifier to infer which factor was fixed from pairwise representation differences.Source: §4 · p. 4
  • 100% accuracy can occur when only K −1 of K factors are disentangled, because the classifier detects that the remaining factor has all other values non-zero.Source: §4 · p. 4, Figure 3 · p. 4
  • The proposed metric normalizes representation dimensions, computes conditional variances, and classifies each factor by the dimension with the lowest variance.Source: §4 · p. 4
  • Because its inputs and outputs are discrete, the new metric uses a deterministic majority-vote classifier without optimization hyperparameters.Source: §4 · p. 4

5. Related Work

Related work includes unsupervised and supervised approaches that disentangle representations through predictability penalties, information-theoretic objectives, and divergence-based methods. The paper also situates FactorVAE among methods using discriminators and connects disentanglement to invariance and equivariance.

  • Adversarial Autoencoder uses a density-ratio trick to match the aggregated posterior to a prior, but its objective is not a lower bound on marginal log likelihood.Source: §5 · p. 5
  • Earlier unsupervised methods penalized predictability between latent dimensions or used Boltzmann Machines to disentangle factors.Source: §5 · p. 5
  • Several methods quantify disentanglement by predicting ground-truth factors from latent codes, using linear maps or Lasso regression weights.Source: §5 · p. 5
  • Invariance removes changes caused by task-irrelevant nuisance factors, whereas equivariance requires stable and predictable changes under altered factors.Source: §5 · p. 5
  • A later work proposed a minibatch-based alternative for estimating Total Correlation and introduced an information-theoretic disentanglement metric.Source: §5 · p. 5

6. Experiments

Experiments compare FactorVAE with β-VAE and InfoWGAN-GP across datasets, using reconstruction quality, disentanglement metrics, latent traversals, and Total Correlation diagnostics. FactorVAE generally improves the disentanglement–reconstruction trade-off, while performance remains limited for some discrete factors and InfoWGAN-GP settings.

2D Shapes

  • FactorVAE’s best 2D Shapes score is 0.889, compared with 0.814 for β-VAE.These are the best-scoring models shown in the latent-traversal comparison.Source: Figure 6 · p. 6
  • FactorVAE achieves higher disentanglement than β-VAE at similar reconstruction quality on 2D Shapes.At reconstruction error around 45, FactorVAE reaches about 0.82 versus about 0.73 for β-VAE.Source: §6 · p. 6
  • Both models capture position and scale more reliably than orientation and shape, with neither robustly capturing the discrete shape factor on 2D Shapes.β-VAE struggles especially with orientation and shape.Source: §6 · p. 6

Total Correlation

  • The discriminator underestimates true Total Correlation, but true Total Correlation decreases during training and with larger γ.The authors report that the resulting gradients are sufficient to encourage independence in the code distribution.Source: §6 · p. 6

InfoWGAN-GP

  • InfoWGAN-GP obtains low disentanglement scores on 2D Shapes and mainly learns scale while placing positional information in the discrete code.Some latent traversals produce blank reconstructions, indicating poor generalization across the prior domain.Source: §6 · p. 7

3D Shapes

  • On 3D Shapes, FactorVAE has lower reconstruction error than β-VAE despite similar top mean disentanglement scores: 3515 versus 3570.The reported settings are γ = 36 for FactorVAE and β = 24 for β-VAE.Source: §6 · p. 7

7. Conclusion and Discussion

FactorVAE improves disentanglement over β-VAE at matched reconstruction quality and introduces a simpler, hyperparameter-free metric. The paper also identifies scope limitations and proposes future work on discrete and mixed latent factors.

  • The proposed metric is conceptually simpler, hyperparameter-free, and avoids failure modes identified in the Higgins et al. metric.Source: §7 · p. 8
  • Low Total Correlation is necessary but not sufficient for disentangling independent factors, because collapsed latent dimensions can yield TC = 0 without disentanglement.Source: §7 · p. 8
  • The metric requires generating samples with one factor fixed and is unsuitable when factors are nonindependent.Source: §7 · p. 8
  • Future work targets discrete latent variables and combinations of discrete and continuous factors.Source: §7 · pp. 8–9

B. Details for the Disentanglement Metrics

The paper analyzes sensitivity and failure modes of disentanglement metrics, then specifies a majority-vote alternative and explains why batch density estimates are unreliable for Total Correlation.

Metric sensitivity

Proposed metric

Batch TC estimation

  • Batch estimates of q(z) can produce negative Total Correlation estimates, even with batch size 10,000, so this approach does not work.The problem arises because finite batches poorly approximate q(z) in higher dimensions as posterior variances shrink.Source: D. Using a Batch Estimate of q(z) for… · p. 13

E. Log Marginal Likelihood and Samples

On 3D Shapes, FactorVAE obtains a better log marginal likelihood than β-VAE despite producing less coherent-looking samples, illustrating a trade-off between these evaluation views.

F. Losses and Experiments for other related Methods

The paper compares FactorVAE with related divergence- and mutual-information-based methods, finding lower disentanglement for AAE and a FactorVAE variant, while noting comparison constraints for other methods.

AAE and FactorVAE variant

Comparison scope

InfoGAN and InfoWGAN-GP

H. Empirical Study of InfoGAN and InfoWGAN-GP

Experiments found that InfoGAN suffers from discriminator degeneracy, inconsistent factor assignment, noise entanglement, and sensitivity to the number of latent codes. InfoWGAN-GP stabilizes training but leaves these disentanglement weaknesses largely unresolved.

Training stability

Disentanglement behavior

Hyperparameter sensitivity

InfoWGAN-GP

I. Further Experimental Results

Further experiments examine FactorVAE’s discriminator behavior and latent marginals. They show that increasing γ lowers discriminator accuracy and that FactorVAE can factorize q(z) without forcing each marginal to match the Gaussian prior.

FactorVAE discriminator

  • Higher γ values led to lower FactorVAE discriminator accuracy, consistent with bringing q(z) and the product of its marginals closer together.The discriminator distinguishes samples from these two distributions.Source: §I · p. 17, Figure 37 · p. 18

Latent marginals

  • FactorVAE marginals q(z_j) were substantially different from the Gaussian prior across the tested β and γ settings.The comparison used histograms of q(z_j) with N(0, 1) overlaid.Source: §I · p. 18, Figure 40 · p. 19
  • The results indicate that FactorVAE can focus on factorizing q(z) rather than pushing its individual marginals toward the prior.This distinction was offered as a reason for differing results between FactorVAE and an alternative objective.Source: §I · p. 18

Related papers

Loading 1802.05983v3…