PaperlayerView exact revision on arXiv

Source-linked AI summary

Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, Olivier Bachem

arXiv:1811.12359v4cs.LGcs.AIstat.ML

TL;DR

The paper examines whether unsupervised learning can recover disentangled representations and challenges assumptions about their learnability and usefulness. It combines an impossibility result with a large-scale study, finding that supervision affects model selection and that increased disentanglement does not reduce downstream sample complexity.

  • Problem

    Disentangled representations are expected to capture explanatory factors in compact, interpretable, task-independent forms useful for downstream, transfer, and few-shot learning.

  • Method

    The paper proves an impossibility result and evaluates six methods, six metrics, and seven data sets in a large-scale empirical study.

  • Results

    Random seeds and hyperparameters seem to matter more than model choice, tuning seems to require supervision, and increased disentanglement did not imply decreased downstream sample complexity.

  • Takeaways & Limitations

    Future work should make inductive biases and supervision explicit, demonstrate concrete benefits, and use sound, reproducible experiments across diverse data sets.

  • Takeaways & Limitations

    For arbitrary generative models, unsupervised disentanglement is impossible without inductive biases on both the models and the data sets.

Abstract

from arXiv · show

The key idea behind the unsupervised learning of disentangled representations is that real-world data is generated by a few explanatory factors of variation which can be recovered by unsupervised learning algorithms. In this paper, we provide a sober look at recent progress in the field and challenge some common assumptions. We first theoretically show that the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data. Then, we train more than 12000 models covering most prominent methods and evaluation metrics in a reproducible large-scale experimental study on seven different data sets. We observe that while the different methods successfully enforce properties ``encouraged'' by the corresponding losses, well-disentangled models seemingly cannot be identified without supervision. Furthermore, increased disentanglement does not seem to lead to a decreased sample complexity of learning for downstream tasks. Our results suggest that future work on disentanglement learning should be explicit about the role of inductive biases and (implicit) supervision, investigate concrete benefits of enforcing disentanglement of the learned representations, and consider a reproducible experimental setup covering several data sets.

1. Introduction

The paper challenges assumptions behind unsupervised disentanglement learning by combining theoretical analysis with a reproducible large-scale study. It argues that inductive biases and supervision remain central, while the practical benefits and reliable identification of disentangled models are unresolved.

  • Motivation: Disentangled representations are intended to separate informative factors of variation while retaining compact, interpretable information useful across tasks.A change in one underlying factor should change one learned representation factor.
  • Contributions: Unsupervised disentanglement learning is theoretically impossible without inductive biases on both learning approaches and data sets.The paper presents this as a central theoretical contribution.
  • Contributions: More than 12 000 models were trained across six methods, six disentanglement measures, and seven data sets using a reproducible experimental protocol.The authors implemented the methods and metrics from scratch.
  • Empirical findings: The considered methods reduced correlations in sampled aggregated posteriors, but representation dimensions defined by posterior means remained correlated.This contrasts a loss-encouraged property with the measured representation structure.
  • Empirical findings: Random seeds and hyperparameters seemed more important than model choice, and well-disentangled models could not be reliably identified without ground-truth labels.This remained true even when good hyperparameter values were transferred across data sets.
  • Empirical findings: The study found no evidence that increased disentanglement decreases downstream-task sample complexity.The conclusion is limited to the considered models, data sets, and tasks.

2. Other related work

Related work connects unsupervised disentanglement to independent component analysis and to settings that add supervision or inductive structure. This paper instead studies the setting where variation factors are entirely unobserved.

  • Independent component analysis: Nonlinear independent component analysis seeks to recover independent signal components, but its sources generally cannot be identified without additional assumptions.The lack of nonlinear identifiability motivated alternative approaches and more explicit supervision.
  • Study setting: Prior disentanglement research has considered weak or explicit supervision and inductive biases, whereas this study focuses on observing only samples from P(x).The paper emphasizes the fully unsupervised, static setting.

3. Impossibility result

The impossibility result shows that observations alone cannot distinguish disentangled from completely entangled latent explanations. Therefore, successful unsupervised recovery requires inductive biases on both models and data, with the formal theorem applying to factorized priors.

  • Impossibility result: Without inductive biases on models and data sets, unsupervised disentanglement is fundamentally impossible for arbitrary generative models.The paper states this as the result of Theorem 1.
  • Theorem 1: Theorem 1 constructs an infinite family of bijections that preserve the latent distribution while producing completely entangled coordinates.The construction applies to distributions with a factorized density.
  • Implication: A representation perfectly disentangled for one factorized latent model can be completely entangled under an observationally equivalent reparameterized model.Changes in one original latent dimension can affect all transformed dimensions.
  • Implication: Because unsupervised methods access only observations x, they cannot distinguish the equivalent generative models and must be entangled with respect to at least one.Both models induce the same marginal distribution of observations.
  • Scope: The theorem applies to factorized priors, although the authors expect a similar result may extend to non-factorizing priors.This is an explicit scope qualification.
  • Relation to prior work: The result explicitly constructs completely entangled latent spaces with the same observed marginal distribution, extending an identifiability concern beyond rotation-invariant distributions.The authors relate this to causality and nonlinear ICA.

4. Experimental design

The study compares six regularized VAE-based methods across seven data sets using six disentanglement metrics under a controlled, reproducible protocol. It standardizes architecture and optimization choices while varying regularization strength and random seeds.

  • Considered methods: Six methods augment the VAE loss with regularizers that alter bottleneck capacity or encourage learning distinct factors of variation.β-VAE constrains bottleneck capacity, while AnnealedVAE progressively increases it so the encoder can focus on one factor at a time.
  • Considered metrics: Six metrics evaluate disentanglement through classifier accuracy, mutual-information gaps, modularity, importance entropy, or prediction-error differences.The metrics include BetaVAE, FactorVAE, MIG, Modularity, DCI Disentanglement, and SAP.
  • Data sets: Seven data sets cover both deterministic and stochastic observation processes given the underlying factors of variation.The deterministic sets are dSprites, Cars3D, SmallNORB, and Shapes3D; Color-dSprites, Noisy-dSprites, and Scream-dSprites introduce stochasticity.
  • Inductive biases: All methods use the same convolutional architecture, optimizer, optimizer hyperparameters, and batch size to separate regularization effects from other inductive biases.Each method also uses a Gaussian encoder whose latent means and log variances are parameterized by a deep neural network.
  • Hyperparameters: Six regularization strengths are tested for every method, using broad ranges intended to support different data sets rather than optimize known dataset-specific values.The ranges are partly based on values prescribed in prior literature, including author-recommended hyperparameters.
  • Reproducibility: Each method is run on every data set with 50 random seeds and evaluated on the selected metrics under a setup fixed in advance.The paper discusses the experimental setup, its limitations, and implementation differences from prior work in the appendices.

5. Key experimental results

Across the large-scale experiments, regularization often enforces uncorrelated sampled posteriors without making mean representations uncorrelated, while metric agreement and model-selection reliability vary across datasets. Random seeds and hyperparameters substantially influence disentanglement scores, and higher disentanglement does not reliably improve downstream sample efficiency.

  • Aggregated posterior and representation: Regularization generally decreases total correlation for sampled representations but increases it for mean representations on Color-dSprites.DIP-VAE-I is an exception because its objective directly optimizes the mean covariance toward a diagonal matrix.
  • Metric agreement: All disentanglement metrics except Modularity appear correlated, but correlation strength changes across data sets.BetaVAE and FactorVAE scores, and MIG and DCI Disentanglement, capture particularly similar notions.
  • Models and hyperparameters: Random seeds and regularization strength substantially influence disentanglement scores, while objective function choice appears less influential.On average, objective function alone explains 37% of score variance, and score ranges for different methods heavily overlap.
  • Model selection: Unsupervised scores do not yield a clear model-selection strategy, and hyperparameter transfer works consistently only between dSprites and Color-dSprites.The authors conclude that unsupervised model selection remains unresolved because good and bad random seeds cannot be distinguished on the target task.
  • Downstream usefulness: Observed downstream correlations are concentrated on dSprites and some Shapes3D variations, and may partly reflect representation informativeness rather than disentanglement.The authors state that disentanglement itself may not explain the observed correlations.
  • Downstream usefulness: Higher disentanglement scores do not reliably lead to higher sample efficiency for downstream learning.The statistical efficiency score is average accuracy with 100 samples divided by average accuracy with 10,000 samples.
  • Downstream usefulness: The downstream conclusions are limited because the evaluated models fail to reliably produce disentangled representations and other notions of usefulness were not tested.The paper specifically mentions semi-supervised or fully supervised models, interpretability, and fairness as omitted alternatives or criteria.

6. Conclusions

The paper concludes that unsupervised disentanglement requires inductive biases and supervision, while its practical benefits remain unvalidated. It recommends stronger reproducibility and broader evaluation.

  • Unsupervised disentanglement is fundamentally impossible without inductive biases, and supervision appears crucial for selecting hyperparameters.The authors argue that future work should make inductive biases and implicit or explicit supervision more explicit.
  • The study found no evidence that increased disentanglement reduces downstream-task sample complexity.This conclusion is limited to the studied setting and downstream task.
  • Future work should demonstrate concrete benefits of disentangled representations, including interpretability, fairness, and interactive applications.The paper also identifies independent causal mechanisms and causal inference as a possible framework for incorporating inductive biases and generalization.
  • Sound, robust, reproducible experiments across diverse data sets are needed because restricted method, metric, or data-set selections can produce spurious conclusions.

A. Proof of Theorem 1

The proof constructs infinitely many alternative latent transformations that preserve the latent distribution while changing coordinate structure, establishing non-identifiability without suitable inductive biases.

  • The proof begins with a latent variable of dimensionality d > 1 and maps its coordinates to a d-dimensional uniform distribution using marginal cumulative distributions.The map g is bijective and has zero cross-coordinate derivatives.
  • A coordinatewise bijection then maps the uniform distribution to a d-dimensional standard normal distribution.The transformation h is bijective and preserves coordinatewise structure.
  • An arbitrary orthogonal matrix with nonzero entries mixes the transformed coordinates while preserving the multivariate standard normal distribution.The covariance remains the identity because the matrix is orthogonal.
  • Applying the inverse coordinatewise transformations produces a bijection with the same marginal distribution as the original latent variable.The construction therefore yields alternative latent representations consistent with the same prior distribution.
  • Because the orthogonal matrix is arbitrary, the construction gives an infinite family of such functions.

C. Implementation of metrics

The implementation evaluates disentanglement using multiple metrics on expected or sampled representations, alongside total-correlation and downstream-task procedures.

  • The metrics generally use expected representations, while total correlation is additionally evaluated on sampled representations.
  • The BetaVAE metric predicts the fixed ground-truth factor from coordinate-wise absolute representation differences between two mini-batches.
  • The FactorVAE metric uses majority voting based on the least-variance representation dimension after excluding collapsed dimensions.
  • The Mutual Information Gap compares the highest and second-highest mutual information between each ground-truth factor and representation dimensions.The relevant dimension is selected by maximizing mutual information with the factor.
  • Modularity measures whether each representation dimension depends on at most one factor, while DCI also evaluates completeness and informativeness.
  • The fitted-Gaussian total correlation uses the mean and covariance of representations; a high score signals non-factorization, but a low score does not establish factorization.

D. Experimental conditions and guiding principles.

The experiments use known ground-truth generative models while training remains unsupervised, enabling controlled evaluation of learned representations against latent factors.

  • The protocol samples factors from P(z) and observations from P(x|z) using a well-defined ground-truth generative model.
  • During training, models observe only samples of x after marginalizing over the latent-factor distribution.
  • After training, the representation r(x) is obtained from the encoder by taking or sampling its output representation.

E. Limitations of our study.

The study’s conclusions are constrained by deliberate choices about the data, architectures, hyperparameters, and implementation procedures. These choices target community-standard inductive biases but limit generalization.

  • Interpretive boundary: The authors designed these choices to capture state-of-the-art inductive bias while acknowledging that they constrain the study’s conclusions.The limitation applies to the experimental design rather than to a claim that all possible implementations were evaluated.
  • Dataset scope: The experiments focus heavily on synthetic images and omit other modalities, confounding variables, and nonuniform or continuous factor settings.The datasets contain few independent discrete factors and assume access to a data-generative process with uniformly distributed factors.
  • Method scope: The methods use only convolutional architectures and do not test alternative architectures, activations, reconstruction losses, or layer counts.The study also fixes latent-space size, optimizers, batch sizes, and most hyperparameters rather than varying them.
  • Implementation scope: Implementation differences from original papers introduce additional uncertainty into comparisons across methods and metrics.The authors implemented methods and metrics from scratch because few official open-source implementations were available.

F. Differences with previous implementations.

The study standardizes architecture and training choices for fairness, while documenting departures from previous implementations in metrics and model settings. These changes reflect the authors’ assumed ground-truth structure and comparison goals.

  • Model implementations: A single architecture, batch size, and optimizer are used across methods to enable fair comparisons, even when this differs from original papers.For DIP-VAE and β-TCVAE, the batch size is changed to 64 instead of 400 and 2048, respectively.
  • Metric implementations: The metrics treat all ground-truth factors as discrete variables, using classification rather than regression for SAP and the Eastwood–Williams disentanglement score.This matches the assumed ground-truth model used by the study’s datasets.
  • Experimental configuration: The main experiment uses a shared encoder–decoder architecture and a one-hyperparameter sweep per model.The supplied tables identify the common architecture and model-specific hyperparameters, but do not provide their individual values here.
  • Representation choice: The study evaluates metrics on the mean representation rather than Gaussian samples, because the mean is the representation used in the paper.Sampling and dataset sizes are fixed at large values across metrics.

G. Main experiment hyperparameters

The main experiment fixes nearly all training choices and varies one model-specific hyperparameter, allowing comparisons centered on regularization while holding other settings constant.

  • Hyperparameter sweep: Each model varies only one hyperparameter, while model-specific settings are listed separately.All hyperparameters reported as single values remain fixed and are selected from the literature.
  • Controlled settings: The study fixes the common architecture and remaining hyperparameters across models, isolating the selected sweep from other training choices.The passage refers to the common architecture and fixed settings in Tables 2 and 4a.

H. Data sets and preprocessing

The experiments examine reconstruction, representation correlations, metric agreement, and model-selection reliability across datasets and methods. Results distinguish properties enforced by losses from properties of the mean representation and show strong dependence on randomness and hyperparameters.

  • Reconstructions: Continuous-noise dSprites variants are harder to reconstruct; Noisy-dSprites noise is ignored, while Scream-dSprites loses shape information.The figure compares real samples with reconstructions across datasets and methods.
  • Representation correlations: Increasing regularization decreases total correlation for sampled representations but can increase it substantially for mean representations.The latter effect is reported for β-VAE and β-TCVAE, and less strongly for FactorVAE.
  • Representation correlations: The considered methods generally enforce an uncorrelated aggregated posterior, but this does not imply uncorrelated dimensions in the mean representation.The study treats the mean as the representation used for downstream interpretation.
  • Metric agreement: All disentanglement metrics except Modularity appear correlated, although correlation levels vary across datasets.The strongest reported correlation is between MIG and DCI Disentanglement.
  • Variance analysis: 37% of disentanglement-score variance is explained by objective function alone, rising to 59% when objective function and regularization strength are combined.The remaining variance is attributed to other factors in the analysis, including randomness.
  • Model selection: Disentanglement scores are heavily influenced by random seed and regularization strength, while objective function appears less influential.Across 50 random seeds, no model dominates consistently and good versus bad training runs cannot be distinguished without supervision.
  • Model selection: Unsupervised model selection remains unresolved because transferring good hyperparameters across metrics and datasets does not reliably identify good random seeds.The model-selection comparison is summarized using repeated random draws.

I.6. Are these disentangled representations useful for downstream tasks in terms of the sample complexity of learning?

The study finds no reliable evidence that higher disentanglement improves downstream sample efficiency, although it can correlate with downstream performance in some data sets. These results are limited by models that may not reliably produce disentangled representations and by the study’s restricted set of usefulness criteria.

  • Downstream performance: Higher disentanglement appears correlated with better downstream performance on some data sets, but the relationship is weaker or absent on others.The correlation is observed especially on dSprites variants and, to some degree, Shapes3D, but not consistently on data sets such as Cars3D.
  • Metric relationships: The study does not find enough evidence that reducing total correlation of the aggregate posterior benefits downstream task performance.Rank correlations between unsupervised scores and downstream performance range from mildly negative to mildly positive depending on the data set.
  • Sample complexity: The authors find no conclusive evidence that higher disentanglement scores produce higher statistical efficiency across models and data sets.Statistical efficiency is measured using accuracy with 100 samples relative to accuracy with 10,000 samples.
  • Downstream performance: DCI Disentanglement and MIG seem associated with better gradient-boosted-tree performance on Shapes3D, with similar patterns appearing for some higher-scoring model groups.The authors still conclude that increased disentanglement does not reliably improve sample efficiency overall.
  • Implications: The authors caution that these conclusions could change with semi-supervised or fully supervised models and that interpretability and fairness were not evaluated.They also note that prior successful downstream applications do not establish that performance was caused by disentanglement.
Loading 1811.12359v4…