PaperlayerView exact revision on arXiv

Source-linked AI summary

On Disentangled Representations Learned From Correlated Data

Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, Stefan Bauer

arXiv:2006.07886v3cs.LGstat.ML

TL;DR

Disentanglement methods seek independent factors, but real-world observations often contain correlated factors, and prior evaluation on independent-factor data leaves their behavior in that setting uncertain. This paper studies 4260 models on correlated data and finds that the correlations are reflected in latent representations, then demonstrates approaches to resolve that entanglement.

  • Problem

    Disentanglement methods have mostly been trained and evaluated on independent factors, leaving their performance on correlated data uncertain despite the prevalence of such correlations in realistic settings.

    Source: §1 · pp. 1–2

  • Method

    The authors conduct a large-scale study of 4260 models on systematically correlated data and investigate weak supervision during training and post-hoc correction with few labels.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    The study finds that strongly correlated factors are learned as entangled latent dimensions, while some learned latent structures support meaningful extrapolation to unseen examples.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The paper demonstrates that latent correlations can be addressed through fast adaptation and other weakly supervised approaches.

    Source: §1 · p. 2, §7 · p. 9

    The systematic study focuses on linear correlations, and the authors caution that its conclusions do not establish behavior in highly nonlinear correlated settings.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

The focus of disentanglement approaches has been on identifying independent factors of variation in data. However, the causal variables underlying real-world observations are often not statistically independent. In this work, we bridge the gap to real-world scenarios by analyzing the behavior of the most prominent disentanglement approaches on correlated data in a large-scale empirical study (including 4260 models). We show and quantify that systematically induced correlations in the dataset are being learned and reflected in the latent representations, which has implications for downstream applications of disentanglement such as fairness. We also demonstrate how to resolve these latent correlations, either using weak supervision during training or by post-hoc correcting a pre-trained model with a small number of labels.

1. Introduction

Disentanglement methods are usually evaluated where factors vary independently, although real-world data often contains correlated factors. This study tests how current methods learn from such data and examines ways to address latent entanglement.

1. Introduction

  • Prior evaluations have mostly used datasets where factors of variation are independent, leaving performance on correlated training data unknown.The introduction motivates this gap with real-world factors such as foot length and body height that can be correlated.Source: §1 · pp. 1–2
  • The authors present a large-scale empirical study of 4260 models examining disentanglement learners on data with correlated ground-truth factors.They introduce dataset correlations in a controlled manner and evaluate multiple datasets with ground-truth labels, including generalization and fair decision making.Source: §1 · p. 2
  • The study finds that factorization-based inductive biases do not disentangle strongly correlated factors, leaving latent dimensions statistically entangled.The authors note implications for downstream applications such as fairness.Source: §1 · p. 2
  • The paper proposes post-hoc alignment of entangled latents using few labels and investigates weakly supervised methods for resolving entanglement during training.The introduction also notes that disentangled representations have been associated with interpretability, generalization, and other potential benefits.Source: §1 · pp. 1–2

2. Background

The background describes disentanglement methods built on variational autoencoders and the regularizers used to structure their latent spaces. It also notes that unsupervised disentanglement is not identifiable from marginal likelihood alone, motivating weak supervision.

2. Background

  • Variational autoencoders model observations through latent features and optimize a variational lower bound on data log likelihood.The prior is typically fixed, while neural networks parameterize the generative model and approximate posterior.Source: §2 · p. 2
  • Disentanglement methods such as β-VAE, FactorVAE, and DIP-VAE add regularizers because the basic objective does not impose semantic structure on latent representations.The passage also notes that researchers use evaluation metrics to quantify different notions of disentanglement.Source: §2 · p. 2
  • Unsupervised disentanglement by marginal-likelihood optimization is impossible, while grouping information, labels, or weak supervision can provide additional constraints.Weakly supervised approaches use pairs of observations differing in a known number of factors; such supervision has facilitated disentanglement on uncorrelated datasets.Source: §2 · p. 2
  • Popular benchmark datasets commonly enforce independence among their factors, an assumption that may not hold in practice.The passage lists dSprites, Cars3D, SmallNORB, Shapes3D, and MPI3D variants as examples.Source: §2 · pp. 2–3

3. The Problem with Correlated Data

When observed factors are correlated, a factorized latent prior can make a perfectly disentangled representation assign probability to combinations absent from training data. The section formalizes this tension and motivates resolving latent correlations, while noting that its systematic experiments focus on pairwise linear correlations.

3. The Problem with Correlated Data

  • Correlations can arise from direct causal relationships or shared confounders, yet the paper is interested in representing correlated factors independently.The motivating example identifies age or sex as possible confounders for foot length and body height.Source: §3 · p. 3
  • A factorized latent prior can generate combinations of correlated factors outside the training distribution, so perfect disentanglement yields sub-optimal log likelihood.The section uses this mismatch to explain why generative models may favor encoding correlated factors together.Source: §3 · p. 3
  • With correlated ground-truth factors and a factorized latent prior, matching the optimal likelihood requires entangled representations under the stated smooth-bijection assumption.The paper defines disentanglement by a Jacobian diagonal up to permutation and argues that likelihood-based objectives may therefore bias against it in this setting.Source: §3 · p. 3
  • The authors motivate resolving rather than exploiting training correlations for distribution-shift robustness, independent factor interventions, and fairness-related downstream decisions.They note that intervening on sensitive attributes such as sex or race may be impossible in some fairness applications.Source: §3 · pp. 3–4
  • The paper investigates post-training correction with few labels and weak supervision using observation pairs that differ in a known number of factors.For Ada-GVAE, shared latent factors are estimated from coordinate-wise KL divergences, then the approximate posteriors for shared factors are averaged.Source: §3 · p. 4

4. Unsupervised Disentanglement under Correlated Factors

Across controlled correlated-data experiments, unsupervised factorization-based methods entangle correlated factors, yet their latent spaces can still support generalization to unseen combinations. The section also examines how this behavior varies with correlation strength and affects fairness-related scores.

4.1. Experimental Design

  • The study introduces linear, Gaussian-noise correlations between factor pairs across five dataset variants and trains six VAE methods across hyperparameter settings and random seeds.Correlation strength is controlled by σ, with lower values indicating stronger correlation; the experiments total 2,880 models in this setup.Source: §4.1 · pp. 4–5

4.2. Unsupervised Methods for Correlated Data

  • The correlated factor pair is more entangled than other pairs under stronger correlations, while entanglement decreases as correlation weakens and factors become easier to disentangle at σ ≥ 0.7.The pairwise entanglement scores quantify how difficult it is to separate factors from their latent codes.Source: Table 2 · p. 5, §4.2 · p. 6
  • Unfairness scores rise with stronger correlations between the factors, aggravating concerns when one factor is sensitive and the other is predicted.The evaluation uses a demographic-parity variant based on pairwise mutual information between latents and factors of variation.Source: Figure 3 · p. 6, §4.2 · p. 6

4.3. Generalization Under Changing Correlations

  • Models reconstruct configurations with zero probability under the correlated training distribution, and their latent encodings meaningfully extend the structure learned from training examples.The result concerns decoder generalization; the two entangled latent dimensions still encode both correlated factors.Source: Figure 4 · p. 6, §4.3 · pp. 6–7

5. Recovering Factorized Latent Codes

The section presents two routes to reduce entanglement in correlated data: fast adaptation with a few FoV labels after training, and weak supervision during training. Weak supervision often recovers strong disentanglement and reduces unfairness, while fast adaptation can reduce pairwise correlations with as few as 100 labels.

Post-hoc alignment correction with few labels

  • Fast adaptation with linear regression reduces pairwise entanglement in Shapes3D with as few as 100 correlated-factor labels.Those labels amount to less than 0.02% of all ground-truth labels in Shapes3D.Source: §5 · p. 7, Table 3 · p. 7
  • Linear substitution can fail when correlated factors are not isolated in two latent dimensions or lack a unique natural ordering; an MLP can further reduce entanglement in these cases.The MLP results are reported in Appendix D.1.Source: §5 · p. 7

Weak supervision mitigates learning latent entanglement

  • Weak supervision often recovers perfect DCI disentanglement and reduces unfairness across correlation strengths, with Figure 5 showing these results on Shapes3D.The weakly supervised representations also tend to align their coordinates with ground-truth label axes.Source: §5 · pp. 7–8, Figure 5 · p. 8

Summary

  • When labels are unavailable, weakly supervised training offers an alternative; together, the approaches demonstrate ways to leverage weak supervision to learn disentangled representations from correlated data.Additional weak-supervision studies found the same trends across datasets and training regimes.Source: §5 · p. 8

6. Other Related Work

Related work spans nonlinear ICA and methods for rotated or entangled latent spaces, while systematic large-scale evaluation on correlated data has been limited. Prior studies vary in scale, metrics, and supervision requirements.

ICA and Disentanglement

  • Nonlinear ICA has a long history, and recent work combines it with disentanglement.The cited literature includes Comon (1994), Jutten & Karhunen (2003), Hyvärinen & Pajunen (1999), Hyvarinen et al. (2019), Hyvarinen & Morioka (2016), Khemakhem et al. (2020), and Sorrenson et al. (2020).Source: §6 · p. 8

Entangled and rotated latent spaces

  • Work on rotated or entangled latent spaces examines topological similarity, orthogonal PCA embeddings encouraged by diagonal priors, or rotationally asymmetric priors.This paper instead studies dependencies in the data-generating prior.Source: §6 · p. 8

Studies on correlated data

  • Prior correlated-data studies were small or used different settings; some relied on global disentanglement metrics, while others required strong FoV-label supervision at training time.The authors argue global metrics may be insufficient to diagnose latent structure in correlated-data models.Source: §6 · p. 8

7. Conclusion

The study finds that existing methods fail to disentangle strongly correlated factors, while some learned latent spaces still support meaningful extrapolation to unseen examples. It demonstrates corrections through fast adaptation and weakly supervised training, and identifies extensions to higher-resolution inputs and real-world downstream effects as open questions.

7. Conclusion

  • Existing methods fail to learn disentangled representations for strongly correlated factors, but some learned latent spaces accommodate unseen examples through meaningful extrapolation.The authors report this as an out-of-distribution generalization capability.Source: §7 · p. 9
  • The study demonstrates correcting latent entanglement through fast adaptation and weakly supervised training methods.Source: §7 · p. 9
  • Whether these findings extend to high-resolution inputs and how they affect real-world downstream tasks remain open questions.Source: §7 · p. 9

A. Proof of Proposition 1

The proof relates the learned factor distribution to the true factor distribution under a change of variables. When the true factors are correlated but the model prior factorizes, a disentangled generative model cannot attain optimal likelihood, whereas an entangled model can.

A. Proof of Proposition 1

  • Because the true factor distribution does not factorize while the model prior does, the change-of-variables map has a Jacobian that is not permutation-diagonal.A disentangled representation would require a permutation-diagonal Jacobian, so it cannot equal the change-of-variables map almost everywhere.Source: A. Proof of Proposition 1 · p. 13
  • A disentangled generative model therefore has a positive KL divergence between true and learned factor distributions and achieves less than optimal expected likelihood.Source: A. Proof of Proposition 1 · p. 13
  • An unconstrained entangled model can match the change-of-variables map, making the factor-distribution KL zero and attaining optimal likelihood.Source: A. Proof of Proposition 1 · p. 13

B. Implementation details

The study applies six VAE-based disentanglement methods to controlled correlations across datasets and evaluates both representation structure and fairness-related measures.

B. Implementation details

  • The six evaluated VAE methods use different posterior-aggregate regularizers and share an encoder-decoder architecture with 10 latent dimensions.The methods are β-VAE, FactorVAE, AnnealedVAE, DIP-VAE-I, DIP-VAE-II, and β-TC-VAE.Source: B. Implementation details · p. 13
  • The study varies correlated-factor sampling distributions across datasets A–E and correlation strengths σ = 0.2, 0.4, 0.7, and ∞.Datasets A–C use ordinal correlated factors, whereas D and E include a correlated factor without natural ordering.Source: B. Implementation details · pp. 13–14, Figure 6 · p. 14
  • Pairwise entanglement tracks when two factors share latent-code links, while unfairness is measured as demographic parity for a target given a sensitive factor.The study also evaluates BetaVAE, FactorVAE, SAP, MIG, and DCI disentanglement scores.Source: B. Implementation details · p. 14
  • Across datasets and correlation strengths, stronger correlations correspond to greater pairwise latent entanglement than the uncorrelated baseline.Table 4 aggregates across models; Table 5 separates results by disentanglement regularizer.Source: Table 4 · p. 14

C.1. Section 4.2

Correlated factors are more entangled in the learned latents than other factor pairs, although common global disentanglement scores do not clearly reveal this pattern. Restricting evaluation to the correlated pair makes DCI the only metric that tracks the entanglement.

C.1. Section 4.2

  • For a weakly correlated Shapes3D model, latent traversals and feature importance suggest that selected dimensions do not mix object size and azimuth.The shown model has correlation strength σ = 0.7.Source: Figure 7 · p. 15, §C.1 · p. 16
  • The correlated factor pair has higher pairwise entanglement than the median of other pairs across trained models and regularizers.Table 5 separates the pairwise scores by regularizer; Table 4 reports aggregated results.Source: Table 5 · p. 15, §C.1 · p. 16
  • Common global disentanglement metrics show no clear trend with correlation strength when evaluated on correlated or uncorrelated data.The authors attribute this limitation partly to averaging across many factors.Source: §C.1 · p. 16, Figure 9 · p. 16, Figure 10 · p. 16
  • When evaluation is restricted to the two correlated factors, only DCI tracks their latent entanglement; the other metrics show no or weak trends.The comparison excludes all remaining factors from the metric evaluation.Source: Figure 11 · p. 16, §C.1 · p. 17

C.2. Section 4.3

The paper examines whether representations trained on correlated data handle unseen factor configurations and whether a small labeled set can correct latent alignment. The reported results show meaningful out-of-distribution reconstructions and reduced pairwise entanglement after nonlinear post-hoc adaptation.

C.2. Section 4.3

  • Latent traversals from an unseen object-size–azimuth configuration yield expected reconstructions as the remaining factors change.Additional models on strongly correlated Shapes3D variants support meaningful encoding of out-of-distribution examples and decoding from those representations.Source: §C.2 · p. 17, Figure 13 · p. 17

D.1. Post-hoc alignment correction with few labels

  • Using 1000 labels, a one-hidden-layer MLP for post-hoc adaptation significantly reduces pairwise entanglement thresholds for correlated factors.The study samples labels from a uniform independent distribution; the results are reported for Shapes3D variants A, D, and E.Source: §D.1 · p. 18
  • Linear-regression adaptation can fail when correlated factors are not isolated in two latent dimensions or lack a unique natural ordering.The authors also note that the latent-manifold form beyond the training distribution is unknown and generally expected to be nonlinear.Source: §D.1 · p. 18, Figure 14 · p. 18

D.2. Alignment during training using weak supervision

Weak supervision during training is evaluated as a way to disentangle correlated factors, including settings with intervention-derived observation pairs. The reported scenarios show improved or strong disentanglement, including under a causal sampling assumption.

D.2. Alignment during training using weak supervision

  • Ada-GVAE with k = 1 provides a strong inductive bias for finding the factorization and resolving spurious correlations across unknown correlation strengths.The study reports axis-aligned latent spaces across Shapes3D A, D, and E.Source: §D.2 · p. 18, Figure 15 · p. 19
  • Under Scenario I-1, weakly supervised models consistently achieve high disentanglement, often perfect DCI, despite correlations in the dataset.Scenario I-1 assumes an unobserved confounder and breaks the correlation when the intervention changes one of the correlated factors.Source: §D.2 · p. 18, Figure 16 · p. 19
  • Scenario I-2 reports significant improvement in disentangling correlated factors when observation pairs follow the assumed causal generative model.The study includes Shapes3D A models at σ = 0.1 in addition to the correlation strengths used elsewhere.Source: §D.2 · pp. 18–19, Figure 17 · p. 19

Related papers

Loading 2006.07886v3…