Source-linked AI summary
Weakly-Supervised Disentanglement Without Compromises
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, Michael Tschannen
TL;DR
The paper asks whether disentangled representations can be learned from realistic image pairs without annotations identifying their shared or changed factors. It proves identifiability under assumptions, develops adaptive methods for such pairs, and finds reliable representations useful across covariate shifts, fairness, and abstract reasoning. The paper concludes that weak supervision supports useful disentanglement, while noting that challenging real-world settings with unobserved factors remain future work.
Problem
Unsupervised disentanglement from i.i.d. observations is theoretically impossible without inductive biases, motivating learning from natural changes with limited annotations.
Method
The paper learns disentangled representations from non-i.i.d. image pairs sharing an unknown subset of factors, using adaptive group-based methods without group or factor annotations.
Results
The learned representations are reliably selected using weak supervision alone and are useful across covariate-shift generalization, fairness, and abstract visual reasoning tasks.
Takeaways & Limitations
Weakly supervised pairs can provide sufficient information for identifiable and practically useful disentangled representations under the paper’s assumptions.
Takeaways & Limitations
Future work should apply the framework to challenging real-world data sets where the factors of variation are not observed and extend it to interactive reinforcement-learning settings.
Abstract
from arXiv · showhide
Intelligent agents should be able to learn useful representations by observing changes in their environment. We model such observations as pairs of non-i.i.d. images sharing at least one of the underlying factors of variation. First, we theoretically show that only knowing how many factors have changed, but not which ones, is sufficient to learn disentangled representations. Second, we provide practical algorithms that learn disentangled representations from pairs of images without requiring annotation of groups, individual factors, or the number of factors that have changed. Third, we perform a large-scale empirical study and show that such pairs of observations are sufficient to reliably learn disentangled representations on several benchmark data sets. Finally, we evaluate our learned representations and find that they are simultaneously useful on a diverse suite of tasks, including generalization under covariate shifts, fairness, and abstract reasoning. Overall, our results demonstrate that weak supervision enables learning of useful disentangled representations in realistic scenarios.
1. Introduction
The paper addresses the difficulty of learning disentangled representations without annotations by using pairs of observations that share some unknown factors. It combines theoretical identifiability results, unannotated adaptive methods, and broad downstream evaluation.
- Unsupervised disentanglement from i.i.d. observations is theoretically impossible without inductive biases and is unreliable across hyperparameters and random seeds.
- Natural-environment changes provide weak supervision because they typically alter only a few underlying factors of variation.
- The proposed setting uses different image pairs with an unknown identity and number of changed factors, requiring only non-dense changes and no group annotations.
- Adaptive group-based methods learn from non-i.i.d. observations without annotated groups and apply to varied settings containing unannotated groups.
- Under weak assumptions, identifiability is possible for non-i.i.d. pairs, while existing group-based VAE methods arise as special cases with relaxed assumptions.
- Weak supervision reliably produces disentangled representations useful for covariate-shift generalization, fairness, and abstract visual reasoning without supervised disentanglement metrics for model selection.
2. Related work
Related work connects disentanglement to independent-component recovery, task-dependent supervision, and renewed unsupervised and weakly supervised approaches. The paper builds on this progression toward less restrictive supervision.
- Independent-component analysis seeks to unmix independent non-Gaussian sources, while nonlinear identifiability is impossible from i.i.d. observations.
- Prior applied approaches used explicit, task-dependent supervision, including known relations between factors of variation.
- After renewed interest in unsupervised disentanglement and quantitative evaluation, theoretical impossibility results shifted attention toward semi-supervised and weakly supervised methods.
3. Generative models
The paper models paired images as sharing a subset of independent latent factors while resampling the rest, then uses these relations to learn disentangled representations without knowing the shared factors.
- The standard generative model samples independent factors and generates observations from their conditional distribution.
- Disentanglement maps observations so each factor affects exactly one nonoverlapping coordinate or coordinate group.
- The weakly supervised model treats paired observations as images sharing some factors through an intervention on a random subset, without requiring knowledge of which or how many are shared.
- The shared-index subset S has size d − k, and the shared and resampled latent entries use identical marginal distributions.
- To form the second observation, the model retains z entries indexed by S and substitutes resampled factors for the complementary indices.
- The generative model assumes additive noise is not separately modeled; noise is instead represented as a latent variable whose effects pass through g⋆.
4. Identifiability and algorithms
The paper establishes identifiability for paired observations under weak assumptions and derives adaptive algorithms that estimate unknown changed factors without group annotations. These algorithms modify β-VAE inference by averaging selected latent coordinates, while practice differs from theory through finite data, approximate matching, and potentially unknown random k.
- Identifiability: Identifiability is possible for the paired generative model under weak additional assumptions, unlike the standard unsupervised setting.The theorem assumes continuous matching factor priors, suitable overlap among changed-factor sets, a smooth invertible generator, unlimited data, and known fixed k.
- Identifiability: Matching the true paired-observation marginal forces learned generative models to be disentangled up to coordinate-wise reparameterizations and factor permutations.This yields a one-to-one correspondence between ground-truth factors and learned coordinates.
- Practical adaptive algorithms: The practical methods use β-VAE variants with adaptive group inference and a selection heuristic for unknown and random k.They are called Adaptive-Group-VAE and Adaptive-ML-VAE, depending on whether GVAE or ML-VAE averaging is used.
- Practical adaptive algorithms: For each image pair, the algorithms estimate shared coordinates by selecting those with the smallest KL divergence between encoder posteriors, then replace them with an average.Averaging imposes a hard shared-coordinate constraint and implicitly encourages non-shared dimensions to encode changing factors efficiently.
- Practical adaptive algorithms: The unknown-k heuristic thresholds δ_i = D_KL(q_φ(ẑ_i|x_1)||q_φ(ẑ_i|x_2)) and averages coordinates with δ_i < τ.The estimate assumes not all factors changed and is reported to work well in practice; it is always correct when the encoder is disentangled.
- Relation to prior work: Unlike prior group-based methods, the approach estimates which factors changed for every pair and removes requirements for group annotations, known fixed k, or factor-change labels.The theory assumes known fixed k, whereas practice may involve finite data, approximate distribution matching, and unknown random k.
5. Experimental results
Across five datasets, weakly supervised methods reliably learn disentangled representations without group annotations, adapting to unknown and varying numbers of changed factors. The resulting Ada-GVAE representations support downstream accuracy, covariate-shift generalization, fairness, and low-sample abstract reasoning.
- Experimental setup: Five datasets were used to train and evaluate weakly supervised and unsupervised disentanglement models.The study includes dSprites, Cars3D, SmallNORB, Shapes3D, and MPI3D.
- Disentanglement performance: Weakly supervised models outperform unsupervised ones, including when unsupervised baselines receive fully supervised model selection.Model selection for the weakly supervised methods uses only weakly supervised reconstruction loss.
- Adaptation to pair structure: Ada-GVAE adapts to unknown values of k and performs better when paired observations differ in fewer factors.The method is agnostic to which factors are shared, though known constant sharing patterns could improve performance.
- Supervision-performance trade-offs: Incomplete group knowledge can significantly harm GVAE and ML-VAE performance, whereas adaptive methods avoid requiring precise group annotations.Precisely known groups can improve disentanglement, but the adaptive method is often comparable without those annotations.
- Downstream usefulness: Weakly supervised reconstruction loss is a useful proxy for downstream accuracy, strong generalization under covariate shifts, and fairness.For Ada-GVAE, lower reconstruction loss is generally associated with more accurate downstream classifiers and correlates with strong generalization and unfairness.
- Downstream usefulness: Ada-GVAE representations improve fairness and abstract reasoning sample efficiency, with the reasoning benefit occurring at low sample sizes.The fairness result concerns demographic parity and does not require observing the target or sensitive variable for model selection.
6. Conclusion
The paper studies disentanglement from pairs of non-i.i.d. observations sharing an unknown, random subset of factors, proves identifiability under technical assumptions, and develops weakly supervised model selection for downstream tasks. It identifies future application to challenging real-world datasets and interactive reinforcement-learning settings.
- Pairs of non-i.i.d. observations with unknown, random shared factors can yield an identifiable disentangled generative model under certain technical assumptions.
- The proposed models can be selected using weak supervision while achieving strong performance across a diverse suite of downstream tasks without supervised disentanglement metrics.
- Future work should apply the framework to challenging real-world datasets where factors of variation are unobserved and extend it to interactive reinforcement learning.
A. Proof of Theorem 1
The proof shows that pairwise sharing constraints first recover two disentangled coordinate subspaces for fixed shared factors, then force coordinate-wise reparameterization when the shared subset is randomly sampled. This establishes disentanglement up to permutation and coordinate-wise reparameterization.
- Proof outline: The proof proceeds by characterizing posterior constraints, parameterizing candidates, establishing fixed-subset disentanglement, and analyzing random subset sampling.
- Step 4: Randomly sampling shared-factor subsets exposes every factor through subset intersections and forces the transformation Jacobian to become diagonal.
- Step 2: Candidate posteriors are parameterized through smooth invertible transformations composed with the true generative function.
- Step 3: For fixed shared factors, admissible transformations align shared and non-shared coordinate subspaces, yielding a block-diagonal Jacobian.
- Conclusion: The resulting aggregate posterior is a coordinate-wise reparameterization of the true factor distribution up to permutation, so changing one factor changes one corresponding learned coordinate.
B. Implementation Details
The experiments fix architectures and most hyperparameters while sweeping one model-specific hyperparameter, and evaluate representations on prediction, covariate-shift, fairness, and abstract-reasoning tasks. Adaptive methods are compared with unsupervised baselines across disentanglement measures.
- Experimental setup: The study fixes the architecture and most hyperparameters, sweeping one hyperparameter per model; unsupervised settings follow the prior benchmark configuration.The adaptive models usually select β = 1, while values below one are not explored.
- Downstream tasks: The vanilla downstream task predicts each factor from representations using logistic regression or gradient-boosted classifiers across training-set sizes.Training sets contain 10, 100, 1000, or 10 000 examples, with a 5000-point test set.
- Downstream tasks: The covariate-shift task trains on data where another factor is fixed, then tests on examples where that factor takes a different value.
- Downstream tasks: The fairness task predicts each factor with gradient-boosted classifiers and measures unfairness using the referenced fairness formula.
- Downstream tasks: The abstract-reasoning task trains a relational network on frozen representations for grids whose rows contain 1, 2, or 3 constant ground-truth factors.
- Evaluation: Adaptive methods are compared with unsupervised baselines on model selection, Completeness, and other disentanglement scores.
C.1. Section 5.1
Adaptive Ada-GVAE and Ada-ML-VAE methods remain competitive with unsupervised baselines across disentanglement evaluations, including Completeness, while performance is strongest when image pairs differ sparsely. Weakly supervised model selection is also competitive with fully supervised selection.
- Model selection: Adaptive methods are competitive with fully supervised model selection on unsupervised methods when the number of shared factors varies randomly across pairs.
- Disentanglement scores: The same broad performance trends appear across DCI Disentanglement, Completeness, and other disentanglement scores.
- Training metrics: Rank correlations between training metrics and disentanglement scores are examined separately for Ada-GVAE and Ada-ML-VAE.
- Effect of k: Ada-GVAE and Ada-ML-VAE perform best when image changes are sparse, with k = 1, and increasingly resemble vanilla β-VAE as k grows.
C.3. Section 5.3
The experiments show that known factor-change information can improve disentanglement, but incomplete group knowledge can harm it. The adaptive methods achieve broadly competitive results while requiring less supervision than prior group-based approaches.
- Known factor-change information improves disentanglement when k = 1, whereas incomplete knowledge harms disentanglement.This pattern holds regardless of the averaging procedure.
- The adaptive methods achieve ball-park similar results to Change and Share GAN-based approaches, but the comparison uses different experimental conditions.The GAN-based methods also receive additional supervision identifying which factor is shared or changed.
- When k = d −1, the adaptive methods perform worse than the GAN-based methods, consistent with the latter methods receiving factor-level supervision.The GAN-based methods perform especially well on SmallNORB and MPI3D, where VAE approaches struggle with reconstruction.
- Adaptive Ada-GVAE and Ada-ML-VAE variants are compared with vanilla methods that require or assume group knowledge.Figures 12 and 13 report comparisons for k = Rnd and k = 1.
- Adaptive group-based methods are competitive with unsupervised methods in downstream performance when the number of shared factors varies across pairs.The comparison considers no model selection, DCI Disentanglement selection, and test-downstream-performance selection.
C.4. Section 5.4
The adaptive approach is competitive with unsupervised methods on downstream performance, while reconstruction-based evaluation tracks downstream accuracy, disentanglement, fairness, and abstract reasoning performance.
- The adaptive method is competitive in downstream performance even when unsupervised methods receive model selection using test downstream performance.The adaptive models themselves are always selected using reconstruction error.
- Weakly-supervised reconstruction loss generally has a negative rank correlation with downstream prediction performance for Ada-ML-VAE.The relationship is evaluated with Logistic Regression and Gradient Boosted decision-Trees at different sample sizes.
- Weakly-supervised reconstruction loss is reported as a proxy for classifier unfairness across the data sets.The comparison includes disentanglement scores, reconstruction loss, and GBT10000 unfairness, alongside unsupervised baselines and Ada-ML-VAE with k = 1.
- Abstract visual reasoning experiments compare disentanglement metrics, prediction accuracy, reconstruction, and downstream accuracy throughout training.The analysis tracks these quantities for Ada-ML-VAE across different sample sizes.