Source-linked AI summary
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations
Abhishek Kumar, Prasanna Sattigeri, Avinash Balakrishnan
TL;DR
The paper studies how to learn disentangled representations from unlabeled observations, motivated by the benefits of separating latent generative factors. It introduces a regularized amortized variational-inference approach and reports improved disentanglement and data likelihood, together with the SAP metric. The paper also identifies sampling biases as a challenge for defining disentanglement.
Problem
The paper asks how to infer disentangled latent factors from raw, unlabeled observations, despite prior methods relying on supervision or facing scalability and likelihood trade-offs.
Method
The method applies amortized variational inference with a regularizer over the induced inferred prior to encourage disentangled latent factors, and introduces the SAP metric.
Results
The approach outperforms β-VAE on quantitative disentanglement metrics while matching VAE generated-sample quality and achieving better quality than β-VAE.
Takeaways & Limitations
SAP is better aligned than Z-diff with qualitative disentanglement in decoder latent traversals, while the proposed framework preserves the data-likelihood objective.
Takeaways & Limitations
Sampling biases in the generative process make disentanglement challenging and, for natural biases, somewhat less well defined.
Abstract
from arXiv · showhide
Disentangled representations, where the higher level data generative factors are reflected in disjoint latent dimensions, offer several benefits such as ease of deriving invariant representations, transferability to other tasks, interpretability, etc. We consider the problem of unsupervised learning of disentangled representations from large pool of unlabeled observations, and propose a variational inference based approach to infer disentangled latent factors. We introduce a regularizer on the expectation of the approximate posterior over observed data that encourages the disentanglement. We also propose a new disentanglement metric which is better aligned with the qualitative disentanglement observed in the decoder's output. We empirically observe significant improvement over existing methods in terms of both disentanglement and data likelihood (reconstruction quality).
1 INTRODUCTION
Disentangled representations separate latent generative factors across feature dimensions, supporting invariant, transferable, and interpretable representations. The paper addresses unsupervised disentanglement from raw observations and proposes a variational approach intended to preserve likelihood while improving disentanglement.
- Disentangled representations assign generative factors to disjoint subsets of feature dimensions, so changing one factor produces a sparse representation change.
- They can simplify invariance by marginalizing nuisance-factor dimensions and support transferability and interpretability.
- Earlier disentanglement methods often rely on supervision about factors, known transformations, temporal changes, or complementary signals.
- Unsupervised learning is challenging because realistic settings provide only raw observations, while many earlier approaches do not scale well.
- β-VAE improves empirical disentanglement but creates tension between disentanglement and observed-data likelihood, resulting in poorer generated-sample quality.
- The proposed method uses amortized variational inference with a regularizer over the induced inferred prior and reports improved disentanglement without sacrificing generated-sample quality.
2 FORMULATION
The formulation uses amortized variational inference to approximate latent posteriors and encourages disentanglement by matching the inferred latent distribution to a disentangled prior. DIP-VAE implements this idea through covariance-based regularization while retaining tractable stochastic optimization.
- 2 FORMULATION: A latent variable z is sampled from p(z), observations are generated from pθ(x|z), and inference seeks pθ(z|x) from observed samples.For high-dimensional data, the exact posterior is generally intractable, motivating approximate inference.
- 2 FORMULATION: Amortized inference uses a recognition model qφ(z|x) to share information across observations and optimizes the expected posterior KL through the ELBO.The ELBO lower-bounds observed-data log-likelihood and can be optimized with minibatch stochastic first-order methods when the KL is analytic.
- 2 FORMULATION: VAE generative models typically use a simple prior with independent latent dimensions, while pθ(x|z) absorbs dependencies needed to model complex observations.The cited related work notes that implicit GAN models lack an effective inference mechanism and that prior visual approaches can reconstruct samples far from their inputs.
- 2.2 INFERRING DISENTANGLED LATENTS: The inferred prior qφ(z), also called the expected variational posterior, is required to factor across latent dimensions so inferred factors are disentangled.This can be encouraged by minimizing a distance between qφ(z) and the disentangled generative prior p(z).
- 2.2 INFERRING DISENTANGLED LATENTS: Variational inference can naturally encourage approximate disentanglement, but mismatches between data distributions and non-convex ELBO optimization can leave a substantial gap.The paper motivates an additional regularizer because minimizing the posterior objective need not sufficiently minimize the distance between the inferred prior and p(z).
- 2.2 INFERRING DISENTANGLED LATENTS: DIP-VAE regularizes the inferred prior toward the disentangled prior, with λ controlling the regularizer's contribution to the overall objective.The method is named Disentangled Inferred Prior VAE and is proposed as a tractable alternative to directly optimizing an intractable KL distance.
- 2.2 INFERRING DISENTANGLED LATENTS: For Gaussian approximate posteriors, total covariance decomposes into expected conditional covariance plus covariance of posterior means, which is matched toward the identity matrix.Because conditional covariance is commonly diagonal, the method separately controls diagonal and off-diagonal losses; DIP-VAE-I can regularize posterior-mean variances directly, whereas DIP-VAE-II relies on their sum with expected encoder variance.
- 2.2 INFERRING DISENTANGLED LATENTS: The covariance regularizer is minibatch-compatible, while higher-order moments and alternative distances such as MMD are possible but introduce additional optimization considerations.A running covariance estimate was not observed to improve significantly over estimating covariance from the current minibatch.
3 QUANTIFYING DISENTANGLEMENT: SAP SCORE
The SAP score evaluates whether individual latent dimensions selectively predict generative factors, while accounting for competing dimensions. It is designed as a classifier-free alternative that better reflects qualitative disentanglement in decoder outputs.
- The Z-diff score trains a low-capacity multiclass classifier to identify generative factors from averaged absolute differences between inferred representations.
- SAP constructs a d × k matrix whose entries measure how well each latent dimension predicts each generative factor using linear regression or classification.
- For regression factors, SAP uses the R2 score from a fitted line; for categorical factors, it uses balanced accuracy from thresholds on a single latent.
- Table 1 reports Z-diff, SAP, and per-pixel reconstruction error on test sets for 2D Shapes and CelebA.The caption specifies separate hyperparameter settings for the two datasets.
- Inactive latent dimensions receive zero scores when their inferred values have near-zero variance.
- The final SAP score averages, across factors, the difference between the top two latent-specific prediction scores.
4 EXPERIMENTS
DIP-VAE is evaluated on CelebA, 3D Chairs, and 2D Shapes against VAE and β-VAE, using disentanglement scores, reconstruction error, qualitative traversals, and attribute classification. Results show that DIP-VAE can improve quantitative scores and CelebA attribute prediction, while SAP better tracks qualitative disentanglement than Z-diff.
- Datasets and baselines: DIP-VAE is evaluated on CelebA, 3D Chairs, and 2D Shapes against VAE and β-VAE.CelebA uses celebrity face images, 3D Chairs uses rendered chair masks, and 2D Shapes varies shape, position, scale, and rotation.
- Evaluation limits: For 3D Chairs, only two ground-truth factors are available and quantitative scores saturate near their peak values, so evaluation emphasizes latent traversal plots.Subjective evaluation focuses on reconstruction quality and disentanglement.
- Evaluation: The experiments report SAP and Z-diff disentanglement scores together with reconstruction error on CelebA and 2D Shapes.The plots vary β for β-VAE and λod for DIP-VAE variants; reconstruction error directly corresponds to data likelihood.
- Quantitative and qualitative results: DIP-VAE-I reaches a Z-diff score of 98.7 with almost an order of magnitude lower reconstruction error than β-VAE at β = 60 on 2D Shapes.Despite this quantitative advantage, the corresponding latent traversals indicate better disentanglement for β-VAE at β = 60.
- Metric comparison: The SAP score is better aligned with qualitative disentanglement in decoder outputs than Z-diff and does not require training a classifier.Qualitative comparisons use single-latent traversals while holding other latents fixed.
- Attribute classification: DIP-VAE outperforms VAE and β-VAE for most evaluated CelebA attributes, while β-VAE performance worsens as β increases further.The reported table highlights the attribute with the largest change; most other attribute accuracies remain similar across methods.
5 RELATED WORK
Related work includes adversarial autoencoders, which match the aggregated posterior to the prior, and representations framed through invariance and equivariance. These approaches differ from the paper’s variational inference objective and its sparse-action view of disentanglement.
- Adversarial autoencoders: Adversarial autoencoders match the aggregated posterior qφ(z) to the prior p(z) using a discriminator-induced distance.The discriminator distinguishes samples from qφ(z) and p(z), inducing a Jensen–Shannon divergence in the described setup.
- Objective distinction: Unlike variational inference, adversarial autoencoders do not primarily minimize KL(qφ(z|x)||pθ(z|x)).The comparison identifies the objective distinction between matching aggregated distributions and posterior inference.
- Invariance and equivariance: Equivariance requires transformed inputs to produce correspondingly transformed features, whereas disentanglement restricts that feature transformation to a small subset of dimensions.Disentanglement is presented as a special case of the broader equivariance notion.
6 CONCLUDING REMARKS
The paper proposes a variational framework for inferring disentangled latents from unlabeled observations and reports improved alignment between SAP and qualitative latent traversals. It identifies sampling biases as a challenge that makes disentanglement less well defined in some natural-data settings.
- Contributions: The proposed variational objective is reported to avoid β-VAE’s conflict between data log-likelihood and disentanglement.The authors connect this claim to their empirical results.
- Contributions: SAP is reported to correlate much better with qualitative disentanglement in latent traversals than the Z-diff score.This metric is intended to better reflect the decoder-level behavior observed in the experiments.
- Limitations and future directions: Natural and artificial sampling biases can make the factor-independence problem challenging and somewhat less well defined.The paper gives gender–beard and gender–smile correlations in face-image data as examples of such biases.
- Limitations and future directions: Effective use of disentangled representations for transfer learning is identified as a direction for future work.
A LATENT TRAVERSALS FOR 2D SHAPES AND CHAIRS DATASET
Figures 5 and 6 present qualitative disentanglement results for the CelebA and Chairs datasets, respectively.
- Figure 5 reports qualitative disentanglement results for the CelebA dataset.
- Together, the figures provide qualitative results for disentanglement across CelebA and Chairs.
- Figure 6 reports qualitative disentanglement results for the Chairs dataset.