PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured Disentangled Representations

Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, N. Siddharth, Brooks Paige, Dana H. Brooks, Jennifer Dy, Jan-Willem van de Meent

arXiv:1804.02086v4stat.MLcs.LG

TL;DR

Unsupervised disentanglement approaches often rely on simple diagonal-Gaussian priors and do not reliably disentangle discrete factors. This paper derives a two-level hierarchical objective for controlling independence within and across variable groups; experiments report improved discrete-factor disentanglement and recovery of unseen combinations of latent features.

  • Problem

    Prior disentanglement work has largely focused on uncorrelated scalar factors, while real-world data may involve dependencies among higher-dimensional variables.

    Source: §6 · p. 9

  • Method

    The paper derives a two-level hierarchical objective from the ELBO that controls statistical independence across variable blocks and among variables within blocks.

    Source: §5 · p. 6

  • Results

    HFVAE captures digit information in a discrete variable more clearly than β-VAE and recovers combinations of latent features absent from training data.

    Source: §1 · p. 2, §5.3 · p. 8

  • Takeaways & Limitations

    The work extends disentangled representations toward modeling dependencies among higher-dimensional variables.

    Source: §6 · p. 9

    The authors identify a need for datasets that more rigorously test the approach's ability to characterize correlations between higher-dimensional variables.

    Source: §6 · p. 9

Abstract

from arXiv · show

Deep latent-variable models learn representations of high-dimensional data in an unsupervised manner. A number of recent efforts have focused on learning representations that disentangle statistically independent axes of variation by introducing modifications to the standard objective function. These approaches generally assume a simple diagonal Gaussian prior and as a result are not able to reliably disentangle discrete factors of variation. We propose a two-level hierarchical objective to control relative degree of statistical independence between blocks of variables and individual variables within blocks. We derive this objective as a generalization of the evidence lower bound, which allows us to explicitly represent the trade-offs between mutual information between data and representation, KL divergence between representation and prior, and coverage of the support of the empirical data distribution. Experiments on a variety of datasets demonstrate that our objective can not only disentangle discrete variables, but that doing so also improves disentanglement of other variables and, importantly, generalization even to unseen combinations of factors.

1 Introduction

The paper introduces hierarchically factorized VAEs (HFVAEs), a two-level objective for controlling statistical independence across groups of latent variables and within each group. The authors report that HFVAEs disentangle discrete and continuous factors, improve over related methods, and recover unseen combinations of latent features.

Introduction

  • Disentangled representations are motivated by latent dimensions that need not encode semantically meaningful variations, making individual-dimension perturbations hard to interpret.Prior approaches include weak supervision and unsupervised modifications to the VAE objective, such as β-VAE.Source: §1 · pp. 1–2
  • HFVAEs disentangle discrete factors alongside continuous variables, improve over related methods, and recover latent-feature combinations absent from training examples.The authors evaluate on dSprites, MNIST, F-MNIST, CelebA, and 20NewsGroups; learned representations show interpretable features, and quantitative metrics demonstrate improvement over related methods.Source: §1 · p. 2
  • The HFVAE objective uses a two-level decomposition to control independence between variable groups and among individual variables within each group.At both levels, it induces statistical independence by minimizing total correlation (TC), a generalization of mutual information to more than two variables.Source: §1 · p. 2
  • Unlike related TC-based approaches [Kim and Mnih, 2018, Chen et al., 2018, Gao et al., 2018], HFVAEs apply a two-level decomposition to control independence.The paper also reinterprets the standard VAE objective as a KL divergence between generative and inference models, providing a unified perspective on trade-offs in objective modifications.Source: §1 · p. 2

2 A Unified View of Generalized VAE Objectives

The section reframes the VAE objective as a KL divergence between the generative and inference models, then decomposes it to expose trade-offs between conditional and marginal consistency. The decomposition clarifies what each term encourages and how removing terms changes the learned model.

2 A Unified View of Generalized VAE Objectives

  • The VAE combines a generative model pθ(x, z) with an encoder qφ(z | x), together forming a probabilistic autoencoder.The generative model uses a prior p(z) and likelihood pθ(x | z).Source: §2 · p. 2
  • The objective can be viewed as KL divergence between pθ(x, z) and qφ(z, x), minimized when the joint distributions and their data and latent marginals match.The inference marginal qφ(z) averages the encoder distribution over the empirical data distribution.Source: §2 · pp. 2–3
  • The four-term decomposition assigns terms 1 and 2 to conditional consistency: term 1 promotes identifiable codes, while term 2 regularizes it by minimizing I(z; x).Term 1 is intractable because pθ(x) cannot be evaluated pointwise; combining terms 1 and 3 recovers the likelihood objective.Source: §2 · p. 3
  • Terms 3 and 4 enforce marginal consistency: term 3 maximizes marginal likelihood, while term 4 makes inference marginal qφ(z) approximate prior p(z).Source: §2 · p. 3
  • Removing term 1 can let an expressive decoder ignore z, while removing term 2 encourages disjoint encodings and can drive I(x; z) to its log N upper bound.Term 2 often saturates at log N even when included; this suggests maximizing term 1 can outweigh its regularization cost in commonly used architectures.Source: §2 · p. 3
  • Removing term 3 or 4 permits pθ(x) to diverge from q(x) or qφ(z) to diverge from p(z), respectively.Source: §2 · p. 3

3 Hierarchically Factorized VAEs (HFVAEs)

HFVAEs decompose the objective to control independence both between groups of latent variables and within each group. The two-level objective separates mutual-information regularization from these total-correlation penalties, while optimization approximates the required inference marginals using minibatches.

3 Hierarchically Factorized VAEs (HFVAEs)

  • The KL penalty separates total-correlation terms that encourage independence from marginal-KL terms that match priors, and this decomposition can be recursively applied to form hierarchies of disentangled features.For grouped variables, the inner decomposition similarly separates total correlation from per-variable KL divergence; the paper restricts its objective to two levels.Source: §3 · p. 4, Figure 3 · p. 4
  • In the two-level HFVAE objective, α weights I(x;z), β weights between-group total correlation, and γ weights within-group total correlation; β=γ recovers a non-hierarchical case of related objectives [Kim and Mnih, 2018] [Chen et al., 2018].Unlike those related objectives, HFVAE also includes a component that matches total correlation in q(z) to that in p(z) when the prior is not factorized.Source: §3 · p. 4

3.1 Approximation of the Objective

  • For optimization, qφ(z) is estimated from the same minibatch as other objective terms; Jensen’s inequality yields a biased but consistent lower bound, with likely small bias at batch sizes of 512–1024.The estimate treats the observed data point deterministically and Monte Carlo-samples the remaining terms.Source: §3.1 · pp. 4–5

4 Related Work

The section situates the proposed objective among modifications to the VAE objective and other approaches to learning disentangled representations. It also describes uses of β-VAE objectives for reconstruction and models with both Concrete and Gaussian variables.

4 Related Work

  • The objective generalizes recent VAE-objective modifications; Zhao et al. [2017] eliminate mutual information and reweight KL, while Kumar et al. [2017] match the aggregate posterior and prior covariance.Source: §4 · p. 5
  • Gao et al. [2018] reduce mutual-information regularization for anchor variables, while Achille and Soatto [2018] add a total-correlation term from an information-bottleneck perspective.Source: §4 · p. 5
  • InfoGAN [Chen et al., 2016a] learns disentangled representations by maximizing mutual information between individual features and data under the generative model.Source: §4 · p. 5
  • β-VAE with β < 1 has been used to improve reconstruction quality when inducing disentanglement is not the primary aim [Alemi et al., 2016, Engel et al., 2017, Liang et al., 2018].Source: §4 · p. 5
  • When I(x; z) already saturates, reducing the relative weight of the KL term does not influence the learned representation; likelihood–KL tradeoffs and KL effects on mutual information are studied in depth [Alemi et al., 2018, Burgess et al., 2018].Source: §4 · p. 5
  • Dupont [2018] considered models with both Concrete and Gaussian variables, using a Burgess et al. [2018]-based objective that was not decomposed to obtain A.Source: §4 · p. 5

5 Experiments

Experiments evaluate HFVAE on image and text datasets, using priors and objective settings to control independence between latent groups and dimensions. The reported results cover interpretable factors, topic correlations, discrete-label separation, hyperparameter trade-offs, and reconstruction of unseen factor combinations.

5.2 Disentangled Representations for Text

  • HFVAE learns correlated topic groups while pruning to 11 topics with significant mutual information, whereas all 50 VAE topics have nonzero mutual information.On 20NewsGroups, the HFVAE uncovers related topics such as religion and Middle East politics; its latent dimensions also achieve higher disentanglement than the NVDM VAE baseline.Source: Figure 6 · p. 6, §5.2 · p. 8

5.1 Qualitative and Quantitative Evaluation

  • MNIST yields seven interpretable factors and zero mutual information in unused dimensions; F-MNIST and CelebA also yield interpretable factors, while dSprites scores are similar to other approaches.The reported MNIST factors include slant, width, height, openness, stroking, thickness, and roundness.Source: §5.1 · pp. 6–7, Figure 5 · p. 6

5.3 Unsupervised Learning of Discrete Labels

  • HFVAE separates digit identity from thickness, encoding label information in the discrete variable, while β-VAE spreads it across latent dimensions.How well discrete factors are disentangled depends on class separability: F-MNIST classes are visually distinct, whereas dSprites shapes are harder to distinguish.Source: Figure 7 · p. 7, §5.3 · p. 8

5.4 Hyperparameter Analysis

  • Across γ=β={2, 3, 4, 6}, TC({z, c}) decreases by 5 nats while I(x; {z, c}) decreases by 0.5 nats; increasing β at fixed γ=3 raises MIG.The analysis reports a positive correlation between mutual information and total correlation when γ=β, and motivates setting γ≠β for better disentanglement.Source: Figure 8 · p. 7, §5.4 · p. 9

5.5 Zero-shot Generalization

  • HFVAE reconstructs MNIST digits with stroke and width values absent from training and correctly encodes features from those unseen combinations.The evaluation removes selected factor combinations, retrains on the pruned data, and tests on the removed examples.Source: Figure 9 · p. 8, §5.5 · p. 9

6 Discussion … A.2 Mutual Information between label y and representation z

The discussion presents structured disentanglement as a step toward handling correlated, higher-dimensional factors and calls for datasets to test this capability. The appendix describes estimators for the inference-marginal entropy and mutual information involving labels and representations.

6 Discussion

  • The method can disentangle higher-dimensional variables by enforcing or relaxing independence between groups, while characterizing correlations remains future work.The discussion also calls for datasets that more rigorously test correlations between higher-dimensional variables.Source: §6 · p. 9

A.1 Approximating the Inference Marginal

  • The appendix derives a Monte Carlo estimator for the entropy of the inference marginal qφ(z), approximating expectations with samples from qφ(z).Source: §A.1 · p. 11
  • To approximate the marginal, the derivation pulls the term corresponding to the sampled input xᵇ out of the sum.Source: §A.1 · p. 11
  • The resulting estimator is unbiased, and the defined entropy estimator is an upper bound on Hφ(z).The upper-bound relationship follows from Jensen’s inequality.Source: §A.1 · p. 11

A.2 Mutual Information between label y and representation z

  • For continuous representation dimensions, the appendix bins each dimension into 10 equal-frequency bins and computes I(x; zd).Each bin contains 10% of the training data.Source: §A.2 · p. 11
  • For a concrete representation variable, the appendix defines the joint probability q(z = l, y = k) as q(y = k)q(z = l|y = k) when formulating mutual information.Source: §A.2 · p. 12

B Model Architectures

The section describes the four evaluation datasets and the latent-variable and optimization setup used across them.

B Model Architectures

  • Experiments covered dSprites, MNIST, F-MNIST, and CelebA [Liu et al., 2015].The datasets comprise 737,280 binary 64 × 64 shape images; 60,000 grayscale 28 × 28 digit images; 60,000 grayscale 28 × 28 clothing images across 10 classes; and 202,599 RGB 64 × 64 × 3 celebrity-face images, respectively.Source: B Model Architectures · p. 13
  • Each dataset used a continuous normal latent variable zc and a categorical Concrete latent variable zd.The two variables represent continuous factors and categories, respectively.Source: B Model Architectures · p. 13
  • The models were optimized with Adam at learning rate 1e-3 using the default settings.Source: B Model Architectures · p. 13

C Latent Traversals · D Disentangled Representation for Text

The section presents latent traversals for MNIST and F-MNIST and visualizes learned topics and mutual-information measures for 20NewsGroups. The MNIST traversals use 10% supervision to extract digits more reliably.

C Latent Traversals

  • Figure 10 varies one MNIST latent dimension from -3 to 3 with the others fixed at 0, using 10% supervision to extract digits more reliably.This visualizes the dataset’s style features.Source: Figure 10 · p. 14
  • Figure 11 shows F-MNIST traversals varying one latent dimension from -3 to 3 while keeping the others fixed at 0.Source: Figure 11 · p. 15
  • Figure 12 compares learned 20NewsGroups topics under HFVAE and VAE, showing frequent words, mutual information with x, and category-indicator mutual information.The frequent words are for the three most informative latent dimensions.Source: Figure 12 · p. 16

D.1 Model Architectures

This section describes the 20NewsGroups data representation and the model configurations used, with encoder and decoder architectures documented for ProdLDA and NVDM.

D.1 Model Architectures

  • 20NewsGroups contains 11,314 documents across 20 categories, represented as 2,000-dimensional bag-of-words vectors after Mallet stopword removal.Source: §D.1 · p. 16
  • HFVAE used two 25-dimensional latent variables, while ProdLDA and NVDM used Adam with learning rates of 1e-3 and 5e-5, respectively.ProdLDA used β1 = 0.99 and β2 = 0.999; NVDM used default settings.Source: §D.1 · p. 16
  • Encoder and decoder architectures for ProdLDA and NVDM are presented in Tables 7 and 8, respectively.Source: Table 7 · p. 16, Table 8 · p. 16

D.2 Neural variational document model

The HFVAE yields more disentangled document-category representations than the standard NVDM baseline, while a single feature can still capture two distinct topics through opposite-signed likelihood weights.

D.2 Neural variational document model

  • HFVAE latent dimensions are more disentangled than the standard NVDM baseline: document-category indicators generally correlate with only one latent feature.The HFVAE uses two 25-dimensional latent groups with β = 7 and γ = 4, allowing within-group but not across-group correlations.Source: §D.2 · p. 16
  • A single HFVAE feature can capture two distinct topics, represented by negative and positive weights in the likelihood model.The passage notes that only one of the two topics is shown.Source: §D.2 · pp. 16–17

D.3 Binary Indicator Variables for Document Category

The section defines binary document-topic indicators for 20NewsGroups and examines their mutual information with individual latent dimensions.

D.3 Binary Indicator Variables for Document Category

  • The authors formed 10 binary topic-membership indicators by grouping closely related newsgroups, then computed each indicator’s mutual information with individual latent dimensions.The mutual-information results are shown in Figure 12.Source: §D.3 · p. 17

Related papers

Loading 1804.02086v4…