PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations

Diane Bouchacourt, Ryota Tomioka, Sebastian Nowozin

arXiv:1705.08841v1cs.LGstat.ML

TL;DR

The paper asks how minimal grouping supervision can produce disentangled representations when standard probabilistic models assume i.i.d. observations. It introduces ML-VAE, which separates group-shared content from observation-level style through group-aware amortised inference. Experiments report meaningful disentanglement, latent manipulation, and generalisation to unseen groups.

  • Problem

    The central gap is learning semantically meaningful, independently controllable factors from grouped observations when existing VAEs assume i.i.d. data.

  • Method

    ML-VAE uses group-level supervision to separate shared content from observation-specific style while retaining amortised inference.

  • Results

    ML-VAE learns semantically meaningful disentanglement, enables latent-representation manipulation, and generalises to unseen groups.

  • Takeaways & Limitations

    Grouping information can anchor semantics in a disentangled representation that supports manipulation and applies to groups not seen during training.

  • Takeaways & Limitations

    With nonlinear Product-of-Normals or Mixture-of-Normals content approximations, subsampled-group gradient estimates are biased and require future correction analysis.

Abstract

from arXiv · show

We would like to learn a representation of the data which decomposes an observation into factors of variation which we can independently control. Specifically, we want to use minimal supervision to learn a latent representation that reflects the semantics behind a specific grouping of the data, where within a group the samples share a common factor of variation. For example, consider a collection of face images grouped by identity. We wish to anchor the semantics of the grouping into a relevant and disentangled representation that we can easily exploit. However, existing deep probabilistic models often assume that the observations are independent and identically distributed. We present the Multi-Level Variational Autoencoder (ML-VAE), a new deep probabilistic model for learning a disentangled representation of a set of grouped observations. The ML-VAE separates the latent representation into semantically meaningful parts by working both at the group level and the observation level, while retaining efficient test-time inference. Quantitative and qualitative evaluations show that the ML-VAE model (i) learns a semantically meaningful disentanglement of grouped data, (ii) enables manipulation of the latent representation, and (iii) generalises to unseen groups.

1 Introduction

The paper addresses how to learn semantically meaningful, disentangled representations from minimally supervised grouped observations rather than assuming i.i.d. data. It proposes ML-VAE, which separates group-level and observation-level factors while retaining amortised inference.

  • Motivation: Grouped observations provide inexpensive weak supervision because samples share an unknown factor, allowing semantics such as shape and color to anchor the representation.The grouping factor need not be identified explicitly.
  • Motivation: Existing VAE models assume i.i.d. observations, so they do not directly exploit grouping information.VAE amortised inference enables efficient test-time inference, but its i.i.d. assumption limits use of grouped data.
  • Proposed approach: ML-VAE extends amortised inference to non-i.i.d. grouped observations while retaining efficient test-time inference.The model uses grouping information without resorting to the expensive iterative inference associated with SVI.
  • Proposed approach: ML-VAE separates latent representations into semantically meaningful style and content parts by operating at both group and observation levels.Content is shared within a group, while style can differ across observations.
  • Contributions: The paper reports that ML-VAE learns meaningful disentanglement, enables latent manipulation, and generalises to unseen groups.These outcomes are presented as experimental contributions.

3 Model

The ML-VAE models grouped observations with group-shared content and observation-specific style, using amortised variational inference. Its group content posterior accumulates evidence from encoded observations, while the training objective is a group-level ELBO.

  • Grouped model: For grouped observations, ML-VAE assigns one shared content variable C_G to each group and independent style variables S_i to its observations.The likelihood decomposes across samples given group content and individual styles.
  • Grouped model: Unlike SVI, ML-VAE performs amortised inference for grouped, non-i.i.d. observations, avoiding iterative test-time inference.SVI also requires more passes over training data.
  • Objective: The group marginal log-likelihood equals the group ELBO plus the KL divergence between the variational approximation and the true posterior.Because the KL divergence is nonnegative, the ELBO is a lower bound on the marginal log-likelihood.
  • Objective: Training maximises an average group ELBO estimated from minibatches of groups rather than the average ELBO over individual samples.This objective incorporates the grouped-observation structure.
  • Accumulating group evidence: The content variational approximation is formed by multiplying normal densities produced from each observation’s encoding.The resulting product is proportional to a normal distribution with group-level mean µ_G and variance Σ_G.
  • Accumulating group evidence: Increasing group size decreases the resulting content variance, which the paper calls accumulating evidence.The variance is inversely proportional to the sum of the observations’ inverse variances.

4 Experiments

The experiments evaluate ML-VAE on image data using disentanglement operations, standard datasets, and two strategies for incorporating grouped test observations. Results show meaningful latent control, generation and interpolation, while accumulating group evidence improves reconstructions.

  • Experimental setup: Experiments evaluate ML-VAE on images, including MNIST grouped by digit label and face-aligned MS-Celeb-1M grouped by identity.The study also compares test-time inference with and without using multiple images from a group.
  • Experimental setup: The training procedure samples minibatches of groups, constructs the shared content posterior from group observations, samples content and style, and decodes each observation.The algorithm updates θ, φc, and φs using gradients of the group objective.
  • Qualitative evaluation: Swapping tests whether rows preserve style and columns preserve content, while generation and interpolation assess latent-space structure and data-manifold coverage.The qualitative evaluation uses latent manipulations to assess whether the learned representation is relevant and disentangled.
  • Qualitative evaluation: Accumulating evidence from multiple test images improves inference, correcting a digit label and making same-identity facial traits more constant across age variation.The comparison contrasts reconstructions without group information against reconstructions using evidence from four images.
  • Qualitative evaluation: ML-VAE encodes identity into facial traits that remain constant across style swaps, while background color and face orientation vary with style on MS-Celeb-1M.The model achieves this without being told that the groups represent identity.

5 Discussion

The discussion concludes that ML-VAE learns meaningful disentanglement from grouped observations and supports varying group structures between training and testing. Experiments further demonstrate latent control and generalization to unseen groups.

  • Discussion: ML-VAE handles an arbitrary number of observation groups, which need not be the same at training and testing.The model is presented as applicable when group structures differ across these stages.
  • Discussion: Experiments show semantically meaningful disentanglement, generalization to unseen groups, and control over the latent representation.The paper identifies applying ML-VAE to text data as future work.

Grouped Observations Supplementary Material

This supplementary section identifies the authors and their affiliation with the Machine Intelligence and Perception Group in Cambridge, UK.

  • Authorship and affiliation: The paper lists Ryota Tomioka and Sebastian Nowozin as authors from the Machine Intelligence and Perception Group.Their affiliation is given as Cambridge, UK.

1 Mixture of Normals Method

The Mixture of Normal method constructs the group-level content posterior by combining per-observation densities and estimates its intractable divergence through sampling. Qualitative results favor the Product of Normal method for disentanglement, while retaining the mixture as a possible alternative for other settings.

  • Mixture construction: The Mixture of Normal method constructs q(CG|XG, φc) as a mixture of |G| densities, each evaluating content given one observation.This is presented as an alternative to the Product of Normals method.
  • Mixture construction: The per-observation content posterior is assumed Normal, but the KL divergence to the prior cannot be computed in closed form.The approximation uses q(CG|Xi = xi, φc) = N(µi, Σi).
  • Estimation: The intractable divergence term is estimated by sampling L content values from q(CG|XG, φc).The supplementary method describes Monte Carlo estimation of this term.
  • Estimation: The experiments use L = |G|, reusing samples drawn for the first term of the objective.This choice ties the sampling count to the group size.
  • Comparison: Qualitative evaluation favors Product of Normal for disentanglement, whereas Mixture of Normal can store grouping information in style on MS-Celeb-1M.When style is transferred, facial features that should remain constant for an identity tend to change; the mixture may suit other datasets or tasks.

2 Experimental Details

The experiments use MNIST and MS-Celeb-1M architectures with separate style and content encoders, group-content construction, and a shared decoder. Comparisons include ML-VAE variants and stochastic variational inference.

  • MNIST: MNIST uses a 500-unit encoder followed by separate linear layers for style and content means and variances.Each latent representation has dimension d, and the decoder receives concatenated group content and individual style codes.
  • ML-VAE architecture: The ML-VAE constructs group content by sampling a shared content code and an individual style code for each observation before decoding.The decoder maps the concatenated 2 × d-dimensional representation to Normal-distribution parameters for the observation.
  • MS-Celeb-1M: MS-Celeb-1M uses convolutional encoder layers followed by linear layers that produce style and content representations.The convolutional layers use stride 2, kernel size 4, batch normalization, and ReLU activations.
  • MS-Celeb-1M: The MS-Celeb-1M decoder uses deconvolutional layers to output the mean and log-variance of the observation model.The decoder includes batch normalization and ReLU activations in its intermediate layers.
  • Comparison model: The experiments compare ML-VAE against SVI, whose encoder uses non-shared parameters for each observation’s style and each group’s content.The SVI decoder is the same as the ML-VAE decoder.

3 Quantitative Evaluation details

Disentanglement is evaluated with a classifier-based conditional-entropy proxy using latent codes from grouped observations. The MNIST evaluation considers group sizes up to K = 100.

  • Classifier evaluation: A two-layer neural classifier with cross-entropy loss is trained on test-image features to evaluate information about the class in each latent code.The classifier uses 256 hidden units in each linear layer and a softmax output.
  • Classifier evaluation: Minimizing classifier cross-entropy provides an upper-bound proxy for minimizing the conditional entropy of the class given a latent code.The same procedure is applied to content, style, and original-VAE latent representations.
  • Reported metric: The reported conditional entropy is the classifier test-set value of Ep(Y,C)[log r(Y|C)] expressed in bits.The paper reports analogous values for style codes and the original VAE representation.

4 ML-VAE with Product of Normal Without Accumulating Evidence

The Product-of-Normals variant accumulates group evidence by multiplying per-observation Normal densities for shared content. Qualitative swapping and interpolation keep one factor fixed while varying the other.

  • Qualitative evaluation: Swapping uses rows with fixed style and columns with fixed content, distinguishing test samples, reconstructions, and swapped reconstructions by color.Green boxes mark test samples, blue boxes reconstructions, and red boxes swapped reconstructions.
  • Qualitative evaluation: Interpolation varies content while fixing style across rows, or varies style while fixing content across columns.Figure 4 presents these interpolations without accumulating group evidence.
  • Product of Normal: Group content is constructed by multiplying |G| Normal densities, one for each observation’s inferred content distribution.The product is normalized to form a valid density over the shared group content variable.
  • Product of Normal: The product of the per-observation Normal densities is proportional to a Normal distribution with group-level mean µG and variance ΣG.The resulting distribution is scaled by a normalization factor before being rescaled to integrate to 1.

6 Bias of the Objective

Using all observations in each group gives an unbiased objective estimate, but subsampling large groups introduces bias because the nonlinear content approximation does not preserve the full-group gradients.

  • Full groups: The objective estimate is unbiased when every observation in each group is included.Subsampling is introduced for efficiency when groups are too large.
  • Subsampled groups: Subsampling a group produces a biased estimate of its Evidence Lower Bound when the group is too large to process in full.The estimate is computed from a subsample H of the group G.
  • Source of bias: The gradients with respect to model and content-inference parameters do not decompose unbiasedly under Product- or Mixture-of-Normals approximations.Summing gradients from subsampled groups does not recover the gradient computed using the entire group.
  • Implications: The bias depends on the employed approximation method, and future work proposes correcting it and testing whether it overestimates the true objective.The relevant true objective is the sum of the groups’ Evidence Lower Bounds.

7 Stochastic Variational Inference (SVI) Results

SVI disentangles style and content in the swapping evaluation, but produces poor image quality and requires costly training and test-time inference compared with ML-VAE.

  • SVI disentangles style and content in the swapping evaluation, but its resulting image quality is poor.The evaluation swaps fixed styles and contents across reconstructed samples.
  • SVI required 500 training epochs on MS-Celeb-1M, yet its final average group Evidence Lower Bound remained below ML-VAE.
  • SVI lacks shared encoder parameters across observations, leading to slower training and expensive iterative inference at test time.The MS-Celeb-1M experiment used 200 test-inference epochs, with potentially better quality requiring more.

8 Other Formulations Explored

The paper explores encoder-based and regularized alternatives to ML-VAE. SVI-Encode restores amortized test-time inference but depends on generative-model quality, while regularization can collapse content representations to the prior.

  • SVI-Encode: SVI-Encode trains an encoder on generated observations to approximate the latent representation of a trained SVI generative model.It uses no training data and receives no group information.
  • SVI-Encode: SVI-Encode gives satisfactory qualitative results on MNIST but poor results on MS-Celeb-1M because it depends on generative-model quality.
  • Regularising the objective: A regularized VAE enforces within-group content sharing with a symmetrized Kullback-Leibler divergence penalty.Each observation retains its own style and content latent variables.
  • Regularising the objective: With latent representation size 100 on MS-Celeb-1M, the regularized model set content representations to the prior to avoid the penalty, whereas ML-VAE was more robust.The content information was consequently encoded only in the style representation.
Loading 1705.08841v1…