Source-linked AI summary
Learning Disentangled Representations with Semi-Supervised Deep Generative Models
N. Siddharth, Brooks Paige, Jan-Willem van de Meent, Alban Desmaison, Noah D. Goodman, Pushmeet Kohli, Frank Wood, Philip H. S. Torr
TL;DR
The paper asks how VAEs can learn disentangled, interpretable factors without requiring a fully specified generative model. It introduces partially specified graphical models with a generalized semi-supervised objective, showing strong performance across visual tasks while leaving unspecified variation to flexible latent variables.
Problem
Representations learned for individual tasks require repetitive effort, while flexible deep generative models do not ensure that latent variables acquire interpretable, disentangled semantics.
Method
The framework combines structured graphical-model constraints for supervised variables with unstructured latent variables and a generalized semi-supervised VAE objective.
Results
The framework shows strong performance learning disentangled representations semi-supervised across a variety of visual tasks and datasets.
Takeaways & Limitations
Partially specified models constrain recognition networks to make predictions in an interpretable, disentangled space while retaining flexibility for other variation.
Takeaways & Limitations
The framework assumes supervision labels are available for some interpretable variables, while other latent variables are inferred fully unsupervised.
Abstract
from arXiv · showhide
Variational autoencoders (VAEs) learn representations of data by jointly training a probabilistic encoder and decoder network. Typically these models encode all features of the data into a single variable. Here we are interested in learning disentangled representations that encode distinct aspects of the data into separate variables. We propose to learn such representations using model architectures that generalise from standard VAEs, employing a general graphical model structure in the encoder and decoder. This allows us to train partially-specified models that make relatively strong assumptions about a subset of interpretable variables and rely on the flexibility of neural networks to learn representations for the remaining variables. We further define a general objective for semi-supervised learning in this model class, which can be approximated using an importance sampling procedure. We evaluate our framework's ability to learn disentangled representations, both by qualitative exploration of its generative capacity, and quantitative evaluation of its discriminative ability on a variety of models and datasets.
1 Introduction
The paper addresses the tension between explicitly structured models, which can yield disentangled semantics but are difficult to design, and flexible deep generative models, which avoid feature engineering but do not guarantee interpretable factors. It proposes partially specified graphical models and a generalized semi-supervised VAE objective to constrain selected variables while learning the remaining representation flexibly.
- Motivation: The framework is motivated by the goal of learning factorized representations whose parts retain consistent semantics and can generalize across tasks.Learning separate representations for every task is described as wasteful repetitive effort.
- Motivation: Explicit graphical models can provide disentangled semantics through their structure, but designing suitable likelihoods for complex domains is difficult.Deep generative models instead use neural networks as flexible function approximators, avoiding much feature engineering.
- Proposed framework: Partially specified graphical models let researchers define relationships for selected variables while learning undefined factors with flexible neural networks.This places models along a spectrum between fully structured and more entangled representations.
- Semi-supervised learning: Small amounts of supervision can anchor interpretable factors such as MNIST digit identity while leaving style-related variation unspecified.Without labels, a model with ten classes is not guaranteed to recover the ten digits.
- Proposed framework: The framework generalizes VAEs by supporting arbitrary graphical dependencies, continuous latents, dynamically changing dependencies, and automatically factorized semi-supervised objectives.The objective can be compiled into stochastic computation graphs for end-to-end training and amortized inference.
2 Framework and Formulation
The framework generalizes VAEs to arbitrary graphical structures with interpretable supervised variables and unsupervised latent variables. A generalized semi-supervised objective uses importance sampling so these models can be trained through stochastic computation graphs.
- 2 Framework and Formulation: VAEs in this framework allow arbitrary conditional dependency structures and multiple distribution types for generative and approximate-posterior models.The model retains interpretable variables y with partial supervision and unsupervised variables z.
- 2.1 Objective Function: Semi-supervised training combines unsupervised and supervised evidence terms, with γ controlling the relative strength of the supervised contribution.This weighting helps when the numbers of unsupervised and supervised examples differ substantially.
- 2.1 Objective Function: The supervised objective is generalized beyond factorisations that require evaluating qφ(z|x,y) or qφ(y|x) directly.The derivation targets arbitrary conditional dependence structures in qφ(y,z|x).
- 2.1 Objective Function: Self-normalised importance sampling uses proposals from qφ(z|x) to estimate the re-expressed supervised objective without evaluating qφ(z|x,y).The estimator uses unnormalised weights and a normaliser, with samples generated from the unconditioned encoder distribution.
- 2.1 Objective Function: For a general encoder factorisation, importance weights are formed from conditional probabilities of the supervised variables.In the special case considered by Kingma et al., the weights reduce to constants wm,s = qφ(ym|xm).
- 2.2 Construction of the Stochastic Computation Graph: The computation graph represents each random variable with a stochastic sub-graph, supporting fully supervised, partially supervised, and unsupervised variables.The graph is used to perform gradient ascent on the importance-sampling objective.
- 2.2 Construction of the Stochastic Computation Graph: In the MNIST example, y represents the digit and z captures remaining latent information such as handwriting style and stroke thickness.The generative prior treats y and z as independent, while the recognition model conditions z on y and x.
- 2.2 Construction of the Stochastic Computation Graph: The graph specification assigns distribution types, parameter functions, recognition functions, and reparameterized sampling procedures before repeated forward passes estimate and optimize the objective.Partially observed y is sampled with a Gumbel-softmax relaxation when labels are unavailable.
3 Experiments
Experiments evaluate the framework on classification, disentangled generation, structured face representations, and compositional stochastic models. Results show competitive classification, interpretable identity and lighting variables, and reliable Multi-MNIST counting, reconstruction, and decomposition under partial supervision.
- Overview: The experiments evaluate classification, generative semantics, and increasingly complex structured representations across MNIST, SVHN, intrinsic faces, and Multi-MNIST.The evaluation includes supervised and semi-supervised settings, with standard architectures and optimization choices across datasets.
- MNIST and SVHN: With only partial supervision, the model transforms inputs into a disentangled space and manipulates digit identity while keeping style fixed.MNIST uses 100 labelled examples out of 50,000, while SVHN uses 1,000 labelled examples; the generated digits retain expected visual characteristics.
- MNIST and SVHN: A modicum of over-representation improves generalisation for sparsely labelled data, whereas excessive over-representation leads to overfitting.The supervision weight is controlled by ρ = γM/(N + γM), with the classification-objective scaling fixed separately for MNIST and SVHN.
- MNIST and SVHN: The model performs on par with Kingma et al.’s setup on MNIST and SVHN classification, validating the importance-sampled objective in this special case.The comparison uses direct training on the data without pre-processing or pre-learning; for SVHN, the baseline comparison is a two-stage process.
- Intrinsic Faces: Despite independent latent priors and no hand-specified generative structure, the model learns the intended relationships among identity, lighting, shading, and reflectance.The result is attributed to structure in the recognition model combined with partial supervision.
- Intrinsic Faces: The intrinsic-faces model learns identity and lighting representations for direct identity classification and lighting-direction regression under partial supervision.The recognition model uses categorical identity and continuous lighting variables, and Figure 4 evaluates both generative manipulation and quantitative performance.
- Multi-MNIST: In Multi-MNIST, the framework predicts a stochastic digit count, reconstructs inputs, and decomposes images into constituent digits.The recognition model predicts K from pixels and represents each constituent digit with a Bernoulli-distributed image variable.
4 Discussion and Conclusion
The paper presents partially specified graphical models and semi-supervised VAEs for learning disentangled representations. Across visual tasks, the framework shows strong semi-supervised performance and produces interpretable recognition spaces, while its implementation also points toward more expressive probabilistic-programming extensions.
- Discussion and Conclusion: The framework combines structured graphical models with unstructured latent variables in VAEs to learn disentangled representations through partial supervision.Its recognition networks make predictions in an interpretable space constrained by graphical-model structure and weak supervision.
- Discussion and Conclusion: Applications across visual tasks demonstrate strong semi-supervised performance and flexibility for learning disentangled representations.The paper evaluates the approach on a variety of tasks rather than a single experimental setting.
- Discussion and Conclusion: The PyTorch implementation constructs stochastic computation graphs and motivates future extensions toward probabilistic programming with recursive structures and higher-order functions.These extensions are described as a future direction beyond the presented framework.