Auxiliary Deep Generative Models

Lars Maaløe; Casper Kaae Sønderby; Søren Kaae Sønderby; Ole Winther

Source-linked AI summary

Auxiliary Deep Generative Models

Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, Ole Winther

arXiv:1602.05473v4Feb 17, 2016stat.MLcs.AIcs.LG

TL;DR

The paper addresses limitations of mean-field variational distributions in deep generative models. It introduces auxiliary variables and a skip-connected two-stochastic-layer model, achieving state-of-the-art semi-supervised performance while remaining end-to-end trainable. The discussion notes computational costs for unlabeled multiclass data and that the presented models use Gaussian latent distributions.

Problem
Mean-field latent distributions assume independent latent variables, which may limit the dependencies captured in data.
Method
ADGM introduces auxiliary variables that leave the generative model unchanged while enabling richer variational distributions, and SDGM adds two stochastic layers with skip connections.
Results
The models achieve state-of-the-art performance on several semi-supervised benchmarks and can be trained end-to-end without pre-training or feature engineering.
Takeaways & Limitations
Auxiliary variables can fit complex latent distributions, improve the variational lower bound, and support semi-supervised classification.
Takeaways & Limitations
Evaluating the unlabeled-data variational bound requires summing over all classes, which becomes computationally costly as the number of classes grows.

Abstract

from arXiv · show

Deep generative models parameterized by neural networks have recently achieved state-of-the-art performance in unsupervised and semi-supervised learning. We extend deep generative models with auxiliary variables which improves the variational approximation. The auxiliary variables leave the generative model unchanged but make the variational distribution more expressive. Inspired by the structure of the auxiliary variable we also propose a model with two stochastic layers and skip connections. Our findings suggest that more expressive and properly specified deep generative models converge faster with better results. We show state-of-the-art performance within semi-supervised learning on MNIST, SVHN and NORB datasets.

1. Introduction

The paper introduces auxiliary-variable deep generative models that make variational distributions more expressive while preserving the generative model, and evaluates them for semi-supervised learning. ADGM and SDGM provide end-to-end trainable models with strong benchmark performance and improved distribution fitting.

Mean-field latent distributions assume independent latent variables, potentially limiting their ability to capture dependencies in data.
ADGM adds auxiliary latent variables to make variational distributions more flexible, while SDGM uses two stochastic layers with skip connections.Both models are designed for end-to-end training.
Auxiliary-variable models fit complex latent distributions and improve the variational lower bound.
Using six labeled half-moon data points, ADGM utilizes the data manifold for semi-supervised classification.
The models achieve state-of-the-art results on several semi-supervised benchmarks, including MNIST, SVHN, and NORB.The supplied passages identify SVHN and NORB and state that the benchmark results are state of the art.
Multi-layered deep generative models are trainable end-to-end without pre-training or feature engineering.

2. Auxiliary deep generative models

The paper augments variational inference with auxiliary variables that preserve the generative model while making the approximate posterior more expressive. It applies this design to semi-supervised models and introduces a skip-connected two-layer stochastic variant.

2. Auxiliary deep generative models: The hierarchical posterior q(z|a, x)q(a|x) can represent general non-Gaussian distributions and correlate latent variables through a.This retains the computational efficiency associated with factorized components while increasing posterior flexibility.
2. Auxiliary deep generative models: Auxiliary variables extend the variational distribution while leaving the original generative model unchanged under marginalization.The joint model is specified as p(x, z, a) = p(a|x, z)p(x, z), while q(z|x) is obtained by integrating q(z|a, x)q(a|x).
2.3. Semi-supervised learning: The semi-supervised ADGM introduces a latent class variable y alongside auxiliary variable a and latent features z.Its generative model includes p(y), p(z), pθ(a|z, y, x), and pθ(x|y, z), while inference uses qφ(a|x)qφ(z|a, y, x)qφ(y|a, x).
2.3. Semi-supervised learning: The auxiliary unit acts as a latent feature extractor that provides a richer mapping between x and y for classification.The classifier estimates class probabilities for unlabeled data and supplies a cross-entropy term for labeled data alongside the variational lower bound.
2.3. Semi-supervised learning: Training maximizes a variational lower-bound objective over labeled and unlabeled data, optionally including an explicit classification loss weighted by α.The objective combines labeled-data and unlabeled-data terms, with α controlling generative versus discriminative learning.
2.4. Two stochastic layers with skip connections: The SDGM reverses the auxiliary-to-observation connection to form a two-layer stochastic generative model with skip connections while retaining the ADGM inference distribution.The paper tests this model alongside ADGM because the augmented generative structure is expected to be more flexible.

3. Experiments

The experiments use neural networks with stochastic-variable outputs and reparameterization-based sampling across toy and benchmark settings. Benchmark configurations cover MNIST, SVHN, and NORB with specified latent dimensions, labeled subsets, and optimization settings.

3. Experiments: ADGM and SDGM each use five neural networks covering auxiliary inference, latent inference, classification, and two generative components.The inference models are qφ(a|x), qφ(z|a, y, x), and qφ(y|a, x); the generative models are pθ(a|·) and pθ(x|·).
3. Experiments: Each network uses fully connected hidden layers with rectified linear activations and separate μ and log σ^2 outputs for stochastic variables.The stochastic variables are approximated using the reparameterization trick.
3. Experiments: The toy experiments use three hidden layers with dim(h) = 20, dim(a) = 4, and dim(z) = 2 for unsupervised data.The semi-supervised toy setting instead uses two hidden layers of dim(h) = 100 and dim(a, z) = 10.
3. Experiments: Benchmark models use two fully connected hidden layers with hidden sizes of 500 or 1000 and dim(a, z) of 100 or 300.ADGM uses p(y)p(z)pθ(a|z, y)pθ(x|z, y), while SDGM augments the observation model with a.
3. Experiments: Training uses Monte Carlo reparameterization for a and z, exact enumeration over y, and Adam with learning rate 3e-4.The β scaling constant ranges from 0.1 to 2 across experiments.
3. Experiments: The benchmark datasets are MNIST, SVHN, and NORB, with 100 labeled MNIST samples, 604388 SVHN training points, and 1000 labeled NORB samples.NORB contains 24300 training samples across five classes, while MNIST uses 50000 training and 10000 validation examples combined.

4. Results

The experiments show that auxiliary-variable generative models can fit complex latent posteriors, improve convergence, and achieve strong semi-supervised classification performance across benchmark datasets.

4.1. Beyond Gaussian latent distributions: The auxiliary model fits a complex multimodal latent posterior, although optimization most often selects an equivalent single mode.The one- and two-mode solutions have identical bound values, making the simpler solution easier to infer.
4.3. Generative log-likelihood performance: On permutation-invariant MNIST, the AVAE outperforms the VAE with normalizing flows and the importance-weighted auto-encoder in reported log-likelihood comparisons.The methods are not directly comparable except for the Ladder VAE because training procedures differ.
MNIST experiments: On MNIST, ADGM converges to around 2% error in about 200 epochs and eventually reaches 0.96%, while SDGM accelerates convergence and is more stable.The best MNIST result used multiple Monte Carlo samples for the auxiliary and latent variables.
MNIST experiments: The SDGM learns separated style and class information from only 100 labeled MNIST examples and generates samples for each class.The auxiliary latent space shows clustering, while information flows through both auxiliary and ordinary stochastic variables.
SVHN and NORB experiments: The SDGM reduces SVHN error by more than 30% relative to VAT and is slightly better than VAT on NORB, where improvements are less significant.The model reaches around 19% error on SVHN and around 12% on NORB after 100 epochs; without SVHN extra data, error is 29.82%.

5. Discussion

The models are trainable end-to-end and achieve strong performance, but the framework has computational and architectural limitations that motivate extensions.

ADGM and SDGM use relatively simple neural network architectures while remaining trainable end-to-end.
Summing over all classes for unlabeled data makes evaluating the semi-supervised variational bound computationally costly as the number of classes grows.
The framework currently uses fully connected layers, leaving convolutional layers as a possible avenue for improving performance.

6. Conclusion

The paper introduces a framework that makes variational distributions more expressive and demonstrates its effectiveness in semi-supervised benchmarks.

The framework uses auxiliary variables to learn better variational approximations in toy examples and benchmarks.
The framework achieves state-of-the-art performance across several semi-supervised benchmarks and supports end-to-end training.

A. Auxiliary model specification

The auxiliary-model analysis characterizes optimal auxiliary distributions and shows that uninformed auxiliaries can collapse to the ordinary latent-variable model, with weaker benchmark performance.

The theoretical analysis studies the optimum of the auxiliary variational bound for auxiliary variable a and latent variable z.
The optimal conditional auxiliary distribution is p(a|z) = q(a, z)/q(z), which is generally intractable because it requires marginalization.
For an uninformed auxiliary model, optimizing p(a) yields p(a) = q(a).
Optimizing q(a) produces a δ-function and reduces the model to one without the auxiliary variable.
The uninformed auxiliary model performs competitively on MNIST but not on the other two benchmarks.
Fixing p(a) to a zero-mean unit-variance Gaussian and using deep iid models can yield a local optimum distinct from collapse to the pure z-model.

B. Variational bounds

The appendix defines the generative and variational distributions and describes labeled and unlabeled variational objectives, including treatment of the class variable as latent for unlabeled data.

The ADGM and SDGM generative model is defined using pθ(x, a, y, z).
The labeled lower bound −L(x, y) is defined for the variational objective.
The variational distribution factorizes as qφ(a, z|x, y) = qφ(a|x)qφ(z|a, y, x).
For unlabeled data, the discrete class variable y is treated as a latent variable in the lower bound −U(x).
The labeled and unlabeled objective functions are given in Eq. (12) and Eq. (13).
The class variable y is assumed to be multinomial but can be extended to different distributions.