Source-linked AI summary
Variational Auto-encoded Deep Gaussian Processes
Zhenwen Dai, Andreas Damianou, Javier González, Neil Lawrence
TL;DR
Exact inference in directed nonlinear probabilistic models is typically intractable, while proper parameter regularization remains an open question. The paper augments deep Gaussian processes with a recognition model and a distributed variational lower bound; it reports that the approach handles large datasets and competes with current alternatives.
Problem
Exact inference in directed nonlinear probabilistic models is typically intractable, and how to properly regularize their parameters remains an open question.
Source: §1 · p. 1Method
Results
The approach performs Bayesian inference using large datasets and competes with current alternatives.
Source: §6 · p. 9Takeaways & Limitations
The authors highlight the model's usefulness for unsupervised learning while describing it as general enough for supervised problems.
Source: §6 · p. 9Earlier tractable DGP inference experiments were limited to hundreds of datapoints, mainly due to covariance-matrix inversion costs and variational parameters growing with dataset size.
Source: §3 · p. 3
Abstract
from arXiv · showhide
We develop a scalable deep non-parametric generative model by augmenting deep Gaussian processes with a recognition model. Inference is performed in a novel scalable variational framework where the variational posterior distributions are reparametrized through a multilayer perceptron. The key aspect of this reformulation is that it prevents the proliferation of variational parameters which otherwise grow linearly in proportion to the sample size. We derive a new formulation of the variational lower bound that allows us to distribute most of the computation in a way that enables to handle datasets of the size of mainstream deep learning tasks. We show the efficacy of the method on a variety of challenges including deep unsupervised learning and deep Bayesian optimization.
1 INTRODUCTION
Probabilistic generative models can produce data-like samples and quantify uncertainty, but inference in nonlinear directed models is intractable and regularization remains an open question. The paper introduces a deep Gaussian process augmented with a recognition model and a distributed variational lower bound, and demonstrates the model on mainstream deep-learning datasets.
1 INTRODUCTION
- Probabilistic generative models can produce samples that mimic learned data and assess predictive uncertainty, properties relevant to uncertainty quantification and data imputation.The introduction also connects these models to probabilistic numerics.Source: §1 · p. 1
- Exact inference in directed nonlinear probabilistic models is typically intractable, and how to properly regularize model parameters remains an open question.The introduction contrasts dropout-based overfitting prevention with Bayesian inference as a mathematically grounded regularization framework.Source: §1 · p. 1
- The paper introduces VAE-DGP, extending deep Gaussian processes with a recognition model, and derives a distributed variational lower bound for mainstream deep-learning datasets.These are the paper's three stated contributions: the model extension, the bound, and demonstrations on mainstream datasets.Source: §1 · p. 1
- Figure 2 illustrates a layer-1 loop feature persisting through successive nonlinear warps and potentially transforming across the deep-GP hierarchy.The upper plot shows warping of a two-dimensional input space; red circles mark locations where the loop is created.Source: Figure 2 · p. 2
2 DEEP GAUSSIAN PROCESSES
Deep Gaussian processes compose Gaussian-process mappings across latent layers, increasing function flexibility; the section then motivates variational inference because exact DGP likelihood inference is intractable.
2 DEEP GAUSSIAN PROCESSES
- A deep GP stacks latent-variable layers and uses Gaussian processes to map between adjacent layers; unsupervised models place a unit-Gaussian prior on the top layer.Source: §2 · p. 2
- Successive latent-variable warping lets deep GPs model non-stationarities and non-parametric features, treating a deep network’s function composition as process composition.This offers greater expressive power than a standard GP, whose joint-normal assumption restricts its function class.Source: §2 · p. 2
2.1 VARIATIONAL INFERENCE
- Exact DGP inference must integrate latent variables and is intractable; the likelihood derivation targets unsupervised learning but extends to supervised learning with observed X_L.Source: §2.1 · p. 3
- The variational lower bound combines layerwise free-energy and entropy terms with a KL penalty between the top-layer posterior and prior.Source: §2.1 · p. 3
3 VARIATIONAL AUTO-ENCODED MODEL
VAE-DGP augments deep Gaussian processes with a recognition model that parameterizes latent-variable posteriors from observed data, avoiding variational parameters that grow linearly with dataset size. The approach also supports fast test-time inference and helps the model avoid local optima.
3 VARIATIONAL AUTO-ENCODED MODEL
- The recognition-model reparameterization prevents the number of variational parameters from growing linearly with dataset size.The earlier DGP approach was limited to experiments with hundreds of datapoints, partly because of this growth and expensive covariance-matrix inversion.Source: §3 · p. 3
- The recognition model constrains each latent-variable posterior as a transformation of observed data, enabling fast inference at test time.Its neural-network parameters replace individually optimized variational parameters.Source: §3 · p. 3
- From the same initialization, VAE-DGP’s recognition model helps avoid local optima and produces a better learned latent space than DGP.Figure 4a compares one-layer models on four MNIST digits with noisy backgrounds.Source: Figure 4 · p. 4, §3 · p. 4
- Posterior means are mapped from observed data through an MLP and, for hidden layers, from the lower layer’s posterior mean.The transformations are deterministic, while posterior variances are diagonal and shared across datapoints.Source: §3 · p. 4
- Bayesian inference regularizes the recognition model, so an overly complicated recognition model will not cause the generative model to overfit.This supports choosing a recognition model powerful enough for the task.Source: §3 · p. 5
4 DISTRIBUTED VARIATIONAL INFERENCE
The method derives a distributed variational lower bound for deep Gaussian processes, making most computation data-parallel. It uses inducing variables and recognition-model gradients to support optimization without centrally collecting per-datapoint variational gradients.
4 DISTRIBUTED VARIATIONAL INFERENCE
- For the observed layer, auxiliary inducing variables and inducing inputs yield a closed-form lower bound for kernels including linear and exponentiated quadratic.The optimal inducing-variable distribution is available in closed form, leaving the latent posterior and inducing inputs to optimize.Source: §4 · p. 5
- A distributed lower-bound formulation makes most computation data-parallel, allowing inference to scale to large datasets.The expensive free-energy lower-bound terms are reformulated for distribution.Source: §4 · p. 5
- The data-dependent terms can be summed across datapoints, with only intermediate results that do not scale with dataset size collected centrally.Related distributed BGPLVM formulations require central collection of per-datapoint variational-parameter gradients, which limits scalability.Source: §4 · pp. 5–6
- For hidden layers, the free-energy terms also involve expectations over output-variable posteriors and are reformulated into distributable forms.The first term naturally sums across datapoints, and the second is rewritten into a distributable form.Source: §4 · p. 6
- Gradients of the variational distributions are back-propagated through the recognition model to compute its parameter gradients.The closed-form variational lower bound is optimized with gradients of model and variational parameters.Source: §4 · pp. 5–6
5 EXPERIMENTS
The experiments evaluate VAE-DGP on unsupervised image learning and imputation, supervised regression, and Bayesian optimization. The reported Branin experiment shows a gain over a standard GP, especially early in optimization.
5.1 UNSUPERVISED LEARNING
- On Frey-Yale, VAE-DGP generates face samples by drawing from a 2D unit-Gaussian top layer and sampling down through the model.The experiment combines Frey and Yale face images resized to 20 × 28 pixels.Source: §5.1 · p. 7
- MNIST evaluation compares VAE-DGP test log-likelihood across one-, two-, and three-hidden-layer settings with literature baselines.The test set contains 10,000 images; the model was trained on all 60,000 training images.Source: §5.1 · p. 7, Table 1 · p. 7
5.2 DATA IMPUTATION
- Imputation experiments hide half of test images, with Figure 6 showing inputs, completions, and originals for visual comparison.The paper notes that multiple completions can be plausible; experiments also include SVHN.Source: §5.2 · p. 7, Figure 6 · p. 7
5.3 SUPERVISED LEARNING AND BAYESIAN OPTIMIZATION
- For Abalone and Creep regression, the study compares MSE from VEA-DGP with standard GP and linear regression.The experiments use typical training splits of 1,000 Abalone and 800 Creep instances.Source: Table 2 · p. 8, §5.3 · p. 8
- Because supervised datasets are relatively small, the experiments omit recognition-model back-constraints and use a one-hidden-layer VEA-DGP.The setup uses an exponentiated-quadratic covariance function.Source: §5.3 · p. 8
- In Branin Bayesian optimization, VEA-DGP gains over a standard GP, especially in the first optimization steps.The authors attribute the gain to handling non-stationarity and representing richer, potentially multimodal output distributions.Source: §5.3 · p. 8, Figure 7 · p. 8
6 CONCLUSION
The conclusion presents VAE-DGP as a deep non-parametric generative model that simplifies optimization through layer-wise MLP parameterization and supports distributed computation. It reports performance on large datasets and identifies extensions for future work.
6 CONCLUSION
- VAE-DGP extends deep Gaussian processes with layer-wise multilayer perceptrons, significantly simplifying optimization.The conclusion emphasizes unsupervised learning while noting that the model is general enough for supervised and unsupervised problems.Source: §6 · p. 9
- A reformulated lower bound permits distributed computation, and the approach performs Bayesian inference on large datasets while competing with current alternatives.Source: §6 · p. 9
- Future directions include regularizing perceptron weights, adapting the setup to multi-view problems, and incorporating convolutional structures.Source: §6 · p. 9