Source-linked AI summary
Deep Gaussian Processes
Andreas C. Damianou, Neil D. Lawrence
TL;DR
Deep models traditionally face data requirements and inference difficulties, especially when latent variables and deep hierarchies are involved. The paper introduces deep Gaussian processes with variational marginalization, obtaining a marginal-likelihood lower bound for model selection. The method supports deep hierarchies in scarce-data settings, with a 5-layer model selected for a digit dataset containing 150 examples.
Problem
Deep models often require substantial data, while small datasets raise questions about generalization and whether deep structures are justified objectively.
Method
The paper recursively stacks Gaussian-process mappings and uses variational inference to marginalize latent variables across arbitrarily many hidden layers.
Results
A variational lower bound on marginal likelihood supports model selection, and the best bound in a digits example came from the deepest tested hierarchy, with 5 layers.
Takeaways & Limitations
Deep Gaussian-process models can be evaluated and selected using a fully Bayesian variational objective even when data is scarce.
Takeaways & Limitations
Deep-model inference remains challenging because latent-variable marginalization is generally intractable in existing GP and RBM approaches.
Abstract
from arXiv · showhide
In this paper we introduce deep Gaussian process (GP) models. Deep GPs are a deep belief network based on Gaussian process mappings. The data is modeled as the output of a multivariate GP. The inputs to that Gaussian process are then governed by another GP. A single layer model is equivalent to a standard GP or the GP latent variable model (GP-LVM). We perform inference in the model by approximate variational marginalization. This results in a strict lower bound on the marginal likelihood of the model which we use for model selection (number of layers and nodes per layer). Deep belief networks are typically applied to relatively large data sets using stochastic gradient descent for optimization. Our fully Bayesian treatment allows for the application of deep models even when data is scarce. Model selection by our variational bound shows that a five layer hierarchy is justified even when modelling a digit data set containing only 150 examples.
1 Introduction
The paper addresses whether deep structures can be learned from small datasets and introduces variational deep Gaussian processes to provide principled inference and model selection. Unlike existing approaches, the method supports arbitrarily deep GP hierarchies while producing a lower bound on marginal likelihood.
- Motivation: Deep learning models often require large datasets, motivating the question of whether abstract structures can be learned from only a few examples.For small datasets, objective evidence is needed to assess whether deep structures are justified.
- Limitations of existing approaches: Existing deep-learning approaches commonly use RBM stacks whose inference and likelihood estimation become difficult for deeper hierarchies and small datasets.RBM inference uses approximate methods, while deeper models may provide only a lower bound on data likelihood.
- Limitations of existing approaches: Prior GP-based approaches lacked a principled way to construct truly deep architectures, leaving deep learning mainly associated with RBM-based models.Existing GP extensions used richer covariance functions or probabilistic structures but did not provide principled deep hierarchies.
- Modeling framework: The paper contrasts Gaussian-process mappings with RBM mappings, noting that GP likelihoods can represent nonlinear functions of continuous inputs.The GP formulation places a prior over functions and can analytically integrate them in the Gaussian-likelihood setting.
- Contributions: The proposed variational approach marginalizes latent variables in arbitrarily many GP layers and yields a rigorous lower bound on marginal likelihood.A single layer corresponds to a GP-LVM, while recursive GP priors produce deeper hierarchies.
- Contributions: The variational lower bound enables objective selection of hierarchy structure, including the number of layers and nodes per layer.In a digits example, the deepest tested model, with 5 layers, achieved the best lower bound.
2 The Model
Deep GPs recursively place Gaussian-process priors over inputs, creating latent hierarchies whose inference is made tractable through variational augmentation and inducing points.
- Model construction: Deep GPs recursively place Gaussian-process priors over a GP’s inputs, extending standard GP and GP-LVM modeling into latent hierarchies.The model can be applied recursively to obtain deeper structures.
- Model construction: The architecture contains observed leaves, intermediate latent spaces, and a parent latent node connected by GP mappings.Intermediate nodes serve simultaneously as outputs for one layer and inputs for the layer below.
- Model construction: ARD covariance functions assign separate weights to latent dimensions, allowing irrelevant dimensions to be switched off during Bayesian training.Driving a dimension’s weight toward zero supports automatic structure discovery.
- Bayesian training: Variational inference introduces inducing points and auxiliary distributions so latent variables can be marginalized through a tractable bound.Augmenting the GP prior propagates priors through nonlinear mappings, while selected difficult terms cancel analytically.
- Bayesian training: The resulting bound combines tractable Gaussian terms with entropy and KL-divergence components across observed and latent layers.The leaf term matches the bound used for the Bayesian GP-LVM, while intermediate terms require expectations over their inputs.
3 Extending the hierarchy
The deep GP hierarchy extends vertically by adding latent layers and horizontally by partitioning layers into conditionally independent groups. These extensions preserve a variational-bound formulation while supporting multiple-output structures and controlled complexity.
- Vertical extension: Adding hidden layers extends the variational bound by summing one intermediate-layer term r_Xh for each layer.The hierarchy uses X_H = Z as its parent latent representation.
- Horizontal extension: Partitioning a latent layer into M_h conditionally independent subsets decomposes its bound term into a corresponding sum.The same independence principle can also be applied to the observed leaves.
- Horizontal extension: Horizontal partitioning supports unsupervised multiple-output GP learning by assigning separate leaf terms and ARD weights to output dimensions.Outputs encoding similar information are expected to receive similar relevance-weight vectors.
- Interpretation: The fully factorized structure links ARD parameters to neural-network weights and latent variables to neurons learning feature hierarchies.This connection makes the relationship between the model’s structure and neural-network architectures explicit.
- Parameters and complexity: O(NM^2) complexity replaces the typical O(N^3) cost for each generative GP mapping, while added layers introduce relatively few additional model parameters.Inducing points and variational means and covariances are variational rather than model parameters.
4 Demonstration
The demonstrations evaluate deep GPs on toy regression, motion capture, and handwritten-digit data, showing recovery of latent structure, improved prediction, automatic dimensionality selection, and increasingly abstract representations.
- Toy Data: The deep GP recovered the ground-truth hierarchy from toy data and automatically identified the correct dimensionalities.It was compared with stacked Isomap and stacked PCA reconstructions.
- Toy Data: Across 10 repetitions with 25 training points, the deep GP predicted unseen warped-process data better than a standard GP.The comparison used a standard GP with one fewer layer and a deep GP with two hidden layers.
- Modeling human motion: In motion capture, shared non-zero ARD weights revealed a common intermediate subspace for the two subjects and selected layer and subspace dimensionalities automatically.The model used separate observation modalities with a shared parent latent layer.
- Modeling human motion: Sampling showed that the top latent layer generated whole-sequence variations, whereas the first layer changed only a small subset of output dimensions.The learned latent spaces therefore represented information at different scales.
- Deep learning of digit images: For the digit experiment, the authors evaluated a five-hidden-layer deep GP using ARD weights, nearest-neighbour class separation, and samples from successive hidden layers.The experiment was designed to assess abstraction and model quality for different hierarchy depths on a small digit dataset.
- Deep learning of digit images: Samples from lower digit-model layers encoded local features, while higher layers encoded more abstract information.The analytic model-evidence bound was used to evaluate different overall hierarchy depths.
5 Discussion and future work
The framework learns hierarchical features with Bayesian training and variational model selection, supporting deep representations on scarce data. The authors also identify extensions to deeper architectures, GP-based regression, multitask and nonstationary modelling, and very large datasets.
- Results: The method successfully learned hierarchies describing natural human motion and handwritten-digit pixels, selecting a deep representation from only 150 digit examples.The variational lower bound supported model selection despite relatively scarce data.
- Results: Samples from lower hidden layers encode local features, whereas higher layers encode more abstract information.Figure 8 illustrates this progression through sampled outputs.
- Method: The variational lower bound enables principled selection among models with different initializations and numbers of layers.This supports automatic structure discovery while approximately marginalizing the latent space.
- Future work: Although experiments used at most 5 layers, the methodology is directly applicable to deeper architectures.The authors identify deeper models as future work and note that latent-space marginalization provides simultaneous regularization.
- Future work: The hierarchy can support GP-based regression that is not itself a Gaussian process, with proposed applications to multitask and nonstationary data.These applications target settings where a single-layer GP struggles.
- Future work: Extending the methodology to very large datasets remains a challenge, with stochastic variational inference proposed as a promising direction.The next step is incorporating large-scale variational learning algorithms into deep GPs.