Source-linked AI summary
Doubly Stochastic Variational Inference for Deep Gaussian Processes
Hugh Salimbeni, Marc Deisenroth
TL;DR
Single-layer GP kernels limit expressiveness, and existing DGP inference uses restrictive posterior assumptions. The paper introduces doubly stochastic variational inference that preserves correlations between layers, achieving effective, scalable performance across regression and classification tasks from hundreds to a billion points.
Problem
Single-layer kernels limit expressiveness, while existing DGP inference forces independence and Gaussianity between layers despite likely posterior correlations.
Method
The method uses a sparse inducing-point framework with an exact model-conditioned variational posterior that preserves correlations and samples to handle lost analytic tractability.
Results
DGPs with the proposed inference are effective and scalable across regression and classification tasks, with additional layers generally improving performance without overfitting.
Takeaways & Limitations
DGPs can be applied without hand-tuning across tasks and dataset sizes, including a billion-point dataset, while matching or exceeding single-layer GP performance.
Takeaways & Limitations
High-dimensional inner layers remain the main limitation because the fixed linear mean function was not optimized.
Abstract
from arXiv · showhide
Gaussian processes (GPs) are a good choice for function approximation as they are flexible, robust to over-fitting, and provide well-calibrated predictive uncertainty. Deep Gaussian processes (DGPs) are multi-layer generalisations of GPs, but inference in these models has proved challenging. Existing approaches to inference in DGP models assume approximate posteriors that force independence between the layers, and do not work well in practice. We present a doubly stochastic variational inference algorithm, which does not force independence between layers. With our method of inference we demonstrate that a DGP model can be used effectively on data ranging in size from hundreds to a billion points. We provide strong empirical evidence that our inference scheme for DGPs works well in practice in both classification and regression.
1 Introduction
Single-layer GPs face expressiveness and scalability limitations, while DGPs offer hierarchical non-parametric representations but make inference difficult. The paper introduces a doubly stochastic variational method that preserves inter-layer dependence and performs well across tasks and dataset sizes.
- Single-layer GP expressiveness is limited by its kernel, while richly parameterized kernel learning is expensive and may overfit.Highly parameterized neural-network kernels also require application-specific architectures and regularization.
- DGPs compose GPs hierarchically to obtain richer representations while retaining GP advantages and requiring few hyperparameters.
- Mean-field DGP inference imposes independence and Gaussianity assumptions that can underestimate variance when layers are correlated.
- The proposed variational posterior preserves the full model structure and correlations between layers, using sparse inducing points for tractability within each layer.Because the posterior retains the model’s nonlinearity, the method uses sampling to address the loss of analytic tractability.
- DGP inference works on benchmark regression and classification tasks, including a first demonstrated application to a billion-point dataset.Experiments report no overfitting from additional layers, even with small data.
2 Background
The background develops sparse variational inference for single-layer GPs and extends the setup to hierarchical DGPs. DGP inference is intractable, and the paper contrasts a factorized earlier posterior with its correlation-preserving alternative.
- Single-layer Gaussian Processes: A sparse GP prior factors into a Gaussian inducing-point prior and a Gaussian conditional over function values, with exact Gaussian-likelihood inference otherwise scaling cubically in N.
- Single-layer Gaussian Processes: Sparse GP inference uses inducing locations and values to construct a variational posterior whose function marginals depend only on corresponding inputs.This structure supports minibatch estimation of the data-summed bound for large datasets.
- Single-layer Gaussian Processes: The variational lower bound simplifies because conditional terms cancel and the likelihood expectation requires only variational marginals.Variational parameters and model hyperparameters are optimized through this lower bound, although hyperparameter learning can introduce bias.
- Deep Gaussian Processes: A DGP recursively composes vector-valued GP layers, using outputs from one layer as inputs to the next and assuming independent Gaussian noise between layers.Inducing locations and function values are introduced at each layer.
- Deep Gaussian Processes: DGP inference is intractable, while the earlier variational posterior factorizes layer inputs and outputs, losing correlations and potentially underestimating posterior variance.The earlier approach also requires 2N(D1 + ··· + DL−1) variational parameters and a particular kernel form for a tractable bound.
3 Doubly Stochastic Variational Inference
The method simplifies within-layer correlations while preserving correlations between layers, then optimizes an unbiased stochastic variational bound. Sampling and minibatch subsampling make inference scalable, while predictions are obtained by sampling from the variational posterior.
- The variational posterior simplifies correlations within layers but maintains correlations between layers, avoiding forced inter-layer independence.It uses the exact model conditioned on inducing points, preserving the full model's layered structure.
- Each layer's conditional posterior is Gaussian, with within-layer marginals depending only on corresponding inputs.After analytically marginalizing inducing variables, the resulting distribution remains coupled across layers.
- Unbiased samples are generated recursively with univariate unit Gaussians through the re-parameterization trick.The sampling procedure proceeds layer by layer using isotropic Gaussian noise.
- The evidence lower bound is optimized stochastically using Monte Carlo sampling and minibatch subsampling.The resulting bound has evaluation complexity O(NM^2(D1 + · · · + DL)).
- Predictions are formed by sampling from the variational posterior at the test input location.The same recursive layerwise procedure is applied after replacing training inputs with the test location.
4 Results
Across classification and regression benchmarks, deeper DGPs generally improved performance over single-layer GPs without observed overfitting, including at large scale. The experiments also show strong results on specialized image and billion-example taxi tasks.
- Regression Benchmarks: DGPs did not overfit on small and medium datasets and often outperformed single-layer GPs, including with fewer inducing points.On power, protein, and kin8nm, every DGP outperformed the SGP with five times as many inducing points.
- Rectangles Benchmark: 77.9% accuracy from the four-layer DGP exceeded reported three-layer deep-belief-network and sparse-GP results on Rectangle-Images.The 500-inducing-point single-layer GP was significantly less effective than every deep model.
- Large-Scale Regression: Deeper DGPs significantly improved log likelihood and RMSE over single-layer models on the year and airline datasets, even with 500 inducing points.Performance increased with depth in both large-scale regression experiments.
- MNIST Multiclass Classification: MNIST accuracy increased from 97.48% for a 100-inducing-point single-layer GP to 98.11% for a three-layer DGP.The same initialization and learning schedule was used for all models.
- Large-Scale Classification: 0.877 AUC made all DGP models the highest-performing methods on SUSY, while HIGGS improved from 0.830 to 0.846 across DGP depths 2–4.On HIGGS, DGPs exceeded BDT, NN, and both single-layer GP baselines, but the best five-layer DNN reached 0.885.
- Massive-Scale Regression: A five-layer DGP reduced taxi-trip RMSE by 21% versus SGP on a dataset with 1B training examples.The experiments used 1M examples for testing and observed consistent improvement with additional layers.
5 Related Work
Earlier DGP inference methods factorized variational posteriors between layers or modified the prior with additional approximations, often imposing kernel restrictions. These approaches trade model structure or generality for tractable inference.
- Earlier DGP approaches factorized the variational posterior between layers, requiring variational parameters that grow linearly with the number of data points.
- Bui et al. and Cutajar et al. modified the DGP prior and added approximations for tractable inference in parametric models.
- Bui et al. and Cutajar et al. require specific kernels with analytic expectations or analytic spectral densities.
- Earlier work addressed a zero-mean inner-layer pathology by concatenating original inputs or using a linear mean function.
6 Discussion
The experiments indicate that doubly stochastic DGP inference is effective and scalable across tasks. Additional layers generally improve large-data performance without overfitting small datasets, while remaining broadly applicable and implementationally simple.
- Doubly stochastic DGP inference is effective and scalable across a wide range of tasks.
- On small datasets, DGPs do not overfit, while on large datasets additional layers generally improve performance and never deteriorate it.
- The largest gain from increasing layers occurs on the taxi dataset containing 1B points.
- For the same computational budget, increasing DGP depth can be more effective than improving approximate inference in a single-layer model.
- The method supports domain-specific kernels in any layer and has a simple implementation of fewer than 200 lines integrated with GPflow.
7 Conclusion
The paper introduces a new inference method for DGPs and demonstrates its use across regression and classification tasks without hand-tuning. The approach matches or exceeds single-layer GP performance and is scalable with GPU acceleration, though high-dimensional inner layers remain a limitation.
- The paper presents a new inference method for Deep Gaussian Process models.
- DGPs using this inference perform at least as well as single-layer GPs across regression and classification tasks without hand-tuning.
- The approach often substantially exceeds single-layer GP performance, even when the single-layer approximation is improved.
- The approach is highly scalable and benefits from GPU acceleration.
- High-dimensional inner layers are the approach’s most significant limitation.
Experiment Details
The experiments use inducing-point and hyperparameter initializations, minibatch optimization, and GPU-accelerated computation. Training settings vary by dataset size, and computation is evaluated through per-step timing with depth scaling considered.
- All DGP models use 100 inducing points initialized with K-means, with zero means and identity variances scaled by 10^-5 in inner layers.
- Kernel variances and lengthscales start at 2, likelihood variance at 0.01, and inter-layer noise at 10^-5.
- All hyperparameters and variational parameters are jointly optimized with Adam using a 10K minibatch and learning rate 0.01.
- Training lasts 20,000 iterations for small-to-medium regression datasets, 100,000 for other datasets, and 500,000 for taxi.
- GPU acceleration speeds computation by an order of magnitude, with timing measured per gradient step and slightly sub-linear scaling in depth.
Further results
The further-results section reports regression log-likelihood and RMSE results, plus binary-classification AUC results on high-energy-physics data. Figure 2 specifies that lower RMSE is better and uses a matched sparse GP as a baseline.
- Regression test RMSE results are reported on benchmark datasets, with lower values preferred.
- Regression results include test log likelihoods for large datasets.
- Binary-classification results report AUC values on high-energy-physics data for comparison with Baldi et al. (2014).
- Regression test log likelihood results are reported as means over 20 splits with standard errors.
Derivation of the Lower Bound
The lower-bound derivation introduces the evidence lower bound for the DGP model and identifies likelihood and DGP-prior contributions. It then notes cancellation of terms inside the logarithm to obtain the next form.
- The DGP model’s evidence lower bound is introduced as the quantity being derived.
- The displayed bound distinguishes likelihood and DGP-prior terms.
- Terms inside the logarithm cancel, yielding a simplified expression.