PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Gaussian Processes for Regression using Approximate Expectation Propagation

Thang D. Bui, Daniel Hernández-Lobato, Yingzhen Li, José Miguel Hernández-Lobato, Richard E. Turner

arXiv:1602.04133v1stat.MLcs.LG

TL;DR

Deep Gaussian processes offer hierarchical GP models with flexible representations and uncertainty propagation, but exact Bayesian learning is intractable and existing approximations do not scale well. This paper introduces sparse approximate Expectation Propagation with probabilistic backpropagation for DGP regression, and reports competitive performance across real-world datasets.

  • Problem

    Exact DGP Bayesian learning is analytically intractable, while existing approximate methods do not scale well for regression.

  • Method

    The paper combines a sparse GP approximation, tied-factor approximate Expectation Propagation, and probabilistic backpropagation for DGP learning.

  • Results

    The method is superior on 7 of 11 datasets and comparable on the remainder, while outperforming GP regression and nearly always outperforming approximate Bayesian neural-network methods.

  • Takeaways & Limitations

    DGPs are presented as a competitive alternative to multilayer Bayesian neural networks for supervised learning.

  • Takeaways & Limitations

    Classification shows no substantial gain over a GP, potentially because hidden layers use a diagonal Gaussian approximation whose non-diagonal replacement is more expensive.

Abstract

from arXiv · show

Deep Gaussian processes (DGPs) are multi-layer hierarchical generalisations of Gaussian processes (GPs) and are formally equivalent to neural networks with multiple, infinitely wide hidden layers. DGPs are nonparametric probabilistic models and as such are arguably more flexible, have a greater capacity to generalise, and provide better calibrated uncertainty estimates than alternative deep models. This paper develops a new approximate Bayesian learning scheme that enables DGPs to be applied to a range of medium to large scale regression problems for the first time. The new method uses an approximate Expectation Propagation procedure and a novel and efficient extension of the probabilistic backpropagation algorithm for learning. We evaluate the new method for non-linear regression on eleven real-world datasets, showing that it always outperforms GP regression and is almost always better than state-of-the-art deterministic and sampling-based approximate inference methods for Bayesian neural networks. As a by-product, this work provides a comprehensive analysis of six approximate Bayesian methods for training neural networks.

1 Introduction

DGPs extend GPs into hierarchical, infinitely wide deep models that can learn flexible representations while propagating uncertainty. The paper addresses the intractability and poor scalability of existing Bayesian learning methods with a new approximate procedure for regression.

  • Model motivation: DGPs are hierarchical generalisations of GPs, equivalent to neural networks with multiple infinitely wide hidden layers.Their layered GP mappings retain GP properties while enabling hierarchical representations and uncertainty propagation.
  • Model motivation: Nonlinear hidden layers can learn input warping or dimensionality changes, providing flexible Bayesian kernel design for complex datasets.The resulting input-output mapping is non-Gaussian and can increase representational power more cheaply through added layers than inducing points.
  • Research gap: Exact Bayesian learning is analytically intractable, while existing approximate DGP methods do not scale well and lack comparison with alternative deep Bayesian models.The learning problem includes posterior inference over layer mappings and hyperparameter optimisation through the marginal likelihood.
  • Contribution: The paper proposes an efficient DGP regression algorithm combining sparse GP approximation, approximate Expectation Propagation, and probabilistic backpropagation.The authors describe the resulting method as computationally efficient, scalable, and easy to implement.
  • Contribution: Across medium- to large-scale supervised datasets, the method is reported to outperform GP regression and almost always outperform state-of-the-art approximate inference methods for multilayer neural networks.The evaluation covers various supervised learning tasks.

2 Deep Gaussian processes

DGPs place GP priors over mappings across multiple layers, producing non-Gaussian hidden representations and reducing to ordinary GP regression when only one layer is used. Their posterior inference and marginal-likelihood calculations are analytically intractable, motivating approximate methods.

  • Model: A DGP places zero-mean GP priors over layer mappings, with hidden variables mediating the hierarchy.For one layer, the model collapses to GP regression or classification.
  • Model: Because layer inputs are random, function values and hidden variables are not marginally normally distributed.This non-Gaussianity arises from composing GP mappings through the hierarchy.
  • Illustration: A two-layer DGP with a two-dimensional hidden layer models the mountain-car value function from a small number of noisy evaluations.The learned first-layer functions cover different parts of the input space, while the DGP handles a steep value-function cliff that challenges a standard GP kernel.
  • Inference problem: The target posterior includes latent mappings, intermediate hidden variables, and a marginal likelihood for hyperparameter tuning and model comparison.Nonlinearity makes these quantities analytically intractable, so approximate inference is required.
  • Existing approaches: Variational inference introduces parameters that grow linearly with the number of training datapoints, hindering scalability.MAP estimation avoids this growth but is prone to over-fitting and does not provide uncertainty estimates.
  • Proposed approach: The proposed alternative combines sparse GP approximation, direct approximate EP-energy optimisation, and an extension of probabilistic backpropagation.These approximations are introduced to address computational and analytical intractability.
  • Efficiency: The method has complexity O(NLM^2) and memory O(LM^2), compared with O(NL + LM^2) memory for the cited variational approach.Its memory requirement is described as much lower while computational costs remain competitive with nested variational methods.

3 The Fully Independent Training Conditional approximation

The FITC sparse approximation replaces the cubic cost of full GPs with a semi-parametric model based on inducing outputs. Its accuracy depends on the number of inducing outputs and the underlying function complexity.

  • Motivation: Full GP computation scales cubically with training-set size, motivating sparse approximations for practical inference.Sparse methods either create semi-parametric representations or approximate the exact posterior while retaining nonparametric properties.
  • FITC approximation: FITC belongs to the sparse-approximation class that explicitly creates a semi-parametric representation of the original GP.The approximation is used here to avoid full-GP computational costs.
  • FITC approximation: The inducing outputs have Gaussian prior p(u_l|θ_l) = N(u_l; 0, K_u_l−1,u_l−1) for each layer.This prior forms part of the sparse layer representation.
  • FITC approximation: FITC propagates uncertainty through the network and introduces structured non-stationary noise that captures uncertainty from sparsification.Covariance terms use the preceding hidden representation and inducing inputs as inputs.
  • Complexity and accuracy: Inference and hyperparameter tuning cost O(NM^2), so M must remain smaller than N for computational gain.Approximation quality depends on M and function complexity; as M approaches N with shared inducing and training inputs, FITC reverts to the original GP.

4 Approximate Bayesian inference via EP

The method approximates inference in deep sparse Gaussian processes by tying Gaussian data factors and directly optimizing an approximate EP energy, producing a scalable posterior approximation.

  • Inference objectives: The approximate posterior over inducing outputs supports prediction and hyperparameter tuning through an approximate marginal likelihood.The posterior is used for predictive inference, while the marginal likelihood is optimized for hyperparameter tuning.
  • Approximation trade-off: Full EP is not guaranteed to converge and requires O(NLM^2) memory to store factor means and covariance matrices.These limitations motivate the tied-factor approximation.
  • Tied-factor EP: The inducing-output posterior is approximated with tied Gaussian data factors, reducing EP’s memory footprint while retaining an average likelihood effect.The approximation has the form q(u) ∝ p(u)g(u)^N, where g(u) is an average data factor.
  • Direct energy minimisation: Direct optimization of the tied-factor EP energy avoids the computationally inefficient double-loop algorithm.The shared cavity distribution makes direct optimization of the approximate marginal likelihood possible.
  • Approximation trade-off: The tied-factor objective differs from full EP because it averages moments rather than natural parameters, and therefore generally yields a different solution.The paper describes natural-parameter averaging as arguably more sensible but less tractable.

5 Probabilistic backpropagation for deep Gaussian processes

Probabilistic backpropagation approximates the intractable log Z calculation by propagating Gaussian moments through DGP layers, moment-matching intermediate non-Gaussian distributions, and differentiating the result.

  • Approximate propagation: For nonlinear DGP hierarchies, the likelihood integral log Z is analytically intractable, motivating Gaussian approximations during layerwise propagation.The difficulty arises because Gaussian cavity distributions become complex distributions after passing through nonlinear layers.
  • Approximate propagation: Each layer propagates a Gaussian, moment-matches the resulting non-Gaussian distribution, and passes the Gaussian approximation to the next layer.This procedure is algorithmically identical to Assumed Density Filtering and underlies probabilistic backpropagation.
  • Approximate propagation: The method sequentially reintroduces hidden variables to compute a Gaussian approximation to Z and its parameter gradients.A two-layer network is used as the running example for this sequential approximation.
  • Method distinction: Unlike the variational free energy approach, the algorithm approximately integrates out hidden variables instead of retaining an explicit approximate hidden-variable distribution.This distinction applies when computing log Z.
  • Computational tractability: Analytic kernel expectations for common covariance functions make the moment-matching approximation tractable, with non-diagonal covariance available for multidimensional hidden variables.The cited kernels include exponentiated quadratic, linear, and spectral mixture kernels.
  • Gradient computation: Forward-propagated means and variances are stored, then differentiated backward with the chain rule to obtain gradients through the approximate density-filtering procedure.This backward step motivates the name probabilistic backpropagation.

6 Stochastic optimisation for scalable training

The training objective decomposes into independent data terms, enabling distributed computation and unbiased minibatch estimates for stochastic optimization.

  • Complexity: O(NLM^2) is the overall batch complexity because O(LM^2) propagation is repeated across all training datapoints.The propagation and moment-matching cost O(LM^2) per datapoint.
  • Scalable optimization: Independent terms in the objective can be distributed, substantially reducing computational cost during training.The decomposed objective is also suitable for stochastic optimization.
  • Scalable optimization: Unbiased noisy objective estimates can be formed from minibatches, whose size is denoted |B|.This supports stochastic optimization rather than requiring full-batch evaluation.

7 Approximate predictive distribution

For test prediction, the method approximates the otherwise intractable predictive distribution with a Gaussian obtained by a forward pass through the layers.

  • Predictive approximation: The predictive distribution p(y∗|x∗, X, Y) is not analytically tractable under the approximate posterior.Prediction requires integrating over the inducing outputs.
  • Predictive approximation: A single forward pass propagates a Gaussian test-input distribution through each layer while incorporating the approximate inducing-output posterior.Each layer approximates its output distribution by a Gaussian.

8 Experiments

Experiments compare FITC-DGP variants with sparse GPs and approximate Bayesian neural networks on UCI regression datasets and photovoltaic-molecule prediction. DGPs achieve the strongest overall ranking, while deeper or wider latent representations generally improve predictive performance.

  • 8.2 Regression on UCI datasets: DGP-3-100 achieves the best performance on 6 out of 10 UCI datasets and remains competitive on the other four.The rank analysis identifies DGPs with the proposed inference scheme as the best-performing methods overall.
  • 8.2 Regression on UCI datasets: DGP-1 performs poorly compared with GP regression, although it remains competitive with several Bayesian neural-network methods.DGP-1 is the one-dimensional hidden-layer architecture and corresponds to a standard warped GP.
  • 8.2 Regression on UCI datasets: Sampling-based methods achieve good predictive performance overall, but often require more tuning; HMC does not scale to large datasets.HMC is superior on one dataset and close to DGP performance on three others.
  • 8.2 Regression on UCI datasets: DGP-2 and DGP-3 perform as well as or better than other methods on RMSE, while BNN-SGLD often gives good RMSE despite relatively low MLL rank.
  • 8.3 Predicting the efficiency of organic photovoltaic molecules: On photovoltaic molecules, the five-dimensional DGP significantly outperforms competing architectures in test MLL, including a shallower GP with more pseudo-datapoints.The experiment uses 50,000 training molecules, 10,000 test molecules, binary 512-dimensional features, and 200 inducing outputs per DGP.

9 Summary

The paper introduces a deterministic approximation scheme for training DGPs and finds it competitive with multi-layer Bayesian neural networks, while noting limitations beyond regression.

  • The method combines approximate Expectation Propagation with the FITC approximation to address computational and analytical intractability.A novel probabilistic backpropagation extension handles a difficult marginalisation problem in approximate EP.
  • Across eleven datasets, the proposed DGP training method is superior on 7 out of 11 datasets and comparable on the remainder.The comparison covers state-of-the-art algorithms for Bayesian neural networks.
  • The method demonstrates that DGPs are a competitive alternative to multi-layer Bayesian neural networks for supervised learning tasks.
  • Although applicable in principle to classification and unsupervised learning, the initial classification results show no substantial gain over a GP.The authors relate this issue potentially to the diagonal Gaussian approximation used for hidden layers from the second layer onward.

A.1 Regression

The regression analysis compares DGP approximations with GP and Bayesian neural-network methods across UCI datasets, finding strong average performance but distinct uncertainty and scalability trade-offs.

  • Log likelihood: DGPs with the proposed approximation are superior according to mean test log likelihood and obtain the top average ranking.The full results cover all methods and datasets.
  • RMSE: DGPs average at least as well as or better than BNN-SGLD and other methods on test RMSE, while BNN-SGLD often predicts the test mean well despite inaccurate uncertainty.This separates predictive-mean accuracy from test log-likelihood performance.
  • Architecture effects: Increasing the number of layers and hidden dimensions improves predictive accuracy, whereas the one-dimensional warped-GP architecture performs poorly relative to other architectures.
  • BNN comparisons: Among Bayesian neural-network methods, VI(KW) and two-hidden-layer SGLD perform well, HMC performs well but runs much longer, and VI(G), PBP, and Dropout perform poorly overall.
  • Inference efficiency: EP stores approximate data factors at O(NM2) memory, while tied-factor SEP reduces the DGP requirement from O(NLM2) to O(LM2).SEP ties the data factors into an average factor and updates it iteratively with learning rate β.
  • Gaussian approximation: The Gaussian approximation must preserve correlations induced by integrating hidden variables, motivating a non-diagonal Gaussian with matched moments.
Loading 1602.04133v1…