PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributed Gaussian Processes

Marc Peter Deisenroth, Jun Wei Ng

arXiv:1502.02843v3stat.ML

TL;DR

Large data sets make full Gaussian-process regression computationally and memory intensive, while sparse approaches rely on inducing or variational parameters. The paper introduces the rBCM, which recursively distributes and recombines independent expert computations in closed form. It reports scaling to 10^7 data points and practical million-point training on a laptop.

  • Problem

    Full Gaussian processes scale poorly with data size, while sparse methods address scale using inducing or variational parameters with practical limits.

  • Method

    The rBCM is a hierarchical product-of-GP-experts model that recursively distributes and recombines computations across independent units using closed-form inference.

  • Results

    The rBCM applies to data sets of size O(107), and a GP with one million data points can be trained in less than half an hour on a laptop.

  • Takeaways & Limitations

    The rBCM uses the full data set while distributing computational and memory loads, without inducing or variational parameters.

  • Takeaways & Limitations

    Generalized product-of-GP-experts models require an a-priori setting of βk to support deep computational graphs, with βk = 1/M used in this work.

Abstract

from arXiv · show

To scale Gaussian processes (GPs) to large data sets we introduce the robust Bayesian Committee Machine (rBCM), a practical and scalable product-of-experts model for large-scale distributed GP regression. Unlike state-of-the-art sparse GP approximations, the rBCM is conceptually simple and does not rely on inducing or variational parameters. The key idea is to recursively distribute computations to independent computational units and, subsequently, recombine them to form an overall result. Efficient closed-form inference allows for straightforward parallelisation and distributed computations with a small memory footprint. The rBCM is independent of the computational graph and can be used on heterogeneous computing infrastructures, ranging from laptops to clusters. With sufficient computing resources our distributed GP model can handle arbitrarily large data sets.

1. Introduction

Gaussian processes provide flexible, principled probabilistic nonlinear regression but scale poorly, motivating distributed expert models and the rBCM. The rBCM recursively combines independent computations while operating on the full data set.

  • Training and prediction scale as O(N 3) and O(N 2), respectively, limiting standard GPs to data sets of size O(104).
  • Sparse GP approximations reduce computational burden by using subsets of data and scale to training sets of size N ∈ O(106).
  • Distributed expert models partition data across independent local GPs, but existing approaches can require stationary kernels or omit variance predictions.
  • Product-of-experts models avoid mixture-model weight assignment, but standard PoEs can be overconfident while generalized PoE-GPs can overestimate variances.
  • The rBCM recursively recombines independent expert computations, provides consistent full-GP approximations, and scales through parallelisation.
  • Analytic inference lets the rBCM handle data sets of size O(107), while a million-point GP can train in less than half an hour on a laptop.

2. Problem Set-up and Objective

The paper formulates GP regression and identifies the computational and memory costs that make full GPs impractical for large data sets. It addresses these costs by distributing computation and memory across independent units operating on data subsets.

  • The regression objective is to infer a latent function f from training inputs X and observations y under a Gaussian-process model.
  • A GP is specified by a mean function and covariance kernel, with hyper-parameters controlling the model.
  • GP training typically maximizes the log-marginal likelihood over kernel and noise hyper-parameters.
  • For fixed hyper-parameters, GP prediction produces a Gaussian posterior predictive distribution with a mean and variance.
  • For N > 10,000, full-GP training and prediction become time-consuming and require O(N 2 + ND) memory.
  • The proposed solution distributes computational and memory loads across independent units that operate on subsets of data using a hierarchical product-of-GP-experts model.

3. Distributed Product-of-GP-Experts Models

Distributed product-of-GP-experts models partition training data among independent GP experts, then recursively recombine their predictions in a hierarchical computational graph. The rBCM combines Bayesian correction with adaptive expert weighting to provide consistent, distributed predictions that are more robust to weak experts.

  • Distributed Product-of-GP-Experts Models: Training data are partitioned among independent GP experts, whose local mean and variance predictions are recursively recombined into a multilayer tree.Each parent node combines computations from its direct children, and the top node produces the overall prediction.
  • Distributed Product-of-GP-Experts Models: The independence assumption yields a block-diagonal kernel approximation, enabling efficient training and prediction through parallelised time and memory computation.Each expert handles a subset of size n_k, with n_k much smaller than the full data size N.
  • Product of GP Experts: PoE models avoid explicit expert weighting, but their predictive variances vanish as experts increase, preventing fallback to the prior outside the data range.The standard PoE therefore becomes overconfident, especially in regions without observations.
  • Generalised Product of GP Experts: The gPoE can fall back to the prior, but its variance weighting is often too conservative and its entropy-based setting does not support deep computational graphs.For general computational graphs, the paper instead sets β_k = 1/M, making gPoE predictive means match those of the PoE while precisions differ.
  • Bayesian Committee Machine: The BCM corrects the product by dividing by the prior M − 1 times, producing a consistent model that falls back to the prior, although weak experts can distort its predictive mean.The BCM is usually accurate in the data range but can break when experts receive few data points.
  • Robust Bayesian Committee Machine: The rBCM adaptively weights experts according to predictive power, combines the gPoE’s flexibility with the BCM’s prior correction, and produces more reasonable predictions than the alternatives shown.Its adaptive prior influence slightly increases uncertainty while removing the BCM’s dominant predictive-mean kink around x = 0.

4. Distributed Computations

The rBCM supports equivalent hierarchical computational graphs, allowing distributed GP predictions to be organized across layers and adapted to available infrastructure. These graphs preserve predictions while trading off communication traffic and central-node bottlenecks.

  • Equivalent computational graphs: For a fixed number of GP experts, single-layer and multilayer rBCM graphs produce identical predictive distributions.With 32 experts, a one-layer graph is equivalent to a two-layer graph with 8 parent nodes, each handling 4 experts.
  • Hierarchical computation: The rBCM combines expert predictions through gPoE aggregation, PoE combination, and a prior correction.In the two-layer construction, parent nodes aggregate subsets of experts before the central node applies the overall correction.
  • Hierarchical computation: The general rBCM architecture can be extended to an arbitrarily deep computational graph.Leaf experts compute means, variances, and confidence values; subsequent layers recursively aggregate these quantities and weights.
  • Infrastructure trade-offs: Equivalent graphs allow the architecture to be matched to the computing infrastructure, balancing communication traffic against central-node bottlenecks.Shallow graphs minimize overall traffic but are more vulnerable to communication bottlenecks at the central node.
  • Empirical scaling: Distributed GP computation time for the log-marginal likelihood and its gradient scales favourably as training-set size increases.Figure 4 compares distributed GPs with full and sparse GP computation times as a function of training-data size.

5. Experiments

The experiments evaluate distributed GP training time and predictive quality against full and sparse baselines. Results show substantial scalability, practical laptop training, and strong predictive performance for the rBCM across standard and large-scale flight datasets.

  • 5.1. Training Time for Large Data Sets: Distributed GP training time is evaluated through marginal-likelihood gradients on a 64-node, four-core-per-node architecture using a three-layer computational graph.The evaluation compares distributed, full, and sparse GP computation times as training-set size increases.
  • 5.1. Training Time for Large Data Sets: 1.7 × 10^7 data points required approximately 230 seconds for the distributed GP, comparable to full-GP training at 1.6 × 10^4 points and sparse-GP training at 3.2 × 10^4 points.The distributed model was scaled using a computational graph with increasing numbers of experts.
  • 5.1. Training Time for Large Data Sets: 10^6 data points were trained in about 20 minutes on a standard laptop, indicating relatively small memory consumption without large computing infrastructure.The authors describe this as a full training cycle of the distributed GP.
  • 5.2. Empirical Approximation Errors: On Kin40K, the rBCM consistently outperformed the other evaluated methods in RMSE and NLPD, while SOD was substantially worse than the distributed GP models.The comparison used a full GP as ground truth and SOD as a sparse approximation, with shared full-GP hyper-parameters for approximate models.
  • 5.2. Empirical Approximation Errors: The rBCM maintained better NLPD than BCM and PoE when experts received few data points, avoiding the PoE's variance under-estimation and BCM's weak-expert problem.Figure 5 reports RMSE and NLPD against gradient-computation time and data points per expert; standard errors were below 10^-2.
  • 5.3. Airline Delays (US Flight Data): On the US flight dataset, experiments used 7 × 10^5, 2 × 10^6, and 5 × 10^6 training points with marginal-likelihood computation times of 13, 39, and 90 seconds.The experiments used 4,096, 8,192, and 32,768 experts, respectively.
  • 5.3. Airline Delays (US Flight Data): On the 700K flight dataset, rBCM, gPoE, and PoE performed significantly better than SVIGP and Dist-VGP in RMSE, while BCM and PoE showed variance-related weaknesses.The rBCM and gPoE standard errors were consistently below 0.3, whereas BCM and PoE had outliers and relatively large NLPD values.
  • 5.3. Airline Delays (US Flight Data): Performance decreased as more training data were included because the 700K/100K dataset was more stationary than the 2M/100K and 5M/100K datasets.This effect was reported across the three flight-data experiments.

6. Discussion and Conclusion

The discussion emphasizes that distributed GPs combine simple expert models while avoiding inducing or variational parameters. The rBCM uses shared hyper-parameters, closed-form recursive recombination, and distributed computation to scale efficiently, but sparse methods remain practically bounded by inducing-variable optimization.

  • 6. Discussion and Conclusion: Distributed GPs split data into small pieces, train GP experts jointly, and combine their predictions, using only standard GP hyper-parameters rather than inducing or variational parameters.This design is described as making local-optimum problems less likely than in sparse GP models.
  • 6. Discussion and Conclusion: Shared hyper-parameters automatically regularise the model because the overall gradient averages experts' marginal-likelihood gradients, so individual-expert overfitting is not favoured.The regularisation follows from jointly training experts with common hyper-parameters.
  • 6. Discussion and Conclusion: Sparse methods are practically limited to about O(10^6) data points under assumptions about trainable inducing variables and their data-compression rate.The discussion notes that higher compression or parallelisation can extend this boundary.
  • 6. Discussion and Conclusion: The rBCM recursively recombines independent computations in closed form, incorporates the GP prior, and supports heterogeneous infrastructures from laptops to large clusters.The authors characterize the model as computationally and memory efficient and independent of the computational graph.
  • 6. Discussion and Conclusion: A million-point rBCM can be trained in under 30 minutes on a laptop, while training with 10^7 points can be completed in a few hours with more computing power.The paper presents these results as evidence of practical large-scale scalability.
Loading 1502.02843v3…