PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models

Yarin Gal, Mark van der Wilk, Carl E. Rasmussen

arXiv:1402.1389v2stat.MLcs.LG

TL;DR

Large datasets make GP inference difficult to scale. The paper re-parameterizes variational inference for sparse GP regression and latent variable models into a distributed procedure, and reports scalable inference with balanced load and improved GP performance as data increases.

  • Problem

    Sparse GP regression and latent variable models have been limited by inference scalability on large datasets.

  • Method

    The paper re-parameterizes variational inference so computations decouple across nodes given inducing inputs, enabling distributed Map-Reduce-style updates.

  • Results

    The inference scales with data and computational resources while maintaining balanced node loads, and GP performance improves with increasing data across flight regression and MNIST latent-variable modelling.

  • Takeaways & Limitations

    The approach processes datasets with millions of points and supports GP regression and latent-variable modelling at big-data scale.

  • Takeaways & Limitations

    SCG sometimes converged to poor values or bad local optima, while larger-data training required initialization from a smaller-data model for expected performance.

Abstract

from arXiv · show

Gaussian processes (GPs) are a powerful tool for probabilistic inference over functions. They have been applied to both regression and non-linear dimensionality reduction, and offer desirable properties such as uncertainty estimates, robustness to over-fitting, and principled ways for tuning hyper-parameters. However the scalability of these models to big datasets remains an active topic of research. We introduce a novel re-parametrisation of variational inference for sparse GP regression and latent variable models that allows for an efficient distributed algorithm. This is done by exploiting the decoupling of the data given the inducing points to re-formulate the evidence lower bound in a Map-Reduce setting. We show that the inference scales well with data and computational resources, while preserving a balanced distribution of the load among the nodes. We further demonstrate the utility in scaling Gaussian processes to big data. We show that GP performance improves with increasing amounts of data in regression (on flight data with 2 million records) and latent variable modelling (on MNIST). The results show that GPs perform better than many common models often used for big data.

1 Introduction

The paper addresses limited scalability of sparse GP regression and latent variable models by developing distributed variational inference. Experiments examine scalability, load distribution, and GP performance on large datasets.

  • Sparse GP regression and GPLVMs support several tasks but have been difficult to use with big datasets because inference does not scale adequately.The paper highlights regression, density estimation, data imputation, and dimensionality reduction as application areas.
  • The distributed inference procedure is designed to balance computational load, scale with the number of nodes, and minimize global-step overhead.
  • The paper re-parameterizes variational inference so data-dependent variational parameters can be updated independently across nodes after conditioning on inducing inputs.Communication between nodes requires constant time, and GPLVM embeddings can also be optimized in parallel.
  • Experiments show that inference scales with data and computational resources while maintaining balanced load distribution and comparing favorably with sequential implementations.The evaluation examines runtime, computational scaling, and load distribution across nodes.
  • GP performance improves with more data in flight-data regression and MNIST latent-variable modelling, and exceeds that of many common big-data models.The regression experiment uses 2 million flight records.

2 Related Work

Prior work used stochastic variational inference to scale sparse GPs, but its looser bound and optimization requirements create practical difficulties. The proposed approach retains the tighter Titsias lower bound while also extending scaling to GPLVMs.

  • SVI scaled sparse GP regression with mini-batches and was reported to learn from 700,000 points, with possible applicability to GPLVMs.
  • Compared with the Titsias formulation, SVI uses a less tight variational marginal likelihood bound by explicitly representing q(u) instead of analytically marginalizing its optimal form.
  • SVI additionally introduces noisy gradients, fixed inducing inputs, and optimizer-specific parameters that complicate optimization.
  • The proposed approach preserves the Titsias lower bound, avoids these difficulties, and enables scaling GPLVMs as well.

3 The Gaussian Process Latent Variable Model and Sparse GP Regression

The paper reviews sparse GP regression and GPLVMs, emphasizing the cubic cost of exact inference and the use of inducing points and variational approximations for tractability. GPLVMs extend the regression setting by treating inputs as latent variables.

  • Sparse GP Regression: Exact GP regression requires inversion of an n×n matrix, giving O(n^3) time complexity that is prohibitive for large datasets.
  • Sparse GP Regression: Sparse approximations summarize the regression function with m inducing input-output pairs, reducing prediction cost to O(m^3).The inducing inputs and targets serve as sufficient statistics for predictions.
  • Sparse GP Regression: Learning the sparse model involves inferring the posterior over inducing targets, with an overall computational complexity of O(nm^2).
  • Sparse GP Regression: The Titsias variational approximation treats inducing inputs as variational parameters, tightening the evidence lower bound while reducing over-fitting and retaining improved computational complexity.
  • GPLVM: GPLVMs infer both latent inputs and the function mapping, forming an unsupervised nonlinear generalization of PCA with a prior over the latent inputs.
  • GPLVM: The resulting Gaussian approximation to q(X) is used in the distributed inference scheme for both GPLVMs and sparse GP regression.

4 Distributed Inference

The paper derives a unified variational bound for sparse GP regression and latent variable models that decomposes into independent data-point terms conditioned on inducing points. This factorization supports Map-Reduce inference, with nodes computing partial terms while a central node optimizes global parameters and, for GPLVMs, nodes optimize local variables.

  • 4.1 Decoupling the Data Conditioned on the Inducing Points: The derivation exploits conditional independence given inducing points to express the lower bound as a sum of independent data-point terms.The data-point decomposition follows because each Yi is independent of Fj for j ≠ i.
  • 4.1 Decoupling the Data Conditioned on the Inducing Points: The resulting unified bound is identical to the established sparse GP regression and GPLVM bounds while being re-parametrized as independently computable sums.For observed inputs, q(Xi) has zero variance and mean Xi, yielding ⟨Kmi⟩q(Xi) = Kmi.
  • 4.2 Distributed Inference Algorithm: Map-Reduce distributes input-output subsets and global parameters across nodes, which return only the partial terms needed to aggregate the global objective.The global parameters include kernel hyper-parameters, observation noise, and inducing inputs.
  • 4.2 Distributed Inference Algorithm: The endpoint calculations return m × m × q matrices, giving constant space complexity for fixed m at the central node.The returned quantities include partial sums and derivatives used for global optimization.
  • 4.2 Distributed Inference Algorithm: The central node optimizes global parameters while GPLVM endpoint nodes concurrently optimize local embedding parameters.For regression, the additional derivative-distribution and local-optimization steps are unnecessary.
  • 4.2 Distributed Inference Algorithm: The framework supports global optimization procedures using partial derivatives and parallel optimization of local variables through SCG or local gradient descent.The paper then evaluates the developed inference empirically across multiple tasks.

5 Experimental Evaluation

The experiments evaluate scalability with computational resources and data, compare runtime with standard inference, and examine load balance across nodes. The proposed inference scales nearly ideally with cores, remains nearly constant under proportional data-and-resource growth, and distributes work evenly.

  • Experimental Design: The evaluation measures strong scaling, weak scaling, comparison with existing inference, and load distribution across nodes.Experiments used a synthetic 100K-point dataset and specified initialization and optimization procedures, including PCA, k-means, L-BFGS, and scaled conjugate gradient.
  • Scaling with Computation Power: 1.93× and 1.90× runtime decreases resulted from doubling cores from 5 to 10 and 15 to 30, respectively.The observed scaling was close to the ideal inverse relationship between runtime and available cores.
  • Scaling with Data: A 30× dataset increase with a proportional increase in CPUs made total per-iteration runtime only 4.3% longer.The ideal distributable case would keep computation time constant.
  • Comparison to Standard Inference: The distributed inference significantly outperformed sequential, optimized GPy in running time when additional computational resources were available.This enabled sparse GPs and GPLVMs to run on datasets that standard inference would take too long to process.
  • Distribution of the Load: For 30 cores, the minimum and maximum node runtimes differed by 1.9%, indicating an approximately even distribution of computational load.Because Map-Reduce reduction waits for the slowest worker, the maximum node runtime determines the iteration bottleneck.

6 GP Regression and Latent Variable Modelling on Real-World Big Data

Experiments evaluate Gaussian-process regression on US flight data and latent-variable modelling on MNIST, showing improvements with more data alongside optimisation and modelling caveats.

  • Regression on US Flight Data: GP regression was evaluated on US flight data using subsets up to 2 million points and compared with multiple baseline models.The baselines included mean prediction, linear regression, ridge regression, and random forest regression.
  • Regression on US Flight Data: Figure 4 compares log likelihood against function evaluations for SCG and LBFGS on the 70K flight dataset.The comparison addresses optimisation behaviour as dataset size increases.
  • Regression on US Flight Data: With 2 million data points, the proposed inference required approximately 13.8 minutes per iteration on a 64-core machine.
  • Latent Variable Modelling on MNIST: The MNIST latent-variable experiments compared training on 10K points with the full 60K-point dataset using an additional 10K points for testing.The models were used for digit classification and reconstruction.
  • Latent Variable Modelling on MNIST: MNIST models occasionally became stuck in poor local optima, while no preprocessing was used because the experiment targeted benefits from larger training sets rather than state-of-the-art performance.
  • Latent Variable Modelling on MNIST: 3.03 percentage points lower classification error was observed when training on all 60K MNIST examples, decreasing error from 8.98% to 5.95%.

7 Conclusions

The paper concludes that distributed inference scales sparse GP regression and latent-variable modelling to millions of data points while retaining practical performance and balanced load distribution.

  • The distributed inference algorithm processes datasets with millions of data points for sparse GP regression and latent-variable modelling.
  • Inference scales with data and computational resources while preserving balanced load distribution across nodes.
  • The algorithm is implemented in Map-Reduce and released as an open-source package with documented derivations.
Loading 1402.1389v2…