Source-linked AI summary
GPflow: A Gaussian process library using TensorFlow
Alexander G. de G. Matthews, Mark van der Wilk, Tom Nickson, Keisuke Fujii, Alexis Boukouvalas, Pablo León-Villagrá, Zoubin Ghahramani, James Hensman
TL;DR
GPflow addresses the challenge of building a fast, accurate, usable, and extensible Gaussian process library. It uses variational inference, automatic differentiation, TensorFlow, and GPU hardware, and reports a reduction in MNIST training time from about 41 hours to about 5 hours with a GPU.
Problem
GPflow targets the simultaneous demands of scalable and accurate approximate inference, broad model support, correctness, usability, and extensibility in Gaussian process software.
Method
GPflow uses variational inference as its primary approximation method and combines TensorFlow-based computation with automatic differentiation and GPU support.
Results
Adding a GPU reduced the reported MNIST training time from about 41 hours with 6 CPU threads to about 5 hours.
Takeaways & Limitations
GPflow provides a Gaussian process library combining scalable variational methods, concise differentiated code, TensorFlow computation, and GPU acceleration.
Abstract
from arXiv · showhide
GPflow is a Gaussian process library that uses TensorFlow for its core computations and Python for its front end. The distinguishing features of GPflow are that it uses variational inference as the primary approximation method, provides concise code through the use of automatic differentiation, has been engineered with a particular emphasis on software testing and is able to exploit GPU hardware.
1. Existing Gaussian process libraries
The paper situates GPflow among publicly available Gaussian process libraries, including MATLAB-based GPML and GPstuff and Python-based GPy.
- Existing Gaussian process libraries range from personal projects to major community tools.
- GPML and GPstuff are MATLAB libraries, while GPy is written primarily in Python and NumPy.GPy provides an intuitive object-oriented interface.
- GPy is identified as a key reference for GPflow’s contribution.
2. Objectives for a new library
GPflow is designed to balance speed, accurate approximate inference, broad functionality, correctness, usability, and extensibility better than existing packages.
- GPflow is designed to be fast, particularly at scale, while keeping necessary approximate inference accurate.
- The library aims to support varied kernel and likelihood functions while ensuring that implementations are verifiably correct.
- GPflow emphasizes an intuitive user interface and software that is easy to extend.
3. Key features for meeting the objectives
GPflow combines variational inference, automatic differentiation, GPU computation, a Python object-oriented front end, and strong testing and open-source practices.
- Key features for meeting the objectives: Variational inference is GPflow’s primary approximation method for handling non-conjugacy and scale.
- Key features for meeting the objectives: Automatic differentiation enables relatively concise code by removing the burden of implementing gradients.
- Key features for meeting the objectives: GPflow can leverage GPU hardware for fast computation across a broad variety of GPU capability.
- Key features for meeting the objectives: The library provides a clean object-oriented Python front end and emphasizes testing and open-source software principles.
- Key features for meeting the objectives: GPflow uses TensorFlow for core computations rather than NumPy, unlike GPy.The architecture is heavily influenced by GPy, but the TensorFlow foundation changes its general requirements.
4. Contributing GP requirements to TensorFlow
GPflow extended TensorFlow with Gaussian-process-specific linear algebra capabilities, including differentiated Cholesky decomposition and GPU triangular-system solving.
- TensorFlow represents computations as directed graphs whose nodes are Operations and whose edges are Tensors.Kernels implement Operations on specific devices such as CPUs or GPUs.
- GPflow added functionality for solving linear systems with common linear algebra algorithms.This capability was required for Gaussian process software.
- A new differentiable Cholesky Operation used a C++ implementation of a blocked Cholesky algorithm.
- GPflow contributed GPU solving for matrix triangular systems, which can bottleneck approximate inference.
5. Details of GPflow
GPflow supports exact and approximate inference, including variationally sparse methods intended to improve scalability while remaining close to the posterior. Its class-based Python architecture integrates with TensorFlow’s computational graph, alongside extensive testing and documentation.
- Inference: Variationally sparse methods address GP scaling with data points while keeping the approximation close to the posterior in a Kullback-Leibler sense.GPflow also supports exact inference where possible and other approximation methods, including MCMC.
- Inference: GPflow organizes inference options as classes, including VGP, SGPR, SVGP, and SGPMC.The inference classes are summarized in Table 2.
- Architecture: The Python layer uses an object-oriented interface while computational graphs provide a largely functional computational structure.GPflow is designed to make these two emphases coexist cleanly.
- Quality and usability: GPflow’s source code is openly available, continuous integration runs automated tests, and reported test coverage reaches 99%.A user manual is also provided.
6. Timed experiments
The timed experiment compares GPflow and GPy on multiclass GP classification with stochastic variational inference on MNIST. CPU speeds are similar, while adding a GPU substantially reduces the estimated training time.
- Experimental setup: The experiment measured 50 iterations of stochastic variational inference for multiclass GP classification on MNIST, comparing GPflow with GPy.CPU trials varied thread counts, and GPflow trials also added a GPU to the maximum CPU-thread configuration.
- Experimental setup: Figure 1 reports iterations of stochastic variational inference per second for GPflow and GPy, with error bars showing one standard deviation across five repeats.The experiments used a Linux workstation with an Intel Core i7-4930K CPU and an NVIDIA GTX Titan X GPU.
- Results: CPU speeds for GPflow and GPy are similar in the timing experiments.The comparison used functionally nearly identical algorithms in the two packages.
- Results: 41 hours with 6 CPU threads versus about 5 hours with a GPU is the reported estimated training-time comparison for GPflow.The paper describes the GPU gain as considerable for researcher workflow.