Source-linked AI summary
Gaussian Process Kernels for Pattern Discovery and Extrapolation
Andrew Gordon Wilson, Ryan Prescott Adams
TL;DR
Standard Gaussian-process kernels often provide smoothing and interpolation but are limited for automatic pattern discovery and extrapolation. The paper introduces closed-form stationary kernels by modelling spectral densities with Gaussian mixtures, retaining analytic inference. These kernels discover patterns and extrapolate over long ranges, including atmospheric CO2 measurements, while remaining interpretable and usable as drop-in replacements.
Problem
Standard Gaussian processes commonly use fixed smoothing kernels, while more expressive constructions can be specialized, difficult to interpret, or computationally demanding.
Method
The paper models a kernel's spectral density with a Gaussian mixture to construct expressive, closed-form stationary kernels for Gaussian processes.
Results
The proposed kernels discover patterns and accurately extrapolate atmospheric CO2 measurements over a long range, with the true measurements contained in the 95% HPD region.
Takeaways & Limitations
The kernels provide interpretable, expressive drop-in replacements for popular kernels while retaining simple training and inference procedures.
Takeaways & Limitations
Related expressive-kernel approaches typically target specific structures, use simple interpolating component kernels, and require sophisticated approximate inference.
Abstract
from arXiv · showhide
Gaussian processes are rich distributions over functions, which provide a Bayesian nonparametric approach to smoothing and interpolation. We introduce simple closed form kernels that can be used with Gaussian processes to discover patterns and enable extrapolation. These kernels are derived by modelling a spectral density -- the Fourier transform of a kernel -- with a Gaussian mixture. The proposed kernels support a broad class of stationary covariances, but Gaussian process inference remains simple and analytic. We demonstrate the proposed kernels by discovering patterns and performing long range extrapolation on synthetic examples, as well as atmospheric CO2 trends and airline passenger data. We also show that we can reconstruct standard covariances within our framework.
1. Introduction
Gaussian processes offer Bayesian nonparametric inference over functions, but standard kernels often smooth and interpolate rather than discover hidden patterns. The paper proposes stationary kernels that retain analytic inference while supporting automatic pattern discovery and extrapolation.
- Gaussian processes place a prior directly over functions and allow analytic posterior inference under Gaussian noise.
- Kernel choice controls properties such as smoothness and periodicity and can strongly affect Gaussian-process performance.
- Squared-exponential Gaussian processes mainly learn a length-scale, making them smoothing devices with fixed basis functions rather than adaptive feature discoverers.
- More expressive kernel constructions can target specialized structures but often produce difficult-to-interpret kernels and require demanding approximate inference.
- Kernel compositions can overfit, create unmanageable hyperparameter inference, and make the resulting statistical structure difficult to identify.
- The proposed kernels preserve stationarity, have simple closed forms, and are designed to discover patterns and extrapolate beyond available data.
2. Gaussian Processes
Gaussian processes define distributions over functions through a mean function and covariance kernel. The kernel determines function properties, while Gaussian noise enables analytic prediction and marginal-likelihood-based hyperparameter inference.
- A Gaussian process is a collection of random variables whose finite subsets have joint Gaussian distributions.
- The mean function and covariance kernel define the Gaussian-process distribution over functions and the covariance matrix for observed inputs.
- The kernel determines function properties such as smoothness and periodicity.
- The squared-exponential kernel produces infinitely differentiable functions and can represent long-range trends.
- With Gaussian noise, Gaussian processes support analytic posterior prediction and marginal-likelihood inference for hyperparameters.
3. Kernels for Pattern Discovery
The paper constructs expressive stationary kernels by modelling their spectral densities with Gaussian mixtures, yielding closed-form kernels that support pattern discovery, extrapolation, and negative covariances while retaining analytic GP inference.
- The proposed kernel class supports pattern discovery, extrapolation, and negative covariances while covering a large set of stationary kernels.
- Bochner’s theorem represents stationary covariance functions through a positive finite measure, whose density is the kernel’s spectral density.
- A spectral density entirely determines the properties of a stationary kernel, whereas squared-exponential mixtures cover only Gaussian spectra centered at the origin.
- Non-zero-mean Gaussian mixtures provide wider spectral densities and can approximate any stationary covariance kernel to arbitrary precision with enough components.
- Gaussian-mixture spectral densities produce exact closed-form kernel expressions, making analytic Gaussian-process inference possible despite the class’s expressiveness.
- The kernels are interpretable drop-in replacements: weights indicate component contributions, inverse means give periods, and inverse standard deviations give length-scales.
4. Experiments
Experiments show that spectral mixture kernels discover diverse covariance structure, recover standard stationary kernels, and support long-range extrapolation beyond interpolation.
- 4.1. Extrapolating Atmospheric CO2: The spectral mixture kernel uses automatic relevance determination to select useful frequency components from an initial mixture.In the CO2 experiment, seven of ten components were used after training.
- 4.1. Extrapolating Atmospheric CO2: The learned CO2 spectral density identifies yearly, six-month, four-month, three-month, and one-month periodicities that the squared exponential kernel misses.Peak width represents uncertainty about each discovered feature and affects extrapolation confidence.
- 4.3. Negative Covariances: The SM kernel reconstructs negative covariances by learning an oscillatory covariance with a spectral peak at frequency 0.5, corresponding to a period of 2.Only three of four initial components remained active, and the learned covariance accurately matched the true covariance.
- 4.4. Discovering the Sinc Pattern: For the difficult sinc task, SM nearly perfectly reconstructs the missing region x ∈[−4.5, 4.5], while competing kernels miss the pattern beyond x = 0.5 from training data.The learned spectral density includes a narrow peak at 0.1, representing a repeated sinc pattern every 10 units.
- 4.5. Airline Passenger Data: On airline passenger data, SM both interpolates the training region and extrapolates complex patterns for years beyond the observations within a small 95% predictive band.The learned spectrum separates a smooth rising trend, yearly variation, and a three-month seasonal component.
5. Discussion
The paper shows that expressive closed-form kernels can support pattern discovery and long-range extrapolation while retaining simple GP training and inference. It presents these kernels as flexible, practical alternatives to popular kernels and identifies further opportunities for development.
- Expressive closed-form kernels enable Gaussian processes to discover patterns and extrapolate over long ranges.They retain simple training and inference procedures and can replace popular kernels such as the squared exponential kernel.
- The proposed Bayesian nonparametric approach generalises patterns from a small number of examples.
- Future work could integrate away the spectral density and apply Toeplitz methods to speed inference and predictions.The proposed directions use efficient Markov chain Monte Carlo for GP hyperparameters and recent Toeplitz methods for the SM kernel.