PaperlayerView exact revision on arXiv

Source-linked AI summary

Gaussian Processes and Kernel Methods: A Review on Connections and Equivalences

Motonobu Kanagawa, Philipp Hennig, Dino Sejdinovic, Bharath K Sriperumbudur

arXiv:1807.02582v1stat.MLcs.LG

TL;DR

The paper addresses the conceptual separation between Gaussian-process Bayesian methods and RKHS-based frequentist kernel methods, which complicates transferring results between them. It systematically reviews their equivalences and differences, highlighting connections such as posterior variance as a worst-case RKHS error and clarifying relationships across both frameworks.

  • Problem

    The paper addresses limited mutual understanding between Gaussian-process Bayesian methods and RKHS-based frequentist kernel methods, despite their close relationships.

  • Method

    The paper provides a modern systematic review that juxtaposes concepts and algorithmic quantities from Bayesian and frequentist kernel frameworks.

  • Results

    The review establishes fundamental equivalences and differences, including an RKHS worst-case-error interpretation of Gaussian-process posterior variance.

  • Takeaways & Limitations

    Posterior variance can inform frequentist kernel methods by quantifying a worst-case RKHS error, while the two approaches retain distinct hypothesis-space interpretations.

  • Takeaways & Limitations

    Some reviewed results require technical conditions, including invertibility of the kernel matrix in the stated Gaussian-process expressions.

Abstract

from arXiv · show

This paper is an attempt to bridge the conceptual gaps between researchers working on the two widely used approaches based on positive definite kernels: Bayesian learning or inference using Gaussian processes on the one side, and frequentist kernel methods based on reproducing kernel Hilbert spaces on the other. It is widely known in machine learning that these two formalisms are closely related; for instance, the estimator of kernel ridge regression is identical to the posterior mean of Gaussian process regression. However, they have been studied and developed almost independently by two essentially separate communities, and this makes it difficult to seamlessly transfer results between them. Our aim is to overcome this potential difficulty. To this end, we review several old and new results and concepts from either side, and juxtapose algorithmic quantities from each framework to highlight close similarities. We also provide discussions on subtle philosophical and theoretical differences between the two approaches.

1 Introduction

The paper systematically connects Gaussian processes and RKHS-based kernel methods, while clarifying differences in uncertainty, hypothesis spaces, and interpretation. It shows how results and quantities can transfer between the two frameworks across regression, convergence, integral transforms, and dependence measurement.

  • Gaussian processes model nonlinear functions probabilistically, whereas RKHS methods optimize empirical risk using a loss function.
  • Connections in regression: Using the same kernel, the posterior mean of Gaussian process regression equals the kernel ridge regression estimator.
  • Connections in regression: Posterior variance has an RKHS interpretation as a worst-case error, complementing its Bayesian role as uncertainty over latent quantities.
  • Hypothesis spaces: The Bayesian and frequentist approaches can impose similar structural assumptions while defining different hypothesis spaces: GP support is not identical to the corresponding RKHS.
  • Convergence: The review transfers convergence results between GP regression and kernel ridge regression, using RKHS approximation capability when the regression function lies in a slightly larger space.
  • Integral transforms and dependence: Beyond regression, the paper relates RKHS worst-case and GP average-case integral errors, and gives a GP formulation of HSIC that recovers Brownian distance covariance.

2 Gaussian Processes and RKHSs: Preliminaries

This section introduces positive definite kernels, Gaussian processes, and RKHSs, then characterizes RKHS structure through Fourier behavior and smoothness. It also establishes connections between kernel-induced function spaces and GP covariance-based representations.

  • Foundations: Positive definite kernels generate positive semidefinite Gram matrices and serve as covariance kernels for Gaussian processes.The same kernel notion underlies both GP and RKHS frameworks.
  • Gaussian processes: A Gaussian process is defined by finite collections of function values following multivariate normal distributions determined by a mean function and covariance kernel.Every positive definite kernel and mean function correspond to a Gaussian process.
  • Smoothness: For Matérn kernels, the RKHS norm captures weak derivatives through order s = α + d/2, while classical differentiability is guaranteed only through order α.This distinction follows from the Sobolev embedding theorem and is illustrated for α = 1/2.
  • Spectral characterization: Fourier transforms characterize RKHSs associated with shift-invariant kernels by linking spectral decay of the kernel to spectral decay and smoothness of its functions.The Fourier transform of the kernel determines RKHS membership and function regularity.
  • Spectral characterization: Square-exponential RKHS functions have exponentially decaying Fourier transforms, whereas Matérn RKHS functions have polynomial decay that accelerates as α increases.For Matérn kernels, the associated RKHS is norm-equivalent to a Sobolev space of order α + d/2.
  • Connections: The posterior mean of Gaussian process regression is exactly equal to the kernel ridge regression estimator, while kernel methods generally do not provide an error estimate.The absence of an error estimate may reflect reluctance to impose assumptions such as additive Gaussian noise.

3 Connections between Gaussian Process and Kernel Ridge Regression

This section compares Gaussian process regression and kernel ridge regression, showing equivalences between their estimators, regularization mechanisms, and error interpretations. It also identifies conditions and practical settings where these connections matter.

  • Problem setup: Regression estimates an unknown function from input-output pairs, while interpolation is the noise-free case with exact training values.The setup distinguishes noisy observations y_i = f(x_i) + ξ_i from interpolation, where ξ_i = 0.
  • Gaussian process regression: The GP posterior mean and covariance have closed-form expressions determined by the kernel matrix, observations, noise variance, and prior mean.The posterior mean is given by (16), and the posterior covariance by (17); these expressions can be derived without Bayes’ rule.
  • Equivalences: GP regression and kernel ridge regression produce identical estimators when λ = σ^2/n, making Gaussian noise and RKHS regularization equivalent smoothing mechanisms.Larger λ produces a smoother estimator, and larger noise variance corresponds to assuming a smoother latent function than the observed process.
  • Error estimates: Posterior variance has a frequentist interpretation as a worst-case RKHS error, complementing its Bayesian interpretation as an average-case error.The worst-case error becomes more pessimistic as noise variance increases, because the noise contribution makes the adversarial function harder to fit.
  • Weight-vector viewpoint: In noisy regression, the weight-vector formulation minimizes the sum of noise-free worst-case error and estimator variance, with noise variance acting as a regularizer.The regularization term σ^2∥w∥^2 equals the variance of the regression estimator based on noisy observations.

4 Hypothesis Spaces: Do Gaussian Process Draws Lie in an RKHS?

GP sample paths almost surely fall outside the RKHS associated with their covariance kernel, even though posterior means lie in that RKHS. The paper explains this distinction and characterizes GP sample spaces through related, larger RKHS constructions, with kernel-specific smoothness consequences.

  • Characterization: The paper explains the discrepancy using Karhunen–Loève and Mercer expansions, Driscoll’s theorem, and sample-path characterizations based on powers of RKHSs.Driscoll’s theorem gives a necessary and sufficient membership condition, while the power construction supplies the corresponding RKHS framework.
  • Core distinction: GP sample paths lie outside the covariance-kernel RKHS almost surely when that RKHS is infinite dimensional, whereas posterior means belong to it.Finite-dimensional RKHSs are an exception: a version of the GP sample can belong to the RKHS with probability 1.
  • Characterization: GP sample spaces can be represented by RKHSs constructed from powers of the covariance-kernel RKHS and corresponding powered kernels.This construction makes the larger sample space explicit rather than only testing membership in an unspecified RKHS.
  • Kernel-specific consequences: For square-exponential kernels, GP sample paths lie in an RKHS that is informally infinitesimally larger than the covariance-kernel RKHS.The powered RKHS approaches the original RKHS as θ approaches 1, reflecting the infinite smoothness of the resulting spaces.

5 Convergence and Posterior Contraction Rates in Regression

This section compares convergence and posterior contraction for Gaussian process regression and kernel ridge regression, recovering connections between their rates, regularization, and noise assumptions. It also characterizes how posterior variance contracts with sampling density, dimension, and kernel smoothness.

  • Convergence-rate connections: The paper recovers Gaussian-process convergence rates from kernel-ridge rates for regression functions with finite Sobolev smoothness.The comparison focuses on Sobolev regression and connects posterior contraction rates with kernel-ridge convergence rates.
  • Gaussian process regression: At α = β, Gaussian-process regression attains the minimax-optimal rate n^-2β/(2β+d).Here α denotes the GP sample-path smoothness and β the regression-function smoothness.
  • Kernel ridge regression: For optimal kernel-ridge rates, the Sobolev RKHS smoothness should be s = β + d/2, exceeding the smoothness β of the regression function.This requirement is explained through the GP modeling assumption that noise variance remains constant as sample size increases.
  • Kernel ridge regression: Kernel ridge regression can attain the same optimal rate even when the regression function lies outside the RKHS, provided the RKHS approximates it sufficiently well.Approximation accuracy is controlled by the regularization constant.
  • Convergence-rate connections: Constant GP noise variance corresponds to the kernel-ridge regularization schedule λ_n = cn^-1.The relationship is σ² = nλ_n, linking the Bayesian noise assumption to frequentist regularization.
  • Posterior contraction: Posterior variance contracts as design points become denser, with the contraction rate determined by input dimension and kernel smoothness.For equally spaced points, the rate is O(n^-(2s/d−1)), revealing a curse of dimensionality.

6 Integral Transforms

The paper interprets kernel-based quantities through both RKHS worst-case errors and Gaussian-process average-case errors, establishing equivalences that support cross-framework analysis. It also connects kernel mean shrinkage and dependence measures to Bayesian interpretations while identifying differences and open questions.

  • 6.1 Maximum Mean Discrepancy: MMD is both an RKHS worst-case integration error and a Gaussian-process average-case error.For GP(0, k) draws, MMD is the expected squared difference between integrals under P and Q, with variance MMD(P, Q; Hk).
  • 6.1 Maximum Mean Discrepancy: For characteristic kernels, equality of GP integrals almost surely is equivalent to equality of the underlying probability measures.This gives MMD and its GP interpretation a direct connection to distinguishing distributions.
  • 6.2 Sampling and Numerical Integration: Bayesian quadrature's posterior mean equals the kernel quadrature estimate, while its posterior variance equals the squared MMD between Pn and P.The posterior variance depends only on the kernel, design points, and measure, not on the integrand.
  • 6.2 Sampling and Numerical Integration: The shared error formulation lets worst-case RKHS results transfer to Bayesian quadrature, including posterior contraction rates.The paper cites this equivalence as a basis for transferring numerical-integration results between frequentist and Bayesian settings.
  • 6.2 Sampling and Numerical Integration: Kernel quadrature can impose non-negative weight constraints directly, whereas Bayesian quadrature expresses prior knowledge through probabilistic assumptions.This illustrates a practical difference between optimization-based and probabilistic formulations.
  • 6.3 Kernel Mean Shrinkage Estimator and Its Bayesian Interpretation: The spectral kernel mean estimator is reviewed as a shrinkage estimator with a Bayesian interpretation, although its theoretical superiority over the empirical estimator is not established.Its regularization controls a bias-variance trade-off, and under specified parameters its posterior mean coincides with the shrinkage estimator.
  • 6.3 Kernel Mean Shrinkage Estimator and Its Bayesian Interpretation: The paper notes that using a smoother kernel can break the coincidence between the Bayesian posterior mean and the shrinkage estimator, leaving an open question about whether it improves performance.The favorable empirical performance of θ = 1 with a square-exponential kernel is noted, but the smoother-kernel question remains unresolved.
  • 6.4 Gaussian Process Interpretation of Hilbert Schmidt Independence Criterion: HSIC is interpreted as the expected squared covariance between GP-generated nonlinear transformations of two random variables.This provides a probabilistic reading of a nonparametric dependence measure that is computable with kernels.

7 Conclusions

The conclusion presents kernels and Gaussian processes as central analytical notions, while emphasizing their equivalences and differences across frequentist and Bayesian frameworks. Clarifying these relationships supports knowledge transfer and highlights that the frameworks may be less distinct than they first appear.

  • Kernels and Gaussian processes occupy central roles in machine learning, statistics, and numerical analysis.
  • Kernel methods emphasize regularization and optimization, whereas Gaussian processes operate through marginal and conditional distributions.
  • The review covers both fundamental equivalences and differences between the two frameworks.
  • Clarifying these relationships facilitates transfer of knowledge and methods between the two approaches.
  • The discussion also notes that frequentist and Bayesian statistics may be less different than they initially appear.

A.1 Proof of Lemma 3.9

The proof combines the reproducing property, the Cauchy-Schwarz inequality, and norm bounds to establish the stated assertion.

  • The reproducing property rewrites the right-hand side of equation (38).
  • The Cauchy-Schwarz inequality provides an upper bound for the resulting expression.
  • A function with RKHS norm 1 supplies a lower bound for the right-hand side of equation (95).
  • The assertion follows by combining the lower and upper bounds with equation (95).

A.2 Proof of Corollary 4.13

The proof establishes the corollary through interpolation-space embeddings, Sobolev and Besov regularity, compactness, and eigenvalue or entropy-number arguments.

  • Interpolation spaces are introduced for Banach spaces, with the K-functional defining [E, F]θ,2 for 0 < θ ≤1.
  • The embedding G ,→F ,→E implies [E, G]θ,2 ,→[E, F]θ,2 through a comparison of K-functionals.
  • The RKHS embedding into a Sobolev space leads through interpolation to a Besov space and then into L∞(ν) when θm > d/2.
  • Compactness of the embedding id : Hkγ _ L2(ν) permits use of entropy-number and eigenvalue arguments.
  • Choosing p ∈(0, 1) such that 1−θ > p makes the right-hand side bounded and yields the asserted conclusion.

A.3 Proof of Proposition 6.4

The proof derives an HSIC expression from an earlier identity by representing the kernel through Gaussian-process expectations and centering feature terms.

  • The proof reduces the result to showing that the right side of equation (92) equals HSIC.
  • The identity k(x, x′) = Ef[f(x)f(x′)] connects the kernel to expectations over functions f.
  • Centered kernel and feature terms are defined by subtracting their expectations under PX and PY.
  • The centered tensor expression expands into four terms involving the centered and mean feature components.
Loading 1807.02582v1…