PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Gaussian Processes

Mark van der Wilk, Carl Edward Rasmussen, James Hensman

arXiv:1709.01894v1stat.MLcs.LG

TL;DR

Gaussian processes need kernel structures that generalise well on high-dimensional image inputs, beyond local metrics. The paper introduces convolutional kernels with patch-space inter-domain inducing variables for efficient inference, and reports improved image classification performance, including 1.22% MNIST error for a weighted convolution kernel. It also uses marginal likelihood to combine convolutional and RBF structure, while noting that direct convolutional-kernel computation is costly.

  • Problem

    Gaussian-process kernels commonly use local metrics, motivating structures that support stronger non-local generalisation for high-dimensional image data.

    Source: §1 · p. 1

  • Method

    The paper constructs convolutional kernels with patch-space inter-domain inducing variables and integrates them into sparse variational GP inference.

    Source: §2.2 · p. 3, §2.3 · p. 3, §3 · p. 4

  • Results

    1.22% error: the weighted convolution kernel outperforms 1.9% for the RBF kernel on full MNIST, while the multi-channel kernel reaches 35.4% versus 48.6% for the RBF kernel on CIFAR-10.

    Source: §3.2 · p. 6, §3.4 · p. 8

  • Takeaways & Limitations

    Convolutional structure can improve Gaussian-process image models, and marginal likelihood can select an additional RBF component to improve performance.

    Source: §3.2 · p. 6, §4 · p. 8

    Direct convolutional-kernel implementations are computationally infeasible because kernel evaluations scale with the number of patch pairs.

    Source: §3 · p. 4

Abstract

from arXiv · show

We present a practical way of introducing convolutional structure into Gaussian processes, making them more suited to high-dimensional inputs like images. The main contribution of our work is the construction of an inter-domain inducing point approximation that is well-tailored to the convolutional kernel. This allows us to gain the generalisation benefit of a convolutional kernel, together with fast but accurate posterior inference. We investigate several variations of the convolutional kernel, and apply it to MNIST and CIFAR-10, which have both been known to be challenging for Gaussian processes. We also show how the marginal likelihood can be used to find an optimal weighting between convolutional and RBF kernels to further improve performance. We hope that this illustration of the usefulness of a marginal likelihood will help automate discovering architectures in larger models.

1 Introduction

The paper asks how Gaussian-process kernels can encode the non-local generalisation benefits of deep architectures while retaining Gaussian-process advantages. It introduces convolutional structure directly in the process and distinguishes this from transforming inputs with a convolutional neural network.

1 Introduction

  • Gaussian processes provide flexible priors over functions, but their kernels largely rely on local metrics such as Euclidean distance.The paper frames richer kernel structures as a way to improve generalisation without abandoning Gaussian processes.Source: §1 · p. 1
  • Convolutional structures support non-local influence in image models and motivate equipping Gaussian processes with analogous structure.The proposed approach convolves the process itself rather than applying a deterministic neural-network transformation to kernel inputs.Source: §1 · p. 1

2 Background

The paper combines sparse variational Gaussian-process inference with inter-domain inducing variables and develops an additive convolution-inspired kernel. This design enables flexible basis functions while restricting interactions to local patches rather than all input dimensions.

2.1 Gaussian variational approximation

  • Sparse variational Gaussian processes use inducing variables and an ELBO to obtain scalable approximate posteriors, including for non-Gaussian likelihoods.The variational objective supports minibatch stochastic optimisation and approximates the exact model as resources increase.Source: §2 · p. 2, §2.1 · pp. 2–3

2.2 Inter-domain variational GPs

  • Inter-domain inducing variables replace point evaluations with linear-operator transformations, allowing more flexible effective basis functions within sparse GP inference.The paper constructs inducing variables alongside the convolutional kernel so the basis functions contain a convolution operation.Source: §2.2 · p. 3
  • Inter-domain inducing variables provide a different inducing-input space from the kernel input space, while preserving Gaussian-process structure through appropriate covariance substitutions.The resulting approximation is designed to fit existing sparse GP frameworks.Source: §2.2 · p. 3

2.3 Additive GPs

  • The convolutional kernel applies the same function to input patches, allowing adjacent pixels to interact while imposing an additive structure elsewhere.This restricts high-dimensional interactions more than a squared-exponential kernel while avoiding the independent-variable assumption of a simple additive model.Source: §2.3 · p. 3

3 Convolutional Gaussian Processes

The paper embeds convolutional structure in Gaussian-process kernels and pairs it with patch-space inter-domain inducing variables for tractable inference. Weighted and colour-aware variants improve image classification, while an RBF component further improves MNIST results.

3 Convolutional Gaussian Processes

  • The method constructs convolutional kernels alongside a patch-space inducing domain, enabling sparse variational GP inference with convolutional effective basis functions.The implementation requires minimal changes to sparse variational GP inference and can leverage GPU convolution operations.Source: §3 · p. 4, §3.1 · pp. 4–5
  • The convolutional construction sums one shared nonlinear patch-response GP across image patches, allowing distant patches to inform that shared response function.Unlike convolutional networks' many linear filters followed by nonlinearities, adding more filters here simply produces a summed prior kernel.Source: §3 · p. 4
  • Direct convolutional GP computation is prohibitive because image-image kernel evaluations require P^2 patch-kernel evaluations, with about 3.3 · 10^5 for 5 × 5 MNIST patches.Patch-space inducing points reduce Kfu computation from NMP^2 to NMP evaluations and Kuu computation from M^2P^2 to M^2 evaluations.Source: §3 · p. 4, §3.1 · p. 5

3.1 Translation invariant convolutional GP

  • The translation-invariant convolutional GP reaches 1.4% error and 0.055 nlpp on rectangles using 16 inducing points, versus 5.0% and 0.258 for an RBF with 1200 inducing points.Its patch responses can recognise and count horizontal and vertical bars across image locations.Source: §3.1 · pp. 4–5
  • The same translation-invariant kernel obtains 0.3% error on MNIST zeros-versus-ones, but reaches about 2.1% on full MNIST versus 1.9% for the RBF kernel.This motivates weighting patch responses because identical features at different image locations can imply different classes.Source: §3.1 · p. 5, §3.2 · p. 5

3.2 Weighted convolutional kernels

  • Weighted patch responses relax full translation invariance, reducing full-MNIST error to 1.22% versus 1.9% for the RBF kernel.The weighting introduces one optimised kernel hyperparameter per patch location and improves edge handling in the rectangles task.Source: §3.2 · pp. 5–6

3.3 Does convolution capture everything?

  • 1.17% error and 0.039 nlpp are achieved on full MNIST by combining weighted convolutional and RBF kernels.The convolutional and RBF variances are 14.3 and 0.011, respectively, so the convolutional component explains most of the data variance.Source: §3.3 · p. 7, Figure 2 · p. 7, Table 1 · p. 7

3.4 Convolutional kernels for colour images

  • For CIFAR-10, colour-aware variants model channel structure through full-colour patches, separate channel functions, or additive structure, with multi-channel achieving 35.4% error versus 48.6% for RBF.The multi-channel model uses separate inducing outputs while sharing inducing inputs across channel-specific patch-response functions.Source: §3.4 · pp. 7–8, Figure 3 · p. 8

4 Conclusion

The paper introduces an efficient convolutional structure for Gaussian processes and shows that marginal likelihood can balance convolutional and fully connected structure. The appendix metadata is unrelated to the conclusion.

4 Conclusion

  • The method efficiently embeds convolutional structure in Gaussian processes through inducing inputs placed in patch space, yielding a natural inter-domain approximation.This fits sparse GP approximation frameworks and requires minimal implementation changes.Source: §4 · p. 8
  • Marginal likelihood can determine the balance between convolutional structure and a small amount of fully connected RBF structure.The conclusion states that convolutional structure alone is not sufficient for the best performance.Source: §4 · p. 8
  • The study applies several convolutional-kernel variants to image datasets and reports improved Gaussian-process performance.Source: §4 · p. 8

Convolutional Gaussian Processes: Appendix

A Implementation

The implementation reuses standard inducing-variable GP inference by changing how covariances are computed for inter-domain inducing variables. This confines the required inference changes to kernel and covariance evaluation.

A.1 Inter-domain approximations

  • Standard inducing-point approximations use function evaluations at data inputs and inducing locations for f and u.Source: §A.1 · p. 11
  • Inter-domain approximations replace ordinary inducing-point evaluations with different covariance expressions while leaving the rest of the learning objective unchanged.The proposed inference method therefore requires only modified Kuu and Kfu evaluations.Source: §A.1 · p. 11

A.2 Exploiting convolutions

The implementation accelerates convolutional-kernel computations by exploiting stationarity and the overlap among image patches. Inner products become convolution operations, reducing distance-computation cost and enabling optimized GPU code.

A.2 Exploiting convolutions

  • A general implementation can extract image patches, evaluate the kernel, and sum over patches, but this creates a large PN × M kernel matrix.Source: §A.2 · p. 11
  • For stationary kernels, patch-to-inducing-point distance computation can exploit image-patch overlap instead of treating every patch independently.This avoids the general large-kernel-matrix route for the first computation step.Source: §A.2 · p. 11
  • Inner products between an inducing patch and all patches of an image form a convolution, while patch self-inner-products form a convolution of the squared image with a ones window.Together these operations reduce Euclidean-distance computation from O(E) to O(log E).Source: §A.2 · p. 11, Figure 1 · p. 12
  • The convolutional formulation can leverage highly optimized GPU convolution code developed for convnets.Source: §A.2 · p. 11
  • The paper was presented at the 31st Conference on Neural Information Processing Systems in Long Beach, California.Source: §A.2 · p. 11
  • Figure 1 illustrates the convolution required for stationary kernels and highlights one dot product with a patch.Source: Figure 1 · p. 12

B The convolutional GP can not learn any function, but is nonparametric

Convolutional Gaussian processes are nonparametric but assign zero probability to some sets of functions. This restriction can aid generalisation while leaving some signal unexplained.

B The convolutional GP can not learn any function, but is nonparametric

B.1 Background: nonparametric models and consistency

Kernel methods’ representational capacity is determined by their basis functions, while universality requires coverage of continuous functions. Degenerate kernels lack this capacity, whereas common non-degenerate kernels such as the squared exponential are universal.

B.1 Background: nonparametric models and consistency

  • A kernel determines the functions a kernel method can represent through its eigenvalues and orthogonal eigenfunctions.Gaussian-process sample paths are formed from these eigenfunctions weighted by Gaussian random variables.Source: §B.1 · p. 12
  • Degenerate kernels have finitely many non-zero eigenvalues, so they cannot universally approximate functions with components in zero-eigenvalue directions.An infinite number of non-zero eigenvalues is necessary for universal approximation.Source: §B.1 · p. 13
  • A degenerate kernel with B basis functions fully constrains the function after B values, allowing perturbations outside its RKHS and prior.This finite-basis view explains why an additional function value is determined by the preceding B values.Source: §B.1 · p. 13
  • Universal kernels have RKHSs dense in the continuous functions, and Gaussian processes based on them place probability near every continuous function.Most common non-degenerate kernels, including the squared exponential, are universal.Source: §B.1 · p. 13

B.2 Convolutional kernels: nonparametric but not universal

Convolutional kernels are nonparametric but not universal: they retain infinitely rich representations while imposing constraints that exclude some continuous functions. The proof constructs singular covariance matrices and contrasts this behavior with universal kernels.

B.2 Convolutional kernels: nonparametric but not universal

  • Convolutional kernels are nonparametric because they cannot be represented as finite basis-function models, yet they are not universal.They can arbitrarily specify function values at infinitely many distinct points but still impose structural constraints.Source: §B.2 · pp. 13–14
  • Weighted covariance kernels are not universal, so Gaussian processes using them do not place probability on or near all continuous functions.This establishes the representational limitation directly at the GP-prior level.Source: §B.2 · p. 13
  • For images with smaller patches, single-pixel inputs can produce a linear transformation whose rank is at most the number of distinct patches.With N = wh + 1 inputs, one function value is fully determined by previous responses, forcing zero eigenvalues.Source: §B.2 · p. 13
  • The resulting singular covariance matrix means functions with components in its null space have no density under the prior, so the kernel is not strictly positive definite or universal.The construction provides the concrete mechanism behind the non-universality claim.Source: §B.2 · p. 13

B.3 Remarks

  • Convolutional kernels therefore provide a practically useful example of non-degenerate kernels that are nevertheless non-universal.The remarks connect this behavior to the strong implicit-feature requirements identified by Micchelli et al..Source: §B.3 · p. 14

C Variational bound for separate representation of latent GPs

The appendix represents additive convolutional GP components with separate inducing variables, preserving the sparse posterior construction while accommodating summed and multi-channel outputs.

C Variational bound for separate representation of latent GPs

  • The summed and multi-channel models require separate representations because their inducing inputs or inter-domain inducing outputs occupy separate spaces.The GP output can nevertheless be constructed from components represented in the same space as the inducing variables.Source: C Variational bound for separate… · p. 14

C.1 Defining the inducing variables

  • For summed kernels, M inducing evaluations are chosen for each latent GP, giving 2M inducing variables; multi-channel kernels use M evaluations for each of C outputs, giving MC variables.The sparse posterior is obtained by conditioning the prior on these variables.Source: §C.1 · p. 15
  • Summed-kernel covariances use a block-diagonal Kuu because the RBF and convolutional latent processes are independent in the prior.The cross-covariance matrix combines regular RBF evaluations with the appropriate patch covariances.Source: §C.1 · p. 15
  • Multi-channel kernels use M inducing inputs but MC inducing outputs, with cross-covariances and Kuu ordered by colour channel.The channel covariance includes δcc′, reflecting independent channel processes in the inducing representation.Source: §C.1 · p. 15

C.2 Inference with block-diagonal Kuu matrices

Block-diagonal inducing covariances make variational inference efficient for summed and multi-channel convolutional kernels. The same inter-domain strategy also extends to GP kernels encoding general input invariances.

C.2 Inference with block-diagonal Kuu matrices

  • The marginal-likelihood bound requires expensive matrix operations only on M × M blocks, despite using more than M inducing variables.This remains true regardless of posterior mean-field assumptions.Source: §C.2 · p. 15
  • Block-diagonal Kuu lets posterior marginals and KL terms be computed by splitting inducing variables into correlated blocks.Mean-field structure removes cross-block parameters and summations without changing inverse sizes.Source: §C.2 · pp. 15–16

C.3 Summary

  • The blockwise computation enables efficient inference for the summed and multi-channel convolutional kernels considered in the paper.Expensive matrix operations are performed separately for each prior-covariance block.Source: §C.3 · p. 16

D Inter-domain inducing variables for general invariances

  • The inter-domain inducing-variable trick extends beyond convolutional kernels to kernels producing functions invariant under arbitrary transformations.It applies when the kernel has a summation structure associated with the invariance.Source: D Inter-domain inducing variables for… · p. 16

D.1 Specifying invariances in kernels

  • Invariance under a transformation group requires argumentwise kernel invariance, k(x, x′) = k(g(x), g′(x′)) for all group elements.The group consists of compositions of the underlying transformations.Source: §D.1 · p. 16

D.2 Constructing invariant kernels

Invariant kernels can be built by summing a base kernel over transformation orbits, but direct evaluation is computationally expensive. Inter-domain inducing variables place inducing inputs in the base function to reduce this cost, although empirical evaluation is left for future work.

D.2 Constructing invariant kernels

  • Orbit-summed base kernels construct Gaussian processes invariant to specified transformations, using a structure nearly identical to convolutional kernels.The orbit of x contains the points obtained by applying every element of transformation group G to x.Source: §D.2 · pp. 16–17, §D.4 · p. 17
  • Evaluating an invariant kernel for one point pair requires P^2 base-kernel evaluations, making fully translation-invariant MNIST kernels impractical at P^2 = 5.3 · 10^5.The example considers one-pixel translations with clipping and zero-padding, yielding a (27 × 27)^2 orbit-pair count for MNIST.Source: §D.2 · p. 17

D.3 Inter-domain inducing variables for invariant kernels

  • Inter-domain inducing variables place inducing inputs in the base function rather than using normal inducing inputs, enabling the convolutional inter-domain trick.This construction uses the base function in place of the convolutional patch-response function.Source: §D.3 · p. 17
  • The inter-domain construction significantly reduces the cost of evaluating the required kernels, while empirical demonstration and evaluation remain future work.The paper presents the derivation but does not provide empirical evaluation for these invariant-kernel constructions.Source: §D.3 · p. 17, §D.4 · p. 17

Related papers

Loading 1709.01894v1…