Source-linked AI summary
Deep Gaussian Processes with Convolutional Kernels
Vinayak Kumar, Vaibhav Singh, P. K. Srijith, Andreas Damianou
TL;DR
Standard DGP kernels struggle with image variability, limiting their use for computer vision. The paper introduces convolutional kernels into deep GPs and evaluates these models on image classification benchmarks, reporting better generalization than state-of-the-art shallow GPs.
Problem
Standard RBF kernels in DGPs are inadequate for image variation, while combining convolutional structure with DGPs is challenging.
Source: §1 · p. 2Method
The paper places convolutional kernels inside deep GPs, exploring kernel combinations and random patch subsampling for image classification.
Source: §1 · p. 3, §3.3 · p. 10Results
Convolutional DGPs typically achieve better generalization than state-of-the-art shallow GPs across the evaluated image benchmarks.
Source: §1 · p. 3, §5 · p. 15Takeaways & Limitations
Convolutional kernels let DGPs combine image-structure capture with hierarchical latent representations for image classification.
Source: §3 · p. 8, §5 · p. 15
Abstract
from arXiv · showhide
Deep Gaussian processes (DGPs) provide a Bayesian non-parametric alternative to standard parametric deep learning models. A DGP is formed by stacking multiple GPs resulting in a well-regularized composition of functions. The Bayesian framework that equips the model with attractive properties, such as implicit capacity control and predictive uncertainty, makes it at the same time challenging to combine with a convolutional structure. This has hindered the application of DGPs in computer vision tasks, an area where deep parametric models (i.e. CNNs) have made breakthroughs. Standard kernels used in DGPs such as radial basis functions (RBFs) are insufficient for handling pixel variability in raw images. In this paper, we build on the recent convolutional GP to develop Convolutional DGP (CDGP) models which effectively capture image level features through the use of convolution kernels, therefore opening up the way for applying DGPs to computer vision tasks. Our model learns local spatial influence and outperforms strong GP based baselines on multi-class image classification. We also consider various constructions of convolution kernel over the image patches, analyze the computational trade-offs and provide an efficient framework for convolutional DGP models. The experimental results on image data such as MNIST, rectangles-image, CIFAR10 and Caltech101 demonstrate the effectiveness of the proposed approaches.
1 Introduction
The paper addresses the mismatch between standard DGP kernels and image data by bringing convolutional structure into a fully non-parametric deep GP. It evaluates convolutional variants and patch subsampling for image classification.
1 Introduction
- Deep parametric models can require millions of parameters, provide no predictive uncertainty estimates, and make model selection computationally expensive.DGPs are presented as having potential to address these limitations.Source: §1 · p. 2
- Earlier DGP work varies inference methods, including mean-field variational, amortized, nested variational, expectation-propagation, and random-feature approaches.One cited approach conditions the variational posterior on the previous layer to preserve cross-layer correlations.Source: §1 · p. 2
- Standard RBF kernels capture image similarity globally and fail to handle pose, illumination, and background variation as effectively as convolutional structure.The proposed approach puts convolutional structure inside the deep GP through kernels and remains fully non-parametric.Source: §1 · p. 2
- Convolutional kernels compare images through similarities between their substructures, and GP versions have been effective for object recognition.The image kernel sums a base kernel applied across different patches.Source: §1 · pp. 2–3
- Across image benchmarks, convolutional DGPs typically generalize better than state-of-the-art shallow GPs.The proposed variants are evaluated on MNIST, Rectangles-image, CIFAR10, and Caltech101.Source: §1 · p. 3
- The paper explores stacked convolutional and RBF kernels, weighted convolutional kernels, alternative base-kernel combinations, and random patch subsampling.Patch subsampling is proposed to improve computational efficiency.Source: §1 · p. 3
2 Background
The background introduces GP classification and sparse variational inference, then describes deep GPs as compositions of GP layers. It outlines the approximate inference used to train those models.
2.1 Gaussian Process
- A GP places a prior over real-valued functions, specified by a mean function and a covariance kernel.The kernel determines properties such as stationarity and smoothness; the RBF kernel is a popular choice for smooth functions.Source: §2.1 · p. 3
- In multi-class GP classification, each class has a latent function, and a soft-max link supplies class likelihoods.The posterior is not closed-form, so variational inference approximates it by optimizing a lower bound.Source: §2.1 · p. 4
- Sparse GP approximation uses M inducing variables to reduce covariance-inversion complexity from O(N^3) to O(NM^2), with M much smaller than N.The variational posterior over inducing variables yields an approximate posterior over function values.Source: §2.1 · p. 5
2.2 Deep Gaussian Process
- A DGP stacks GP layers, composing functions so that each layer maps a representation to the next and the final layer produces class probabilities.The paper follows a variant that absorbs noise between layers into the kernel.Source: §2.2 · p. 6
- DGP posterior inference is intractable because earlier-layer representations occur nonlinearly in later-layer covariance matrices.The method uses Monte Carlo sampling for the expected log likelihood and mini-batch stochastic gradients, yielding doubly stochastic variational inference.Source: §2.2 · pp. 6–8
3 Convolutional Deep Gaussian Processes
CDGP combines convolutional GP kernels with deep GPs to capture image features invariant to image variations while learning functions through multiple layers in a Bayesian framework.
3 Convolutional Deep Gaussian Processes
- CDGP combines convolutional GP kernels with deep GPs, capturing salient image features invariant to image variations while learning functions through depth in a Bayesian framework.The authors describe the resulting model as well-calibrated for image classification.Source: §3 · p. 8
3.1 Convolutional kernels
- Convolutional kernels sum function responses over image patches, enabling non-local similarity between image regions under transformations.Unlike RBF kernels, they compare spatial neighborhoods rather than only global image similarity.Source: §3.1 · p. 8
3.2 Deep Gaussian processes with convolutional kernels
- The first convolutional layer forms image representations from multiple GP functions; later GP layers map those features using convolutional or RBF kernels.Weighted convolutional kernels assign different weights to patches.Source: §3.2 · p. 9
3.3 Reducing computational complexity through patch subsets
- Computing convolutional image similarities costs O(P^2); patch-space inducing points reduce specified matrix-entry computations to O(P) or constant time.First-layer full patch-pair computations still cost O(P^2), so the authors use random patch subsets to reduce that cost to O(|S||S′|).Source: §3.3 · pp. 10–11
4 Experiments
Across MNIST, Rectangles-Image, CIFAR-10, and Caltech-101, the experiments compare CDGP architectures with GP baselines. CDGP performs strongly on MNIST and Rectangles-Image; on CIFAR-10 it improves on DGP but trails CGP, while patch subsampling speeds Caltech-101 training with small accuracy losses.
4 Experiments
- The evaluation compares CDGP kernel architectures with SGP, DGP, and CGP across four image-classification datasets, using test accuracy and NLPP.The shared setup uses 100 inducing points, with patch sizes of 3 × 3 for Rectangles-Image and 5 × 5 for the other datasets.Source: §4 · p. 11
4.1 MNIST-10
- 98.66 accuracy and 0.0463 NLPP: the 2-layer weighted-convolutional/RBF CDGP performed best on MNIST, and all CDGP models outperformed DGP and CGP.A 4-layer CDGP was second best; using two RBF length scales as the convolutional base kernel reached 98.46 accuracy but did not improve results on this simple dataset.Source: §4.1 · pp. 12–13, Table 1 · p. 12
4.2 Rectangles-Image
- The 2-layer weighted-convolutional/RBF CDGP achieved the best Rectangles-Image performance, beating DGP, CGP, and SGP by a large margin.The authors report it as the highest accuracy achieved by a GP model on this dataset.Source: Table 2 · p. 12, §4.2 · p. 13
4.3 CIFAR-10
- On CIFAR-10, convolutional kernels boosted DGP performance by 10%, but CDGP did not approach CGP.The authors suggest a single convolutional GP layer may suffice for this dataset; across earlier experiments, 2-layer CDGPs typically outperformed deeper models.Source: Table 3 · p. 13, §4.3 · p. 14
4.4 Experiments with Random Sub-sampling of Patches on Caltech-101 Dataset
- Randomly using one-tenth of Caltech-101 image patches reduced CDGP1 training from 11 h 18 min to 1 h 15 min, with a 0.39% accuracy drop.For CDGP2, subsampling reduced training from 12 h 2 min to 1 h 19 min with a 0.69% accuracy drop; the reported accuracies were low after resizing images to 50 × 50.Source: §4.4 · pp. 14–15, Table 4 · p. 14
5 Conclusion
The conclusion presents CDGP as a convolutional DGP for image classification that learns hierarchical latent representations. It reports benefits on benchmark datasets, while noting limited gains from adding layers and computational challenges for high-dimensional data.
5 Conclusion
- CDGP incorporates convolutional kernels into DGPs to capture image structure and learn hierarchical latent representations for image classification.The experiments demonstrate their usefulness on MNIST, Rectangles-Image, and CIFAR-10.Source: §5 · p. 15
- Increasing CDGP depth did not bring much performance improvement, contrary to the authors’ expectation.The authors identify faster, more effective variational inference as future work to address limitations of convolutional DGPs.Source: §5 · p. 15
- Reducing convolutional-kernel computation and memory requirements remains future work for high-dimensional datasets.The authors expect this could allow larger mini-batches, lower stochastic-gradient variance, and faster optimization convergence.Source: §5 · p. 15