Source-linked AI summary
Deep convolutional Gaussian processes
Kenneth Blomqvist, Samuel Kaski, Markus Heinonen
TL;DR
Deep convolutional Gaussian processes address limitations of shallow GP image models by iteratively convolving GP functions to learn hierarchical probabilistic representations. They improve image-classification results over compared GP approaches and approach hybrid neural-GP performance, while computational cost limited exploration of further model variants.
Problem
Shallow GPs can be suboptimal, while neural networks may require many parameters and do not provide reliable predictive uncertainty estimates.
Source: §1 · p. 1Method
The model iteratively convolves GP functions over images, learning probabilistic representations of increasingly complex pattern combinations without a neural-network component.
Source: §1 · p. 1, §3.1 · p. 4Results
Takeaways & Limitations
The results show that a purely GP-based approach can approach the performance of hybrid neural-GP methods on the compared image-classification benchmarks.
Source: §1 · p. 1, §5 · p. 7Added computational cost limited experiments with stride 1 at the first layer and models with four or more layers.
Source: §5 · p. 7
Abstract
from arXiv · showhide
We propose deep convolutional Gaussian processes, a deep Gaussian process architecture with convolutional structure. The model is a principled Bayesian framework for detecting hierarchical combinations of local features for image classification. We demonstrate greatly improved image classification performance compared to current Gaussian process approaches on the MNIST and CIFAR-10 datasets. In particular, we improve CIFAR-10 accuracy by over 10 percentage points.
1 Introduction
Gaussian processes offer flexible Bayesian function models, but standard stationary kernels and shallow models can be suboptimal, while deep models can become degenerate. The paper proposes a convolutional deep GP for hierarchical image features and reports higher CIFAR-10 predictive accuracy than the prior GP state of the art.
1 Introduction
- Standard stationary kernels underperform, and shallow GPs struggle to capture non-stationary patterns and long-range interactions; deep GPs can become degenerate when their layers are not invertible.Source: §1 · p. 1
- CNNs encode hierarchical translation invariance, while neural networks can require many carefully optimized parameters and do not provide reliable predictive uncertainty estimates.Source: §1 · p. 1
- The earlier convolutional GP aggregates responses over image subpatches but cannot model pattern combinations in a single layer; a deep convolutional GP had not significantly improved on it.The earlier deep-kernel approach stacks a feature-extractor neural network with a GP predictor.Source: §1 · p. 1
- CIFAR-10 predictive accuracy rises from 65% to 76% with deep convolutions, and the approach reaches the performance of hybrid neural-network methods.The model is a fully Bayesian kernel method with no neural-network component.Source: §1 · p. 1
- The proposed deep convolutional GP iteratively convolves GP functions over images to learn probabilistic representations of increasingly complex pattern combinations.The model is a fully Bayesian kernel method without a neural-network component.Source: §1 · p. 1
2 Background
This background introduces image convolution and Gaussian-process priors, then describes inducing variables and variational inference as tools for GP modeling.
2 Background
- Images are treated as 3D tensors of height, width, and channels; RGB images have three color channels.Source: §2 · p. 2
2.1 Discrete convolutions
- A convolution slides a tensor-valued filter across an image and takes dot products with local regions, producing high responses for similar regions.Multiple filters produce output signals that can be stacked, and stride determines which image locations are used.Source: §2.1 · p. 2
2.2 Primer on Gaussian processes
- A Gaussian-process prior assigns a multivariate Normal distribution to function values at any finite set of inputs, with a kernel matrix encoding their covariances.The kernel makes predictions correlate according to input similarity.Source: §2.2 · p. 2
ZZu
- A small set of inducing variables u_j=f(z_j) augments a GP to construct a low-rank function posterior.The kernels KXX, KXZ, and KZZ respectively relate observed inputs, observed-to-inducing inputs, and inducing inputs.Source: §2.2 · p. 2, ZZu · p. 2
2.3 Variational inference
- Exact GP evidence optimization is generally intractable and has cubic complexity O(n^3), motivating stochastic variational inference.The variational approximation is optimized through the evidence lower bound, and its expected likelihood can be computed by numerical quadrature.Source: §2.3 · p. 2
3 Deep convolutional Gaussian process
The model stacks convolutional GP layers to build probabilistic image representations, then uses a convolutional-kernel GP classifier. Doubly stochastic variational inference trains the model by sampling likelihood estimates and optimizing the lower bound.
3 Deep convolutional Gaussian process
- The architecture stacks convolutional GP layers and ends with a GP classifier using a convolutional kernel.The intermediate representations have sizes W1 × H1 × C1 and W2 × H2 × C2.Source: §3 · p. 2, Figure 1 · p. 3
3.1 Convolutional GP layers
- Convolutional GP layers produce probabilistic representations: the first layer is Gaussian, while later layers are non-Gaussian.Each layer maps patch responses to a new image representation; the MNIST example produces 24 × 24 × 10 features after layer one and 20 × 20 × 10 after layer two.Source: §3.1 · pp. 3–4
3.2 Final classification layer
- The final classifier is a GP with a weighted convolutional kernel that aggregates patch similarities from the last convolutional representation.The kernel weights patch similarities, and inducing points for this layer lie in patch space.Source: §3.2 · p. 4
3.3 Doubly stochastic variational inference
- Doubly stochastic variational inference uses Monte Carlo approximation for the expected likelihood and stochastic gradient descent to optimize the lower bound.The approach is applied to a deep GP with convolutional kernels and patch-filter inducing points.Source: §3.3 · p. 5
4 Experiments
The experiments evaluate the model on MNIST and CIFAR-10 and report improved classification results with added convolutional depth. They also examine learned representations and the effect of channel count.
4 Experiments
- Evaluation uses MNIST and CIFAR-10, with comparisons primarily against the shallow convolutional GP and additionally against a hybrid neural-network GP and DenseNet.The benchmarks have standard training and test folds for direct performance comparisons.Source: §4 · p. 6
4.1 MNIST and CIFAR-10 results
- 58.65% to 73.85% on CIFAR-10 follows adding a convolutional layer to the weighted convolutional kernel GP; another layer raises accuracy to 75.9%, while MNIST error falls from 1.42% to 0.56%.The three-layer deep convolutional GP achieves the stated MNIST error.Source: §4.1 · p. 6
- The deep kernel learning method performs similarly to this model but uses a fully connected five-layer DNN and much more parameters.Source: §4.1 · p. 7
- Figure 5 samples show first-layer edge detectors and second-layer pattern extraction whose representations differ across object classes.Rows show image examples and columns show patch response channels.Source: §4.1 · p. 7, Figure 5 · p. 7
- CIFAR-10 embeddings develop class coherence in intermediate layers and show classification boundaries in the last layer; some patch-space regions remain uncovered by filters.The uncovered regions are described as indicating uninformative representations.Source: §4.1 · p. 7
- The two-layer model's ELBO rises through C = 16 response channels, declines at C = 32, and performance is best at approximately C = 10.Source: §4.1 · p. 7
5 Conclusions
The convolutional deep GP improves test results on the compared classification benchmarks over GP-based approaches and approaches hybrid neural-GP performance. The conclusion also notes a possible depth-related performance gain and computational limits on exploring further improvements.
5 Conclusions
- The model greatly improves test results on the compared classification benchmarks over GP-based approaches and approaches hybrid neural-GP performance.Its performance seems to improve as more layers are added.Source: §5 · p. 7
- Stride 1 in the first layer and models with four or more layers were not tested because added complexity increased computational cost.The authors believe both changes would increase performance.Source: §5 · p. 7
- For two-layer models, 10 and 16 patch response functions seem to perform best, while one or two struggle despite having the same number of inducing points.The comparison uses expected evidence lower bound on the training set.Source: Figure 6 · p. 8
- Rank degeneracy was not observed, possibly due to the multiple channel mappings and convolution structure.The paper contrasts its zero-mean patch response functions with identity-mean mappings used in a rank-preserving deep model.Source: §5 · pp. 7–8
- Stochastic Gradient Hamiltonian Monte Carlo and kernel interpolation are proposed as future directions for improving efficiency and modeling capacity.The authors leave both directions for future work.Source: §5 · p. 8