Deep Kernel Learning

Andrew Gordon Wilson; Zhiting Hu; Ruslan Salakhutdinov; Eric P. Xing

Source-linked AI summary

Deep Kernel Learning

Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, Eric P. Xing

arXiv:1511.02222v1Nov 6, 2015cs.LGcs.AIstat.MEstat.ML

TL;DR

Gaussian processes offer flexible, interpretable modeling but standard kernels can lack the adaptive representations of deep networks, while exact inference can be costly. The paper constructs scalable deep kernels by transforming spectral mixture kernels with deep architectures and efficient kernel algebra, then jointly learns them through Gaussian-process marginal likelihood. The resulting models achieve O(n) training and O(1) prediction and improve performance over scalable Gaussian processes and stand-alone deep architectures across diverse applications.

Problem
Standard Gaussian-process kernels can behave as simple smoothing devices, while neural networks involve many architecture and regularization choices and exact Gaussian-process inference scales poorly.
Method
The paper transforms spectral mixture kernels with deep architectures and combines local interpolation, inducing points, and Kronecker and Toeplitz algebra for scalable non-parametric kernels.
Results
The proposed kernels provide O(n) training and learning with O(1) prediction per test point and improve performance over scalable Gaussian processes and stand-alone deep architectures.
Takeaways & Limitations
The kernels can serve as drop-in replacements for standard kernels while combining deep architectures’ structural properties with kernel methods’ non-parametric flexibility.
Takeaways & Limitations
The discussion identifies metric learning for high-dimensional classification as a promising direction for future research.

Abstract

from arXiv · show

We introduce scalable deep kernels, which combine the structural properties of deep learning architectures with the non-parametric flexibility of kernel methods. Specifically, we transform the inputs of a spectral mixture base kernel with a deep architecture, using local kernel interpolation, inducing points, and structure exploiting (Kronecker and Toeplitz) algebra for a scalable kernel representation. These closed-form kernels can be used as drop-in replacements for standard kernels, with benefits in expressive power and scalability. We jointly learn the properties of these kernels through the marginal likelihood of a Gaussian process. Inference and learning cost $O(n)$ for $n$ training points, and predictions cost $O(1)$ per test point. On a large and diverse collection of applications, including a dataset with 2 million examples, we show improved performance over scalable Gaussian processes with flexible kernel learning models, and stand-alone deep architectures.

1 Introduction

The paper combines deep architectures with non-parametric kernels to address the complementary strengths of neural networks and Gaussian processes. Its scalable deep kernels improve performance while enabling efficient training and prediction.

Deep neural networks learn adaptive representations, whereas traditional Gaussian-process kernels typically act as simple smoothing devices.
The paper combines deep architectures with kernel methods to unite deep networks’ structural properties and kernel methods’ non-parametric flexibility.
Scalable deep kernels transform inputs with deep architectures and use spectral mixture covariances, inducing points, local interpolation, and structure-exploiting algebra.
O(n) training and learning and O(1) prediction per test point make the proposed kernels scalable beyond standard Gaussian-process computation.
The method reports substantially improved performance over standard Gaussian processes, expressive kernel-learning approaches, and deep neural networks across diverse datasets.

2 Related Work

Related approaches combine Gaussian processes or kernels with neural architectures, but often remain task-specific, parametric, or dependent on demanding training procedures. The proposed approach instead provides expressive, scalable, jointly trained non-parametric kernels.

Existing Gaussian-process neural models are promising but task-specific and require sophisticated approximate Bayesian inference.
A convolutional-network method with Fastfood RBF features is scalable and flexible but remains parametric and requires separate training and manual calibration.
The proposed method combines deep feedforward or convolutional architectures with spectral mixture covariances and scalable kernel approximations.
The kernels are trained jointly with a unified supervised objective in a non-parametric Gaussian-process framework without approximate Bayesian inference.
The model outperforms stand-alone deep architectures and Gaussian processes with advanced kernel learning procedures across a wide range of datasets.

3 Gaussian Processes

Gaussian processes define distributions over functions through covariance kernels and learn kernel hyperparameters using marginal likelihood. Standard exact inference is limited by cubic computation and quadratic storage in the number of training points.

A Gaussian process assigns jointly Gaussian distributions to collections of function values, with mean and covariance determined by its mean function and kernel.
The covariance matrix K_X,X contains pairwise kernel evaluations on training inputs, while K_X*,X contains covariances between test and training inputs.
RBF kernels encode stronger correlation between function values that are closer in Euclidean input space; shorter length-scales permit faster variation.
Kernel hyperparameters are learned by optimizing the Gaussian-process log marginal likelihood, which separates model fit and complexity terms.
O(n^3) computation and O(n^2) storage are required by standard Cholesky-based inference, with predictive variance costing O(n^2) per test point.

4 Deep Kernel Learning

Deep kernel learning constructs expressive Gaussian-process kernels by transforming inputs through deep architectures and learning the resulting kernel jointly. Local interpolation, inducing points, and structure-exploiting algebra make the model scalable while retaining a non-parametric representation.

Kernel construction: Deep kernels transform inputs with a deep architecture before applying RBF or spectral mixture base kernels.The transformation g(x, w) is parameterized by network weights, while spectral mixture kernels add flexibility beyond the base architecture.
Kernel construction: Spectral mixture kernels capture quasi-periodic stationary structure, while deep transformations capture non-stationary and hierarchical structure.The spectral mixture parameters comprise mixture weights, bandwidths, and frequencies.
Learning: All deep-kernel hyperparameters γ = {w, θ} are jointly learned by maximizing the Gaussian-process log marginal likelihood.The derivatives with respect to network weights use standard backpropagation, while base-kernel and input-transformation derivatives are computed through the chain rule.
Learning: The deep kernel can serve as a drop-in replacement for standard kernels because learning and inference follow the same Gaussian-process procedures.The model is treated as a single covariance-function unit rather than as separate kernel and architecture components.
Scalable representation: KISS-GP approximates the covariance as M K_U,U M^⊤ using local interpolation and inducing points on a regular lattice.The inducing-point covariance admits Kronecker and Toeplitz structure for fast matrix-vector multiplications without requiring grid structure in the data or transformed inputs.
Scalable representation: O(n+h(m)) training complexity and O(1) prediction cost enable scalable learning while allowing m ≈ n for near-exact kernel approximation.KISS-GP’s linear scaling contrasts with conventional scalable GP computations of O(m^2n + m^3).

5 Experiments

Experiments show that deep kernel learning improves predictive performance across diverse regression and vision tasks while retaining scalable computation. The learned kernels capture task-relevant structure, and DKL remains practical for large datasets and uncertainty-aware prediction.

UCI regression tasks: DKL strongly outperforms standard RBF Gaussian processes and alternative scalable kernel-learning methods on most UCI regression datasets.
UCI regression tasks: DKL achieves better results than identically structured stand-alone deep neural networks across all 16 UCI datasets, with spectral mixture base kernels adding further improvements.
UCI regression tasks: About 10% additional runtime is imposed by KISS-GP when combined with DNNs, approximately one order of magnitude less than typical DNN runtime.
Face orientation extraction: The learned convolutional representation isolates face orientation while filtering irrelevant identity and scale factors, and accuracy continues improving as more training data are used.
Face orientation extraction: Spectral mixture deep kernels learn richer frequency structure and stronger orientation-based correlations than RBF deep kernels, explaining their improved performance.
Face orientation extraction: DKL scales roughly linearly with the number of training instances while maintaining reasonable total training time when CNN pre-training and joint training are included.
Step function recovery: DKL-SM accurately represents step-function discontinuities and maintains reasonable uncertainty, whereas conventional GP kernels struggle with sharp covariance changes.

6 Discussion

Scalable deep kernels combine deep architectures with non-parametric kernel methods to improve expressive power and efficiency. The paper reports broad experimental applicability while identifying high-dimensional classification as a promising direction for future metric-learning research.

Scalable deep kernels combine deep architectures with the non-parametric flexibility of kernel methods.
The approach transforms inputs through a deep architecture and uses local kernel interpolation, inducing points, and structure-exploiting algebra for scalable kernel representations.
O(n) training and O(1) testing time make the resulting kernels drop-in replacements for standard kernels with improved expressive power and efficiency.
Expressive transformations learn the whole metric function rather than relying on pervasive Euclidean or absolute-distance measures.
The authors identify high-dimensional classification as a promising direction for future research on learned metrics.

A.1 Convolutional network architecture

The convolutional architectures support face-orientation extraction and digit-magnitude regression. They adapt a LeNet-based design with pooling, fully connected layers, and ReLU activations.

The convolutional networks are used for face-orientation extraction and digit-magnitude extraction tasks.
The architecture is based on LeNet and adds one or two fully connected layers for feature transformation.
Both pool1 and pool2 are max-pooling layers, while ReLU follows full3 and full4.
The MNIST digit-magnitude regression variant omits the full3 layer.