Source-linked AI summary
Stochastic Variational Deep Kernel Learning
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, Eric P. Xing
TL;DR
The paper tackles limitations of conventional kernels and deep kernel learning in representation power, scalability, stochastic training, classification, and correlated outputs. It proposes SV-DKL, combining additive Gaussian processes over deep features with jointly learned network and kernel parameters and structured stochastic variational inference. Across classification benchmarks, including millions of training points, SV-DKL reports improved or superior performance over stand-alone deep networks and scalable Gaussian-process alternatives.
Problem
Conventional smoothing kernels lack the representational power needed for long-range extrapolation, while prior deep kernels did not support stochastic training, multiple outputs, or deep architectures broadly.
Method
SV-DKL applies additive Gaussian-process kernels to subsets of deep-network features, jointly learns kernel and network parameters through marginal likelihood, and uses structured stochastic variational inference.
Results
SV-DKL achieves good predictive accuracy and scalability across classification tasks, with superior performance to alternative scalable Gaussian processes and stand-alone deep networks on many benchmarks.
Takeaways & Limitations
The approach provides a practical probabilistic non-parametric representation supporting classification, multi-task learning, non-Gaussian likelihoods, and mini-batch training.
Takeaways & Limitations
The paper assumes vectorial input-output pairs linked through a Gaussian process and observation model, and identifies uncertainty characterization and feature interpretation as future goals.
Abstract
from arXiv · showhide
Deep kernel learning combines the non-parametric flexibility of kernel methods with the inductive biases of deep learning architectures. We propose a novel deep kernel learning model and stochastic variational inference procedure which generalizes deep kernel learning approaches to enable classification, multi-task learning, additive covariance structures, and stochastic gradient training. Specifically, we apply additive base kernels to subsets of output features from deep neural architectures, and jointly learn the parameters of the base kernels and deep network through a Gaussian process marginal likelihood objective. Within this framework, we derive an efficient form of stochastic variational inference which leverages local kernel interpolation, inducing points, and structure exploiting algebra. We show improved performance over stand alone deep networks, SVMs, and state of the art scalable Gaussian processes on several classification benchmarks, including an airline delay dataset containing 6 million training points, CIFAR, and ImageNet.
1 Introduction
The paper addresses the need for expressive, scalable kernels that retain useful learning biases on large datasets. It introduces SV-DKL, extending deep kernel learning to broader likelihoods, outputs, architectures, covariance structures, and stochastic training.
- Gaussian-process kernels provide smoothing and interpolation but generally cannot learn the representations needed for long-range extrapolation.
- Expressive kernel approaches remain challenging in scalability, general applicability, and interpretability, while prior deep kernels addressed only single-output regression without stochastic training.
- SV-DKL enables classification, non-Gaussian likelihoods, multi-task learning, mini-batch training, deep architectures with many output features, additive covariance structures, and enhanced scalability.
- SV-DKL applies additive Gaussian-process base kernels to subsets of deep-network features and uses stochastic variational inference with local interpolation, inducing points, and structure-exploiting algebra.Its stated complexity is O(m1+1/D), versus O(m3) for efficient stochastic variational methods.
- The approach achieves good predictive accuracy and scalability across a wide range of classification tasks while retaining a probabilistic non-parametric representation.
2 Background
This section motivates scalable Gaussian-process inference and situates the proposed model within assumptions about vectorial inputs and outputs. Existing methods face computational, dimensionality, and stochastic-optimization constraints that the approach targets.
- The paper assumes vectorial input-output pairs in which each output is related to its input through a Gaussian process and observation model.For regression, the example uses a latent vector of independent Gaussian processes with Gaussian observation noise.
- Exact Gaussian-process inference requires O(n3) computation and O(n2) storage, with predictions costing O(n2) per test point.These costs typically limit standard Gaussian processes to at most a few thousand training points.
- KISS-GP combines inducing-point kernel structure and sparse local interpolation to support many inducing points and O(n) inference, but typically does not apply when D > 5.
- Gaussian-process marginal likelihoods do not ordinarily factorize, preventing direct stochastic gradient descent; variational posteriors over inducing points address this issue for regression.
- The proposed model targets multi-task classification, mini-batch training, and scalable interpolation without requiring low-dimensional input spaces.The paper frames scalability and flexibility as complementary goals for learning expressive kernels on datasets with millions of training points.
3 Deep Kernel Learning for Multi-task Classification
The proposed architecture transforms inputs with a deep network, applies additive Gaussian processes to feature subsets, and linearly mixes their outputs to model correlated responses. For non-Gaussian likelihoods, stochastic variational inference exploits interpolation and algebraic structure.
- The model applies a deep nonlinear transformation h(x, w) to observed inputs, producing Q features at the final network layer.
- J Gaussian processes with base kernels act on subsets of final-layer features, allowing additive modeling on relatively low-dimensional inputs.
- A mixing matrix A combines the Gaussian-process outputs to produce correlated multiple responses, providing the model’s multi-task structure.
- For classification, the architecture uses one-shot class-label encodings and a softmax observation model.
- The latent Gaussian processes are independent before the linear mixing layer A(f_i) = Af_i, which induces correlations among outputs.
- Inducing variables and local interpolation establish a sparse relationship between latent functions and inducing values, enabling efficient stochastic variational inference.
- The additive GP and linear mixing layers form a probabilistic deep network whose parameters, including network weights, are jointly trained through the Gaussian-process marginal likelihood.
4 Structure Exploiting Stochastic Variational Inference
The paper develops stochastic variational inference for non-Gaussian Gaussian processes using factorized likelihoods, minibatch gradients, local interpolation, and structure-preserving reparameterization. Grid structure, Kronecker algebra, and closed-form KL computation reduce sampling and optimization costs while retaining covariance structure.
- Variational objective: Non-Gaussian likelihoods make exact Gaussian-process inference intractable, motivating stochastic variational inference with fast sampling.The method targets intractable marginalizations while supporting stochastic gradient training.
- Variational objective: Likelihood factorization enables noisy lower-bound gradients from minibatches while jointly updating variational, kernel, and deep-network parameters.The variational parameters and model parameters are updated using minibatches of the full dataset.
- Structured sampling: Local kernel interpolation makes the latent function a deterministic interpolation of inducing variables, so sampling needs to represent uncertainty only in u.This avoids difficult approximate posteriors over the latent function.
- Structured sampling: Cholesky reparameterization preserves covariance structure, unlike conventional mean-field methods that assume diagonal variational covariance.The variational covariance is represented as S = L^T L.
- Scalability: Grid-based Toeplitz and Kronecker structure reduces sampling cost from O(m^2) to O(m^(1+1/D)) using fast matrix-vector products.Here, m is the number of inducing points and D is the base-kernel input dimension.
- Scalability: The KL divergence between Gaussian distributions has a closed form, while the Kronecker method computes its derivatives with structured algebra.The cited complexity is O(Dm^3).
5 Experiments
Across airline, UCI, and image classification benchmarks, SV-DKL combines deep representations with probabilistic kernel structure to improve accuracy while retaining practical scalability. Experiments also examine training efficiency and the interpretability of learned covariance relationships.
- Classification Results: SV-DKL consistently improves over stand-alone DNNs across the evaluated classification benchmarks.The comparisons cover the airline delay task, UCI datasets, and image benchmarks.
- Airline Delays: On the airline dataset, SV-DKL outperforms KLSP-GP, DNN, DNN+GP, and DNN+ in classification accuracy.The evaluation compares these models using the same core DNN architecture for DNN and SV-DKL.
- Scalability: SV-DKL runtime increases only slowly with inducing points below 2,000 and is more efficient than the compared scalable variational methods.At m = 70, SV-DKL adds 0.39s over DNN runtime of 0.27s.
- Airline Delays: SV-DKL saves over 60% of training time versus KLSP-GP while achieving over an 18% improvement in predictive accuracy on the full airline dataset.SV-DKL has modest additional overhead compared with stand-alone architectures, although the DNN used GPU training while SV-DKL optimized some parameters on CPU.
- UCI Classification Tasks: On UCI tasks, SV-DKL further improves on DNN+GP and shows particularly notable performance on Connect4 and Covtype.DNN+GP consistently improves upon DNN, while the extra hidden layer in DNN+GP can sometimes harm performance.
- Image Classification: On image benchmarks, SV-DKL outperforms CNNs and CNN+GP on all datasets, improves ResNet-20 accuracy on CIFAR10 from 0.901 to 0.910, and reaches 0.7067 on ImageNet.The ImageNet comparison uses a 20-category subset with an AlexNet variant as the base network.
- Interpretation: Learned covariance matrices capture within-class image correlations, while the mixing layer reveals cross-class relationships such as digit 9 with digits 0 and 6.These visualizations illustrate the model’s multi-output structure and its learned similarity representation.
6 Discussion
SV-DKL combines deep learning, stochastic variational inference, structure-exploiting algebra, and additive covariance structures to support scalable probabilistic modeling. The discussion frames kernel methods and deep learning as complementary, with expressive kernels especially valuable for large datasets.
- SV-DKL supports classification, non-Gaussian likelihoods, multi-task learning, and mini-batch training while achieving superior performance over alternative scalable GPs and stand-alone deep networks.
- The approach combines kernel methods’ probabilistic structure with deep learning’s learned representations.
- Expressive kernel functions are particularly valuable for using large datasets effectively.
- Deep learning automatically learns structure that would be difficult to feature-engineer in advance.
- The authors identify uncertainty characterization and interpretation of learned features as future uses for deep kernel learning.
A Negative Log Probability (NLP) Results
The negative log probability evaluation covers airline-delay, UCI, and image-classification tasks. Across these tasks, the authors report trends generally similar to those observed for classification accuracy.
- The authors generally observed trends similar to those from the classification accuracy results.
- Negative log probability results are reported for the airline delay dataset.
- Negative log probability results are reported for the UCI datasets.
- Negative log probability results are reported for image classification benchmarks.
B Stochastic Variational Inference for Deep Kernel Learning Classification
The classification inference procedure samples inducing variables and obtains latent-function samples through deterministic interpolation. It uses structured reparameterization and Kronecker structure to make variational computations efficient.
- The method assumes a variational posterior over the inducing variables.
- Jensen’s inequality is used in deriving the variational objective lower bound.
- Latent function samples are obtained from inducing-variable samples through the deterministic mapping.
- Cholesky reparameterization preserves covariance structure when sampling the inducing variables.
- Grid inducing points enable a Kronecker decomposition of the variational covariance factor.
- The sampling cost is O(m^(1+1/D)) when fast Kronecker matrix-vector products are used.
- Samples are used to evaluate expectations of functions of the inducing variables.
- The marginal-likelihood lower bound and its derivatives are developed using K := K_Z,Z notation.