The Promises and Pitfalls of Deep Kernel Learning

1 INTRODUCTION

• DKL seeks to pair neural networks’ learned representations with Gaussian processes’ uncertainty estimates, while avoiding the multiple forward passes used by fully Bayesian neural networks.DKL maps inputs through a neural network into an intermediate feature space used as the GP input, with network parameters optimized as kernel hyperparameters.Source: §1 · p. 1
• The paper questions the claim that marginal-likelihood optimization automatically calibrates complexity and improves test performance as model flexibility grows.That expectation is tied to the log marginal likelihood’s data-fit and complexity-penalty decomposition.Source: §1 · p. 2
• Marginal-likelihood training can make DKL overfit more than a standard deterministic neural network when the model has many hyperparameters.This challenges the expectation that the marginal likelihood automatically calibrates model complexity and prevents overfitting.Source: §1 · p. 2
• The marginal likelihood can overcorrelate datapoints by trying to correlate all of them, rather than only points whose correlations matter.The authors contrast this with the usual case of selecting only a small number of hyperparameters and note implications for highly parameterized kernels and marginal-likelihood or ELBO optimization.Source: §1 · p. 2
• Stochastic minibatching can mitigate DKL overfitting in practice, while a fully Bayesian treatment can avoid it and retain the benefits of neural networks and Gaussian processes.The paper also identifies prior discussions of GP and DKL overfitting, while adding an explanation of the behavior and its mechanism.Source: §1 · p. 2

2 RELATED WORK

• Fully Bayesian deep-model methods can provide useful uncertainty estimates and reduce overfitting and validation-set hyperparameter tuning, but cost more than plain DNNs.Models that perform inference only in the final layer remain of interest because they provide uncertainty estimates in a single forward pass.Source: §2 · p. 2
• Deep belief networks were first used to pretrain neural feature extractors for GPs, followed by marginal-likelihood fine-tuning; later work removed pretraining and improved scalability with KISS-GP.The latter approach was called deep kernel learning.Source: §2 · p. 2
• Stochastic variational inference extended DKL to non-regression likelihoods and multiple outputs, while neural linear models can be viewed as DKL with a linear kernel.Neural linear models are equivalently Bayesian inference over a neural network’s last layer.Source: §2 · pp. 2–3
• Prior work reports poor calibration and substantial regression hyperparameter tuning in these models; proposed responses include Monte Carlo dropout, fully Bayesian inference, and neural-network regularization.The described regularized approaches still require validation-set tuning of some hyperparameters.Source: §2 · p. 3

3.1 GAUSSIAN PROCESSES

• A Gaussian process places a multivariate-normal prior on function values, with its mean and kernel defining the prior and the kernel matrix defining covariance across inputs.The section uses a zero mean function and describes closed-form latent-function predictions at test points.Source: §3.1 · p. 3
• GP kernel hyperparameters and the noise variance are typically learned by maximizing the log marginal likelihood, whose terms are called data fit and complexity penalty.The paper uses an automatic relevance determination squared-exponential kernel.Source: §3.1 · p. 3

3.2 DEEP KERNEL LEARNING

• DKL maps inputs through a neural network into feature space and applies a standard kernel there, jointly learning network weights and kernel hyperparameters through the marginal likelihood.This allows the GP to use learned data representations.Source: §3.2 · p. 3
• Exact GP inference scales as O(N^3), and exact inference is limited to Gaussian likelihoods, so the paper uses stochastic variational inference to form SVDKL.The variational approach introduces inducing variables in the neural network’s feature space and optimizes their variational parameters with model hyperparameters.Source: §3.2 · p. 3
• SVDKL maximizes an evidence lower bound that can handle unrestricted likelihoods through Monte Carlo estimation; with enough inducing points, it can approximate the marginal likelihood under stated conditions.The tightness result applies to Gaussian likelihoods and bounded inputs; empirical work also reports this behavior for non-Gaussian likelihoods.Source: §3.2 · p. 4

4 BEHAVIOR IN A TOY PROBLEM

• DKL's toy-problem fit is jagged and extrapolates wildly, unlike the smooth, well-fitting SE-kernel GP.The paper presents this as evidence that DKL can overfit.Source: §4 · p. 4
• With the same feature extractor and matched training procedure, DKL overfits more than a deterministic neural network trained with mean squared error.The comparison holds architecture depth, learning rates, full-batch training, and optimization steps constant.Source: §4 · p. 4
• Across five DKL initializations, training is unstable, and different local minima yield fits with different generalization properties.Reducing the learning rate improves stability only slightly.Source: §4 · p. 4
• The toy result motivates investigating when and why DKL's overfitting pathology arises, including on real datasets.Source: §4 · p. 4

5.1 REGRESSION

• For the toy dataset's five initializations, data-fit curves stabilize around -100 nats, so complexity terms account for most differences in final marginal likelihood.Source: §5.1 · p. 4
• For UCI datasets, Figure 4 reports each method’s train and test RMSE and log likelihood, averaged over 20 splits; lower RMSE and higher log likelihood are better, with one-standard-error bars.Source: Figure 4 · p. 5
• At the marginal-likelihood optimum, GP regression with learnable signal variance has a data-fit term of -N/2, leaving the complexity penalty to account for kernel-specific likelihood differences.The proof differentiates with respect to signal variance.Source: §5.1 · p. 5
• The complexity penalty favors smaller covariance log-determinants, which overparameterized kernels can achieve by correlating points even where correlations are unwanted.The paper reports near-unity correlations in DKL and relates this mechanism to feature collapse, where features collapse datapoints onto a low-dimensional surface.Source: §5.1 · p. 5

5.2 CLASSIFICATION

• On a 5,000-point MNIST subset, DKL reaches low loss but significantly overfits; fixed-net DKL does not overfit and has the best test accuracy by a small margin.The standard neural network reaches near-zero loss without worsening test accuracy.Source: §5.2 · pp. 5–6
• For well-separated MNIST classes, the neural network's data-fit term is already near its minimum, while DKL's complexity penalty can be further reduced by over-correlating points.Source: §5.2 · p. 6

6 DKL FOR REAL DATASETS

• The real-dataset experiments examine whether DKL's pathological behavior and practical successes extend across datasets and architectures.The UCI regression set spans different sizes and dimensions; ENERGY, POWER, and PROTEIN were selected as datasets expected to benefit from added depth.Source: §6 · p. 6, §6.1 · p. 6

6.1 DKL FOR UCI REGRESSION

• On UCI regression, full-batch DKL and neural networks both overfit, usually more severely for DKL; minibatching generally reduces overfitting for both.With minibatching, SVDKL is best among deep models on log likelihood except for PROTEIN, and generally outperforms SVGP.Source: §6.1 · p. 6
• UCI results show that higher ELBOs or marginal likelihoods do not reliably predict better test performance; stochastic minibatching, rather than improved marginal likelihood, explains DKL's successful performance.SVDKL can have worse ELBOs than (V)DKL despite generally better test performance.Source: §6.1 · pp. 6–7

6.2 DKL FOR IMAGE DATASETS

• On UTKFace age regression, pretrained SVDKL achieves the best test log likelihood and better test RMSE than neural networks, but a train–test gap remains.Pretrained SVDKL that can update its network obtains the best test log likelihood; the fixed-network variant obtains the best test RMSE.Source: §6.2 · p. 7

6.3 DATA AUGMENTATION

• With data augmentation, larger batches reverse the advantage of pretrained SVDKL over fixed-network SVDKL, so ELBO-driven finetuning can hurt performance.The authors report that last-layer Bayesian inference is worse than not being Bayesian in this setting.Source: Table 3 · p. 8, §6.3 · p. 8

6.2.1 Increasing the Batch Size

• Increasing batch size on UTKFace improves pSVDKL's ELBO and test RMSE but worsens test log likelihood, while fixed-network SVDKL outperforms it despite a worse ELBO.The neural-network results change little, consistent with stronger minibatch regularization for SVDKL than for standard neural networks.Source: §6.2.1 · p. 8

6.2.2 Image Classification

• On CIFAR-10, pSVDKL generally performs best at batch size 100, but increasing batch size hurts it and leaves fixed-network SVDKL ahead despite a worse ELBO.Plain SVDKL struggles to fit the classification task, indicating the importance of pretraining.Source: §6.2.2 · p. 8

• On UTKFace and CIFAR-10, fully Bayesian SVDKL improves uncertainty-related metrics over Bayesian neural networks in nearly every metric, with significant gains on those metrics.For CIFAR-10, the original pSVDKL also outperforms the Bayesian neural network on uncertainty metrics, even at larger batch sizes.Source: Table 4 · p. 9, §7 · p. 9
• HMC sampling of network weights and GP hyperparameters resolves overfitting in the toy problem, including with a subsampled dataset.The posterior retains data-localized concentration while showing greater uncertainty in outer regions than the standard SE fit.Source: §7 · p. 9, Figure 5 · p. 9
• Standard HMC does not scale to the larger datasets because it requires full-dataset gradients and acceptance-probability calculations; the authors therefore use SGLD with minibatches.They note SGLD adds scaled Gaussian noise to gradients, with memory and test-time costs as its main additional costs.Source: §7 · p. 9
• The proposed fully Bayesian treatment aims to prevent overfitting while combining neural-network representations with GP uncertainty estimates.Source: §7 · p. 9

8 CONCLUSIONS

• When stochastic regularization is limited, DKL can overfit more and train less stably than standard neural networks.The authors attribute DKL's good performance in other regimes mostly to implicit regularization from stochastic minibatching, which they say is stronger than for plain neural networks.Source: §8 · p. 9
• With many hyperparameters, marginal likelihood may correlate all datapoints rather than only those that should be correlated, so a higher value does not improve performance.The authors conclude that marginal likelihood is unreliable for model selection in this setting, as is standard maximum-likelihood training loss.Source: §8 · p. 9
• A fully Bayesian treatment of neural-network hyperparameters can overcome this limitation and improve performance over the less Bayesian approach.Source: §8 · p. 9

A PROOF OF PROPOSITION 1

• At the marginal-likelihood optimum, the data-fit term equals −N/2 when signal variance is a learnable hyperparameter.The proposition applies to valid kernels of the stated form, with learnable noise variance and any other kernel hyperparameters.Source: A PROOF OF PROPOSITION 1 · p. 13
• The proof differentiates log marginal likelihood with respect to noise variance, sets the derivative to zero, and substitutes the optimum into the data-fit term.Source: A PROOF OF PROPOSITION 1 · p. 13
• Moore et al. [2016] had proved the result's preceding step, using it to solve for optimal signal variance and speed optimization.They did not substitute the result into the data-fit term.Source: A PROOF OF PROPOSITION 1 · p. 13

B EXPERIMENTAL DETAILS

• Real-dataset experiments used TensorFlow 2 and GPflow for DKL, and ran on single GPUs.The GPUs were NVIDIA Tesla P100-PCIE-16GB and GeForce RTX 2080 Ti.Source: B EXPERIMENTAL DETAILS · p. 13

B.1 DATASETS

• MNIST uses 5,000 training examples and a 10,000-example test set; CIFAR-10 uses its standard 50,000/10,000 train-validation split as train/test.UTKFace uses 20,000 of 23,708 cropped images for training and the remainder for testing; ages are normalized.Source: §B.1 · pp. 13–14
• UCI results average 20 cross-validation splits, with normalization fit on training data; reported RMSE uses normalized targets, where 1 corresponds to predicting zero.Source: §B.1 · p. 13

B.2 MODELS

• For direct comparisons, DKL replaces the neural network's final fully connected layer with an ARD SE GP, keeping the number of layers matched.Neural networks use ReLU activations; SVDKL inducing points lie in feature space at the GP input.Source: §B.2 · p. 14
• The CIFAR-10 and UTKFace models use modified ResNet18 baselines with an added fully connected output layer; SVDKL uses 1,000 inducing points.CIFAR-10 feature width is usually 10, with the global-average-pooling output avoided as a potentially high-dimensional GP input.Source: §B.2 · p. 14

B.3 IMPLEMENTATION DETAILS

• Models use ADAM and comparable gradient-step counts and learning rates; UCI minibatched methods use dataset-specific batch sizes and training schedules.For UCI, the full-batch methods are capped at the lesser of the minibatched gradient-step count or 8,000 steps to support a fair overfitting comparison.Source: §B.3 · pp. 14–15
• CIFAR-10 and UTKFace experiments use distinct training schedules for neural networks, SVDKL, and their Bayesian variants, with data augmentation in selected runs.CIFAR-10 augmentation uses horizontal flips and random crops; UTKFace adds small weight decay and uses corresponding face-image flips and crops.Source: §B.3 · p. 15

C.1 TOY

• Different initializations produce different toy-data fits and marginal likelihoods; the highest values occur when the prior correlates all input-domain points, while training instability often yields lower values.The fits, correlations, and training curves cover the full toy dataset and a subsampled version.Source: Figure 6 · p. 16, §C.1 · p. 16

C.2 CHANGING THE FEATURE DIMENSION

• For UTKFace, two feature neurons are insufficient, while larger dimensions produce only minor performance changes.Source: §C.2 · p. 16, Table 5 · p. 17
• For CIFAR-10, at least 10 feature neurons are needed to fit well, with only minor performance differences above that dimension.Source: §C.2 · p. 16, Table 6 · p. 18

• Table 7 reports BOSTON results as means plus or minus one standard error averaged over the splits.Source: Table 7 · p. 19
• Table 8 presents results for ENERGY.Source: Table 8 · p. 19
• Table 9 presents results for KIN40K.Source: Table 9 · p. 19
• Table 10 presents results for POWER.Source: Table 10 · p. 20
• Table 11 presents results for PROTEIN.Source: Table 11 · p. 20