Approximate Inference Turns Deep Networks into Gaussian Processes

1 Introduction

• DNNs scale to real-world learning but may overfit small datasets and be overconfident, while GPs suit small datasets and provide confidence estimates but lack scalability and require kernel choices [3].Source: §1 · p. 1
• Earlier work connects infinitely wide networks to GPs, but practical links between model training procedures remain unclear.Source: §1 · p. 1
• Gaussian posterior approximations for Bayesian DNNs, including Laplace approximation and variational inference, are equivalent to GP posteriors and connect deep-learning solutions and iterations to GP inference.Source: §1 · p. 1
• The approach derives GP kernels and nonlinear feature maps while training a DNN, and one resulting kernel is equivalent to the neural tangent kernel (NTK).Source: §1 · p. 2

1.1 Related Work

• Unlike prior kernels derived by averaging over weight priors, this approach uses posterior approximations and applies to finite-width DNNs; different approximations can yield other kernels.Source: §1.1 · pp. 2–3

2 Deep Neural Networks (DNNs) and Gaussian Processes (GPs)

• DNNs minimize a loss over weights and can use scalable stochastic-gradient methods, while the loss is typically nonconvex.Source: §2 · p. 3
• A finite-dimensional nonlinear feature map defines a linear basis-function model whose induced GP kernel is the feature inner product scaled by the prior precision.Source: §2 · p. 3
• GP posteriors have closed-form expressions and are usually unimodal, but inference costs O(N^3) and requires choosing a kernel.Source: §2 · p. 3
• The paper frames its central question as relating DNN stochastic optimization to GP closed-form inference in practical settings, without assuming infinite-width networks.Source: §2 · p. 3

• The Laplace step approximates the Bayesian posterior around a loss minimizer, which corresponds to a posterior mode and can come from deep-learning optimizers.Source: §3 · p. 4
• For the standard loss, the GGN choice uses Jacobian features, residuals, and the loss-output Hessian while omitting the second Hessian term.Source: §3 · p. 4
• The construction uses a transformed dataset and a Jacobian-feature linear model with noise precision from the loss Hessian at the minimizer.Source: §3 · p. 4
• The Laplace approximation with the GGN Hessian approximation equals the posterior of a constructed linear model.Source: §3 · p. 4
• The linear model's predictive distribution is recast as GP regression with a multidimensional K × K kernel.Source: §3 · p. 5
• The GP predicts transformed outputs and therefore differs from the DNN's predictions, though the approach can extend beyond Gaussian priors and DNN models.Source: §3 · p. 5
• The GGN approximation is accurate when the model fits the data well, making training residuals close to zero and the omitted Hessian term negligible.Source: §3 · p. 5

4 Relating Iterations of a Deep-Learning Algorithm to GP Inference via VI

• VOGGN uses a variational Gaussian approximation at each iterate, with a GGN Hessian approximation and Monte Carlo expectations, enabling GP inference without assuming infinite-width networks.The section motivates this construction because the Laplace approximation is valid only at the loss minimum.Source: §4 · p. 5
• At iteration t + 1, VOGGN’s Gaussian approximation equals the posterior of linear model (13), whose prior over weights is correlated and derived from q_t(w).The equivalent model uses transformed outputs and a Jacobian-based linear basis-function model.Source: §4 · p. 6
• The GP kernel for one VOGGN weight-space iteration is similar to the NTK, with covariance V_t incorporating the previous q_t(w) as a prior.The resulting GP predicts in transformed output space; predictions for y_t are described in Appendix B.Source: §4 · p. 6
• OGGN replaces expectations with a zeroth-order delta approximation; its fixed points minimize the loss while retaining Gaussian approximations linked to GP inference.Unlike VON, which optimizes a variational objective, the modified method is stated to converge to a loss minimum.Source: §4 · p. 6
• Diagonal VOGN and OGN reduce computational cost by avoiding large-matrix calculations, while kernel computation requires Jacobians and can use a data subset.Full-covariance VOGGN and OGGN are more expensive than RMSprop.Source: §4 · p. 6
• In the modified Snelson example, DNN2GP predicts larger uncertainty than the compared methods wherever data are missing.Figure 3 compares Laplace and VI approximations and includes a GP with an RBF kernel.Source: Figure 3 · p. 7

5.1 Comparison of DNN2GP Uncertainty

• Where data are unobserved in the Snelson task, DNN2GP provides greater uncertainty than the compared methods under both Laplace and VI approximations.The experiment removed observations between x = 1.5 and x = 3.Source: §5.1 · p. 7

5.2 GP Kernel and Predictive Distribution for Classification Datasets

• On MNIST, the Laplace posterior mean reflects correct classification at 99% test accuracy, while CIFAR-10 shows a noisier pattern at around 68%.The MNIST kernel matrix also shows learned correlations among class-grouped examples.Source: §5.2 · p. 7
• For CIFAR-10, predictive variance separates into aleatoric and epistemic components, with low model uncertainty and high label noise indicating limited model flexibility.The passages interpret the first variance term as aleatoric uncertainty and the second as resembling epistemic uncertainty.Source: §5.2 · pp. 7–8
• For digits 0 and 1 trained with VI, the kernel shows strong within-training-class patterns, slight correlations to out-of-class samples, and no overconfident predictions.The visualization covers all ten classes although training uses only digits 0 and 1.Source: §5.2 · p. 8

5.3 Tuning the Hyperparameters of a DNN Using the GP Marginal Likelihood

• On synthetic regression and UCI Red Wine Quality, training GP marginal likelihood selects hyperparameters giving low test error; synthetic train MSE overfits as δ decreases.The real-data tuning used the Laplace approximation and varied regularization, noise variance, and network width one at a time.Source: §5.3 · p. 9

6 Discussion and Future Work

• The theoretical results connect approximate inference on DNNs to GP posteriors and provide a natural way to combine the models.The paper also describes extracting feature maps and GP kernels by training DNNs.Source: §6 · p. 9
• Jacobian computation is a bottleneck that prohibits application to large problems; gradient-based alternatives using a different GGN approximation remain future work.Source: §6 · p. 9
• Exact inference on the derived GP remains computationally infeasible for large problems, though further approximations could enable inference on bigger datasets.Source: §6 · p. 9

A Proofs

• The appendix section is devoted to proving the theorems presented in the main text.Source: A Proofs · p. 12

A.1 Proof of Theorem 1

• The proof begins with a Gaussian Laplace approximation and expresses its natural parameters in terms of the loss gradient and Hessians.Source: §A.1 · p. 12
• Using the first-order stationary condition, the proof expresses both Gaussian natural parameters through gradient and Hessian quantities.Source: §A.1 · p. 12
• The proof then applies the GGN approximation to obtain the Laplace-GGN Gaussian weight approximation.Source: §A.1 · p. 12
• Completing the square yields a posterior for a linear basis-function model on transformed inputs and pseudo-outputs, with Gaussian noise and a Gaussian weight prior.Source: §A.1 · pp. 12–13

A.2 Proof of Theorem 2

• VON’s Gaussian approximation corresponds to a linear-model posterior on a transformed dataset, with the update derived using GGN and Monte Carlo samples.The main-text theorem is the special case S = 1.Source: §A.2 · pp. 13–14

A.3 Linear Model Corresponding to OGGN

• OGGN uses the same linear-model construction, evaluating its gradient and Hessian at the current iterate, equivalent to one sample set to that iterate.The Jacobian, residuals, and noise-precision matrices are defined at the current weights.Source: §A.3 · p. 15

B Approximating Posterior Predictive with DNN2GP Approach

• DNN2GP obtains posterior predictive distributions through GP regression rather than potentially noisy Monte Carlo prediction, and the approach extends from Laplace approximation to VI.For test input x*, the procedure first computes the Jacobian feature map and then uses the linear model.Source: B Approximating Posterior Predictive… · p. 15

B.1 Laplace Approximation and Squared Loss

• Under squared loss, the transformed output and true output have equal predictive variance, while the true-output mean adjusts the transformed-output mean by f_w*(x*) − J*(x*)w*.The paper uses this inversion procedure for the predictive distribution shown in Fig. 3.Source: §B.1 · p. 15

B.2 Laplace Approximation and Logistic Loss

• For Bernoulli likelihood, the transformed-output map is inverted using the sigmoid probability and its noise precision to obtain predictions for the true output.The inversion requires an invertible link function; the approach also generalizes to other Gaussian approximations and generalized-linear-model losses.Source: §B.2 · pp. 15–16
• The two predictive-variance terms are interpreted as aleatoric label noise and epistemic model noise, respectively.The paper attributes this interpretation to converting the DNN into a linear-basis-function model.Source: §B.2 · p. 16

B.3 Generalization to VI

• With VOGGN using multiple Monte Carlo samples, the method averages the sample-specific predictions to obtain the predictive distribution; OGGN and one-sample VOGGN use the corresponding single-map procedure.Source: §B.3 · p. 16

C.1 Further Posteriors and Kernels for MNIST and CIFAR

• The variational MNIST kernel contains more negative correlations between examples from different classes than the Laplace kernel, while their posterior means show very similar structure.Source: §C.1 · p. 16
• Both Laplace and variational kernels on CIFAR-10 show less structure than the MNIST kernel.The CIFAR-10 kernel comparison uses 300 data points.Source: §C.1 · p. 16
• Figure 7 visualizes LeNet5’s GP kernel matrix and posterior mean for VOGN on MNIST and CIFAR-10, with higher posterior means assigned to correct labels.The kernel matrix depicts correlations learned by the DNN.Source: Figure 7 · p. 16

C.2 Uncertainties according to DNN2GP for Classification

• On CIFAR-10, Laplace and variational GP kernels show slight structure, less pronounced than the MNIST kernels.Source: Figure 8 · p. 17
• The toy classifier’s predictive variance separates label noise near the decision boundary from model uncertainty that grows away from the data.Figure 9 shows low predictive uncertainty where data support the prediction and higher model uncertainty farther away.Source: Figure 9 · p. 17, §C.2 · p. 17
• The example fits a nonlinear half-moons decision boundary using a one-hidden-layer, 10-unit tanh network trained on 100 samples for 5000 full-batch epochs.The setting uses δ = 0.26.Source: §C.2 · pp. 16–17