PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Neural Networks as Gaussian Processes

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein

arXiv:1711.00165v3stat.MLcs.LG

TL;DR

The paper addresses the missing Bayesian interpretation of kernels corresponding to multilayer random neural networks. It derives the equivalence between infinitely wide deep networks and GPs, develops efficient covariance computation, and evaluates Bayesian predictions on MNIST and CIFAR-10. NNGP performance is consistently competitive with gradient-trained networks, while optimized finite-width networks approach GP performance as width increases.

  • Problem

    Prior work established GP equivalence for infinite-width single-layer networks, but multilayer neural-network kernels had not been identified as covariance functions enabling fully Bayesian deep-network prediction.

  • Method

    The paper constructs an NNGP through a recursive deterministic kernel computation, develops an efficient covariance implementation, and performs exact GP inference for neural-network models.

  • Results

    NNGP performance is consistently competitive with gradient-trained neural networks on MNIST and CIFAR-10, and optimized finite-width networks approach NNGP performance as width increases.

  • Takeaways & Limitations

    GP priors provide predictions and uncertainty estimates for deep neural networks without stochastic gradient-based training, with competitive performance among fully connected models.

  • Takeaways & Limitations

    The study treats classification as regression on class labels, and whether SGD approximately implements Bayesian inference under typical practical conditions remains unresolved.

Abstract

from arXiv · show

It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.

1 INTRODUCTION

The introduction revisits the correspondence between infinitely wide deep neural networks and Gaussian processes, extending exact Bayesian inference from shallow to deep networks. It positions the Neural Network GP as a deterministic compositional-kernel framework and motivates empirical comparisons with trained networks.

  • 1 INTRODUCTION: Infinite-width fully connected networks with i.i.d. parameters compute functions drawn from Gaussian processes by the Central Limit Theorem.The single-hidden-layer kernel was already known, while the introduction motivates extending the correspondence to deeper networks.
  • 1 INTRODUCTION: The GP substitution enables exact Bayesian inference for regression using covariance matrices over training and test sets.Inference requires matrix construction and straightforward linear algebra.
  • 1 INTRODUCTION: The paper delineates the correspondence between deep, wide neural networks and GPs and applies it to Bayesian training on regression tasks.The construction assumes parameter distributions and nonlinearities satisfying the conditions needed for the Central Limit Theorem.
  • 1.1 RELATED WORK: Unlike deep kernel learning and stacked deep GPs, the proposed GP directly corresponds to a multilayer neural network and uses kernel parameters matching neural-network hyperparameters.The related-work discussion distinguishes the paper from compositional-kernel constructions that do not establish this equivalence.
  • 1 INTRODUCTION: The Neural Network GP uses a recursive, deterministic, differentiable computation of its kernel for generic pointwise nonlinearities in fully connected feedforward networks.The paper also develops an efficient method for computing the covariance function with fixed hyperparameters.
  • 1 INTRODUCTION: Experiments compare Bayesian NNGP predictions with gradient-trained neural networks on MNIST and CIFAR-10, varying depth, nonlinearities, dataset size, and parameter variances.The study treats classification as regression on class labels, while noting that softmax classification is a possible extension.

2 DEEP, INFINITELY WIDE NEURAL NETWORKS ARE DRAWN FROM GPS

In the infinite-width limit, fully connected neural networks with i.i.d. parameters induce Gaussian processes. For deep networks, their covariance is obtained recursively, enabling exact Bayesian prediction through standard GP computations.

  • Infinite-width correspondence: The Central Limit Theorem makes finite collections of network outputs jointly Gaussian as each hidden-layer width tends to infinity.The argument applies layer by layer, with successive widths taken to infinity so each layer receives GP-governed inputs.
  • Infinite-width correspondence: Deep neural networks correspond to GPs whose kernels are computed recursively from the previous layer's covariance and the activation function.The recursion uses a deterministic function F determined by the pointwise nonlinearity and iterates through all layers.
  • Bayesian inference: The equivalent Neural Network GP supports exact Bayesian prediction by forming training and test covariance blocks and evaluating the resulting Gaussian conditional distribution.The predicted distribution is obtained using standard matrix computations, with the covariance determined by depth, nonlinearity, and weight and bias variances.
  • Efficient computation: The covariance pipeline recursively evaluates Gaussian integrals for all training-training and training-test pairs, using analytic integration for some nonlinearities and numerical integration otherwise.The full procedure is deterministic and differentiable, and its kernel computation is typically faster than solving the associated linear system.
  • Efficient computation: The direct implementation is prohibitively expensive, but careful pipelining and identical input norms reduce the computational cost.The improved procedure uses shared preprocessing and pipelining rather than independently computing every datapoint-pair integral at every layer.

3 EXPERIMENTAL RESULTS

Experiments show that NNGPs often outperform finite-width trained networks, while wider networks increasingly resemble NNGP performance. NNGPs also provide uncertainty estimates correlated with prediction error, and their performance depends on signal-propagation regimes.

  • Performance: NNGPs often outperform finite-width networks on MNIST and CIFAR-10.The comparison uses test accuracy and mean squared error for best-performing NNGPs and SGD-trained networks at given widths.
  • Performance: Finite-width network performance approaches NNGP performance as layer width increases.The widest networks are also typically the best generalizing networks.
  • Uncertainty: NNGP prediction uncertainty is highly correlated with empirical prediction error.Each test point receives an explicit prediction-variance estimate from the Bayesian GP.
  • Performance: The best generalizing five-hidden-layer networks are consistently the widest across ReLU and Tanh experiments.The CIFAR-10 study filtered networks to those achieving 100% classification training accuracy.
  • Relationship to Deep Signal Propagation: NNGP performance tracks the neural-network signal-propagation phase diagram.Performance is high near criticality and can degrade rapidly with depth when kernel differences become too small for numerical precision.

4 CONCLUSION AND FUTURE DIRECTIONS

The paper establishes deep infinitely wide neural networks as Gaussian processes with deterministic kernels, enabling Bayesian predictions and explicit uncertainty estimates. Experiments show competitive performance and alignment with signal-propagation theory, while the relationship between SGD and Bayesian inference remains unresolved.

  • Conclusion: Deep neural networks correspond to Gaussian processes with compositional, deterministic, differentiable kernels.This correspondence enables exact Bayesian inference for regression through matrix computations.
  • Signal propagation: NNGP performance is best near the critical line predicted by deep signal propagation.Accuracy drops more quickly away from the phase boundary as kernel depth increases.
  • Future directions: Optimized finite-width neural-network performance approaches GP computation as width increases.The authors connect this observation to a possible relationship between stochastic gradient training and Bayesian inference.
  • Future directions: Further investigation is needed to determine whether SGD approximately implements Bayesian inference in practical settings.The paper identifies this as an unresolved question rather than an established result.
  • Uncertainty: NNGP uncertainty estimates are explicit and may support failure prediction or active learning.The experiments report uncertainty that is highly correlated with prediction error.

A DRAWS FROM AN NNGP PRIOR

This section illustrates the NNGP prior by sampling one-dimensional functions for the ReLU nonlinearity at fixed depth and hyperparameters.

  • NNGP prior: The ReLU example uses a Gaussian process with zero mean and fixed depth L = 10.The construction is presented as an illustration of the GP prior’s nature.
  • NNGP prior: Samples from an NNGP prior represent different draws of one-dimensional functions.The plotted lines use arbitrary colors to distinguish draws.

B ANALYTIC FORM FOR KERNEL AND COMPARISON

The appendix gives an analytic ReLU kernel and compares it with the numerical covariance computation. The kernel’s angular structure flattens with depth while the two computation methods agree closely.

  • Analytic kernel: The ReLU covariance has an analytic arccosine form that includes weight and bias variance.The result is presented as a closed-form recurrence for the kernel.
  • Kernel evolution: Kernel angular structure flattens as depth l increases.Figure 6 plots K_l(θ) for ReLU inputs with equal norms.
  • Kernel comparison: The numerical kernel computation agrees closely with the analytic arccosine kernel.Figure 6 compares numerical results in blue with stars against the analytic form in red.

C BAYESIAN MARGINALIZATION OVER INTERMEDIATE LAYERS

An alternative derivation obtains the deep-network–GP equivalence by marginalizing over intermediate layer second-moment matrices. In the infinite-width limit, the network output is governed by a Gaussian process.

  • Marginalization: The derivation treats intermediate layer second moments as random variables and integrates over them.The output distribution is written as an integral over K_0 through K_L.
  • Layerwise model: Each layer’s pre-activations are Gaussian-process distributed when conditioned on its second-moment matrix.This conditional correspondence is exact even at finite width under Gaussian parameter priors.
  • Graphical model: The joint distribution decomposes according to the directed dependencies between successive activations and second-moment matrices.Each K_l depends on the preceding layer’s activations, while z_l depends on K_l.
  • Infinite-width limit: As layer width grows, Central Limit Theorem arguments make the relevant distributions Gaussian and concentrate intermediate quantities.The variance of the summed terms shrinks as 1/N_l, with the infinite-width limit producing a delta function.
  • Infinite-width limit: In the infinite-width limit, the network output z_L conditioned on x is described by a Gaussian process.This is the resulting equivalence established by the marginalization derivation.

D DETAILS OF THE EXPERIMENTS

The experiments specify dataset handling, hyperparameter searches, GP variance grids, numerical stabilization, and computational costs for NNGP evaluation.

  • MNIST used a 50k/10k/10k train/validation/test split, while CIFAR-10 used 45k/5k/10k.
  • Neural-network hyperparameters were optimized with random search, averaging 250 trials for each training-size, depth, width, and nonlinearity combination.
  • Learning rates and weight decay were sampled log-uniformly, while weight variance, bias variance, and batch size were sampled across specified ranges.
  • GP heatmaps evaluated 30-point grids for weight and bias variance, selecting the best run from 900 total evaluations.
  • For full MNIST, constructing each-layer covariance matrices took 90–140 seconds on 64 CPUs, while Cholesky solves took 180–220 seconds for 1,000 test points.
  • NNGP computations used precomputed lookup tables and increased target noise from 10^-10 by factors of 10 when Cholesky decomposition failed.

E FURTHER RESULTS

Further experiments examine uncertainty, variance settings, nonlinearities, and depth. They report that strong NNGP configurations cluster near the critical line and compare model settings across heatmaps and tables.

  • Uncertainty: The relationship between target MSE and GP uncertainty is shown for smaller training sets.
  • Best-performing NNGP hyperparameters are distributed near the critical line where the phase changes described in Section 3.2.
  • Test-set accuracy heatmaps evaluate NNGPs across a grid of σ2 values, with rows for Tanh and ReLU and columns for varying depth.
  • Traditional neural-network results use the best depth, width, and optimization hyperparameters, while NNGP results are reported for the best depth and σ2 settings.
Loading 1711.00165v3…