PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Neural Networks as Gaussian Processes

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, Jascha Sohl-Dickstein

arXiv:1711.00165v3stat.MLcs.LG

TL;DR

Bayesian GP inference had been established for infinite-width single-layer networks, but the deep-network correspondence remained to be delineated. This paper derives the infinite-width deep-network GP and evaluates it on MNIST and CIFAR-10, finding that NNGP performance is competitive with or often surpasses gradient-trained networks while finite-width networks approach GP performance as width grows.

  • Problem

    Bayesian inference via GPs was established for infinite-width single-layer networks, while the correspondence for deep, wide networks remained to be delineated.

    Source: §1 · pp. 1–2

  • Method

    The paper constructs a deterministic compositional GP kernel from infinite-width deep-network priors and uses GP matrix computations for exact Bayesian regression.

    Source: §2.4 · p. 5, §4 · p. 9

  • Results

    On MNIST and CIFAR-10, best NNGP performance was consistently competitive and often surpassed conventional gradient-trained networks; best finite-width networks approached NNGP performance as width increased.

    Source: §1.2 · p. 3, §3.1 · p. 7

  • Takeaways & Limitations

    NNGPs provide explicit uncertainty estimates that may help predict model failure or select datapoints for active learning.

    Source: §4 · p. 10

    The experiments treat classification as regression on class labels; extending GPs to softmax classification with cross-entropy is left for future work.

    Source: §1.2 · p. 3

Abstract

from arXiv · show

It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.

1 INTRODUCTION

The paper identifies deep, infinitely wide fully connected networks with Gaussian processes, enabling Bayesian inference through corresponding function-space priors, and evaluates this construction against trained networks.

1 INTRODUCTION

  • In the infinite-width limit, the central limit theorem makes each output of a fully connected network with i.i.d. random parameters a draw from a Gaussian process, enabling exact Bayesian regression through the corresponding GP prior.This replaces the network’s weight-and-bias prior with a function-space prior and requires covariance matrices over training and test sets.Source: §1 · p. 1

1.1 RELATED WORK

  • Earlier work derived the one-layer infinite-width correspondence (Neal (1994a)) and single-hidden-layer GP kernels (Williams (1997)); deep-network kernel work had not established the general equivalence developed here.Hazan & Jaakkola (2015) treated at most two hidden layers with nonlinearities, while compositional-kernel and signal-propagation studies examined related recurrences and their behavior.Source: §1.1 · p. 2
  • Unlike deep GPs, which stack Gaussian processes, this work studies GPs directly corresponding to infinitely wide deep networks; unlike deep kernel learning, its kernel corresponds to the network itself.The work also distinguishes its general nonlinearities and neural-network variance hyperparameters from Cho & Saul (2009), and its equivalence contribution from Krauth et al. (2016).Source: §1.1 · p. 2

1.2 SUMMARY OF CONTRIBUTIONS

  • The NNGP kernel is computed recursively and deterministically for fully connected feedforward networks with generic pointwise nonlinearities, with an efficient covariance method for fixed hyperparameters.Its kernels include weight- and bias-variance hyperparameters corresponding to those of the equivalent neural network.Source: §1.2 · p. 2, §1.1 · p. 2
  • The proof-of-concept experiments perform exact Bayesian regression on MNIST and CIFAR-10 class labels, rather than softmax classification, while comparing predictions with gradient-trained networks.The authors note that softmax classification with cross-entropy is possible but leave it for future work.Source: §1.2 · p. 3
  • Across MNIST and CIFAR-10, the best NNGP settings are consistently competitive and often surpass gradient-trained networks; wider trained networks approach NNGP performance, whose uncertainty tracks prediction error.The NNGP’s performance also depends on kernel structure, which the authors connect to signal propagation in random networks.Source: §1.2 · p. 3

2 DEEP, INFINITELY WIDE NEURAL NETWORKS ARE DRAWN FROM GPS

The section establishes that infinite-width fully connected networks induce Gaussian processes: the central limit theorem yields the single-layer result, and induction extends it to deep networks. It also gives a recursive covariance construction for fully Bayesian prediction and an efficient kernel pipeline.

2.2 REVIEW OF GAUSSIAN PROCESSES AND SINGLE-LAYER NEURAL NETWORKS

  • As hidden width N1→∞, the central limit theorem makes each output Gaussian, and the multidimensional theorem makes any finite collection jointly Gaussian, yielding a Gaussian process.The resulting process has a mean and covariance independent of the output index.Source: §2.2 · p. 3

2.3 GAUSSIAN PROCESSES AND DEEP NEURAL NETWORKS

  • Induction extends the Gaussian-process correspondence to deep networks as layer widths tend to infinity successively, with each covariance computed recursively from three prior-layer covariance values.The recursion is a deterministic function whose form depends on the nonlinearity.Source: §2.3 · p. 4

2.4 BAYESIAN TRAINING OF NEURAL NETWORKS USING GAUSSIAN PROCESS PRIORS

  • Under the GP prior, conditioning on training data yields an exact predictive distribution through matrix computations, constituting fully Bayesian training of the deep network.The covariance function depends on the network’s depth, nonlinearity, and weight and bias variances.Source: §2.4 · p. 5

2.5 EFFICIENT IMPLEMENTATION OF THE GP KERNEL

  • A pipelined, constant-norm-preprocessed method computes kernels for well-behaved nonlinearities; the full pipeline is deterministic and differentiable.Because GPs provide exact marginal likelihood estimates, the construction may support principled hyperparameter selection or nonlinearity design.Source: §2.5 · pp. 5–6

3 EXPERIMENTAL RESULTS

NNGPs often outperform finite-width networks, whose performance approaches the GP as width increases. Their uncertainty tracks prediction error, and their accuracy is highest near criticality in the deep signal-propagation phase diagram.

3.1 DESCRIPTION

  • The comparisons use fully connected networks with equal hidden-layer widths, trained with MSE and Adam, against NNGPs on permutation-invariant MNIST and CIFAR-10.The study evaluates ReLU and Tanh networks and tunes learning rate and initial weight and bias variances on validation error.Source: §3.1 · p. 6
  • NNGPs often outperform finite-width SGD networks, whose test performance approaches the GP as layer width increases.Figure 1 compares test accuracy and mean squared error on MNIST and CIFAR-10; Table 1 reports test accuracy on both datasets.Source: §3.1 · pp. 6–7, Figure 1 · p. 7, Table 1 · p. 9
  • Among 180 five-hidden-layer CIFAR-10 networks trained with cross-entropy, the best-generalizing networks were consistently the widest.Figure 2 filters for networks with 100% classification training accuracy and includes 125 Tanh and 55 ReLU networks.Source: §3.1 · p. 7, Figure 2 · p. 8
  • NNGP prediction uncertainty is highly correlated with test-point prediction error, as shown by realized versus predicted mean squared error.Figure 3 bins test points by predicted MSE and averages each plotted point over 100 test points.Source: §3.1 · p. 7, Figure 3 · p. 8

3.2 RELATIONSHIP TO DEEP SIGNAL PROPAGATION

  • For Tanh networks, ordered-phase correlations converge to a common fixed point, whereas chaotic-phase correlations retain a distinct fixed point for dissimilar inputs.The phases reflect the competition between weight and bias variances; in the ordered phase, propagated features become similar even for dissimilar inputs.Source: §3.2 · p. 8
  • NNGP accuracy is high near criticality but degrades faster away from the phase boundary as depth grows, when kernel differences become numerically unrepresentable.Far from criticality, test performance is poor; the model’s data information depends on the difference between the depth-L kernel and its fixed point.Source: §3.2 · p. 9

4 CONCLUSION AND FUTURE DIRECTIONS

The paper establishes that infinitely wide deep neural networks correspond to Gaussian processes, enabling deterministic, exact Bayesian regression with uncertainty estimates. It also identifies critical signal propagation as a condition for best NNGP performance and outlines scalability and SGD inference as open directions.

4 CONCLUSION AND FUTURE DIRECTIONS

  • Infinite-width deep-network priors correspond to Gaussian processes with deterministic, differentiable compositional kernels, enabling exact regression inference and uncertainty estimates without stochastic-gradient training.Performance is competitive with the best neural networks in the specified fully connected class under similar hyperparameter settings.Source: §4 · p. 9
  • The approach was tested on datasets of up to 50k examples, while scaling to larger learning tasks remains future work.The authors suggest potentially leveraging scalable-GP methods (Quiñonero-Candela & Rasmussen, 2005; Hensman et al., 2013).Source: §4 · p. 9
  • The NNGP’s best performance occurs along the critical line predicted by deep signal propagation.This agreement is shown for the tested nonlinearities and depths.Source: Figure 4 · p. 10, §4 · p. 10
  • Optimized neural-network performance appears to approach GP computation as width increases, but whether SGD approximately implements Bayesian inference in typical practice remains open.The paper calls for further investigation of this question (Mandt et al., 2017).Source: §4 · p. 10
  • NNGP uncertainty estimates may help predict model failure in critical applications and identify datapoints to hand-label for active learning.Source: §4 · p. 10

A DRAWS FROM AN NNGP PRIOR

Figure 5 depicts sample 1D functions drawn from an NNGP prior, illustrating the prior for a ReLU Gaussian process.

A DRAWS FROM AN NNGP PRIOR

B ANALYTIC FORM FOR KERNEL AND COMPARISON

The ReLU kernel has an analytic form, and its angular structure flattens with depth while agreeing well with the numerical kernel computation.

B ANALYTIC FORM FOR KERNEL AND COMPARISON

  • Figure 6 shows the angular structure flattening with depth and good agreement between the numerically computed kernel and the analytic arccosine kernel.The comparison uses the ReLU nonlinearity and a particular choice of hyperparameters.Source: Figure 6 · p. 12, B ANALYTIC FORM FOR KERNEL AND… · p. 13
  • For ReLU, the recurrence can be computed analytically, including weight and bias variance; Cho & Saul (2009) derived the result for polynomial rectified nonlinearities.Source: B ANALYTIC FORM FOR KERNEL AND… · p. 12

C BAYESIAN MARGINALIZATION OVER INTERMEDIATE LAYERS

The section derives the infinite-width deep-network–Gaussian-process equivalence by marginalizing over intermediate layers’ second-moment matrices. It shows that each layer is conditionally a Gaussian process and that the second moments become deterministic as width grows.

C BAYESIAN MARGINALIZATION OVER INTERMEDIATE LAYERS

  • With independent zero-mean Gaussian weights and biases, each layer’s pre-activations are exactly Gaussian-process distributed conditioned on its second-moment matrix, even at finite width.Each second-moment matrix depends on the preceding layer’s activations.Source: C BAYESIAN MARGINALIZATION OVER… · p. 13
  • As layer width grows, the Central Limit Theorem makes the second-moment fluctuations shrink as 1/Nl, and the infinite-width limit is a delta function.The passage also states that the dependence of K0 on x can be expressed as a delta function.Source: C BAYESIAN MARGINALIZATION OVER… · pp. 13–14
  • Marginalizing over intermediate second-moment matrices yields a Gaussian-process distribution for zL|x in the infinite-width limit.The output distribution is written as an integral over the intervening matrices, with their dependencies giving a directed decomposition.Source: C BAYESIAN MARGINALIZATION OVER… · pp. 13–14

D DETAILS OF THE EXPERIMENTS

The experiments use validation sets for hyperparameter selection and report test-set performance, with separate dataset splits for MNIST and CIFAR-10. The section also specifies neural-network and GP tuning procedures and NNGP implementation details.

  • MNIST uses a 50k/10k/10k training/validation/test split, while CIFAR-10 uses 45k/5k/10k; test performance is reported after validation-based hyperparameter selection.Source: D DETAILS OF THE EXPERIMENTS · p. 14
  • Neural-network hyperparameters were optimized with about 250 random-search trials per combination of training-set size, depth, width, and nonlinearity, sampling learning rate, weight decay, σw, σb, and mini-batch size.The ranges were 10−4 to 0.2 for learning rate and 10−8 to 1.0 for weight decay, both log-scale; σw was 0.01–2.5, σb was 0–1.5, and batch sizes were 16, 32, 64, 128, or 256.Source: D DETAILS OF THE EXPERIMENTS · p. 14
  • For each GP depth and nonlinearity, 900 grid evaluations covered σ2 w from 0.1 to 5.0 and σ2 b from 0 to 2.0 to generate the heatmap.Source: D DETAILS OF THE EXPERIMENTS · p. 14
  • NNGP experiments used precomputed lookup tables with ng = 501, nv = 501, nc = 500, and smax = 100; target noise σ2 ϵ defaulted to 10−10 and increased tenfold if Cholesky decomposition failed.For full MNIST, constructing KDD per layer took 90–140 seconds on 64 CPUs, and solving for 1,000 test points took 180–220 seconds; grid generation took 440–460 seconds on 6 CPUs and was amortized across experiments.Source: D DETAILS OF THE EXPERIMENTS · p. 14

E FURTHER RESULTS

Further results examine the relationship between GP uncertainty and target MSE for smaller training sets, NNGP accuracy across depth and variance settings, and the location of the best-performing hyperparameters relative to the critical line.

E FURTHER RESULTS

  • For smaller training sets, Figure 8 shows the relationship between target MSE and the GP’s uncertainty estimate.The passage does not specify the direction or strength of the relationship.Source: E FURTHER RESULTS · p. 14, Figure 8 · p. 15
  • Figure 9 presents NNGP test-accuracy heatmaps across variance settings, with rows for Tanh and ReLU and columns for depth.The accompanying text also refers to bias-variance values varying with depth.Source: E FURTHER RESULTS · pp. 14–15, Figure 9 · p. 15
  • The best-performing NNGP hyperparameters are distributed near the critical line where the phase changes.Figure 10 shows weight and bias variance distributions for the 25 best-performing runs at the given training-set size.Source: E FURTHER RESULTS · p. 14, Figure 10 · p. 16
  • Table 2 completes Table 1, reporting validation-selected NNGP settings over depth and weight and bias variances, and traditional NN settings over depth, width, and optimization hyperparameters.The reported settings correspond to the best-performing models for each training-set size.Source: Table 2 · p. 17

Related papers

Loading 1711.00165v3…