PaperlayerView exact revision on arXiv

Source-linked AI summary

Gaussian Process Behaviour in Wide Deep Neural Networks

Alexander G. de G. Matthews, Mark Rowland, Jiri Hron, Richard E. Turner, Zoubin Ghahramani

arXiv:1804.11271v2stat.MLcs.LG

TL;DR

The paper investigates the theoretical relationship between wide deep neural networks and Gaussian processes, motivated by the need to understand neural-network priors and random-function behaviour. It proves Gaussian-process convergence under broad conditions, evaluates convergence empirically, and finds close Bayesian-neural-network agreement with Gaussian-process analogues in several cases, while noting computational and scope limitations.

  • Problem

    Understanding neural-network-induced function distributions is essential for studying initialization, learning dynamics, expressivity, and Bayesian priors.

  • Method

    The paper extends convergence results to deep networks, defines Gaussian-process limits through finite-index-set convergence, and evaluates convergence using maximum mean discrepancy.

  • Results

    Under broad conditions, increasingly wide deep networks converge in distribution to Gaussian processes, while Bayesian neural networks closely agree with Gaussian processes on five of six datasets and in several other cases.

  • Takeaways & Limitations

    Bayesian-neural-network studies should routinely compare against Gaussian-process analogues, and Gaussian-process inference is an alternative when Gaussian-process behaviour is desired.

  • Takeaways & Limitations

    The computationally expensive inference restricts experiments to relatively small, low-dimensional datasets and limits finite networks to 50 hidden units; continuity, boundedness, and differentiability are also outside the theory's scope.

Abstract

from arXiv · show

Whilst deep neural networks have shown great empirical success, there is still much work to be done to understand their theoretical properties. In this paper, we study the relationship between random, wide, fully connected, feedforward networks with more than one hidden layer and Gaussian processes with a recursive kernel definition. We show that, under broad conditions, as we make the architecture increasingly wide, the implied random function converges in distribution to a Gaussian process, formalising and extending existing results by Neal (1996) to deep networks. To evaluate convergence rates empirically, we use maximum mean discrepancy. We then compare finite Bayesian deep networks from the literature to Gaussian processes in terms of the key predictive quantities of interest, finding that in some cases the agreement can be very close. We discuss the desirability of Gaussian process behaviour and review non-Gaussian alternative models from the literature.

1. Introduction

The paper studies random wide deep networks to clarify their induced function distributions and relationship to Gaussian processes. It extends convergence theory to deep networks, evaluates finite-network agreement empirically, and discusses when Gaussian-process behaviour is useful or undesirable.

  • Random neural networks induce distributions over input–output functions, informing studies of initialization, learning dynamics, expressivity, and Bayesian priors.
  • As networks become large, limiting distributions can provide insight into otherwise difficult-to-analyze deep-network distributions, while finite networks may approximate these limits closely.
  • The paper rigorously extends Neal’s one-hidden-layer result by proving convergence of increasingly wide fully connected networks with any fixed number of hidden layers to Gaussian processes.The result assumes increasing hidden-layer widths, appropriately scaled independent Gaussian weights, and nonlinearities satisfying a linear-envelope condition.
  • Maximum mean discrepancy is used to study finite-network distance from Gaussian-process analogues, alongside gold-standard MCMC comparisons of Bayesian neural networks with exact Gaussian-process inference.
  • Gaussian-process behaviour may be desirable when it supports analytic inference, but undesirable when it implies a lack of hierarchical representation and Gaussian statistical assumptions.

2. The deep wide limit

The paper derives Gaussian-process limits for deep fully connected networks as widths grow, using recursive covariance structure and convergence of finite-dimensional distributions. The result requires continuous nonlinearities with a linear envelope and countable input sets.

  • Network setup: The network recursion defines successive activations from weighted previous-layer outputs, biases, and a nonlinearity, with the final layer producing the network output.The output is f(D+1)(x), and φ denotes the nonlinearity.
  • Network setup: Independent normal weights and biases induce distributions over activations, while width-dependent variance scaling prevents activity variances from diverging.The scaling applies to weights in layers µ ≥ 2 as widths increase.
  • Convergence argument: For fixed inputs, arranging activations into vectors exposes independent identically distributed terms, allowing the multivariate central limit theorem to yield Gaussian limits as widths grow.The argument begins with pairs of inputs and all output functions, then extends to larger finite input sets.
  • Convergence argument: Consistent Gaussian limits for every finite input set define a Gaussian process through the Kolmogorov extension theorem.For countable index sets, weak convergence of finite-dimensional marginals is sufficient under the stated metric and topology.
  • Scope: The theory excludes phenomena depending on uncountably many inputs, including continuity, boundedness, and differentiability.The paper identifies extensions using stronger measure and topology results as future work.
  • Main theorem: Theorem 4 proves convergence for strictly increasing width functions, continuous nonlinearities satisfying the linear envelope condition, and any countable input set; equal-width layers follow as a corollary.The limiting process has zero mean and covariance given by the recursion lemma.

3. Specific kernels under recursion

The paper relates recursively defined kernels to deep wide Gaussian processes and uses analytic recursion formulas for rectified linear units in its experiments.

  • Recursive kernels: Cho and Saul’s recursive kernels were designed to mimic computation in large multilayer neural networks and provide closed-form solutions for the recursion in a special case.The paper distinguishes this analytic result from the broader deep Gaussian-process convergence theory.
  • Recursive kernels: Because r = 1 corresponds to rectified linear units, the paper applies the analytic kernel recursion in all experiments.The derivation uses Gaussian equivalence and can optionally incorporate bias by augmenting the input.

4. Measuring convergence using maximum mean discrepancy

The experiment measures how closely finite random networks approximate their Gaussian-process analogues using maximum mean discrepancy across depths, widths, and width-growth schemes.

  • MMD setup: Maximum mean discrepancy measures the largest expectation difference between two distributions over functions constrained by unit Hilbert-space norm.The study uses the unbiased estimator of squared MMD.
  • Experimental design: The experiment compares 2000 random networks with 2000 samples from the corresponding Gaussian-process marginal using 10 standard-normal inputs in four dimensions.The setup varies hidden-layer counts, width functions, and network sequence index.
  • Experimental design: Three strictly increasing width functions satisfy the convergence assumptions, including identity widths and schemes where the largest layer is first or last.For the latter schemes, n is identified with the narrowest hidden-layer width.
  • Results: For each fixed depth, MMD decreases as width increases, while equal per-layer widths show higher distances and slower convergence at greater depth.These observations match the predictions of Theorem 4.
  • Results: The largest-first and largest-last width schemes converge faster than identity widths because their networks contain strictly more units.The comparison between the two larger schemes is made at similar unit counts.

5. Empirical Comparison of Bayesian Deep Networks to Gaussian Processes

The paper compares finite Bayesian deep networks with analogous Gaussian processes using exact GP inference and computationally intensive MCMC, finding close agreement in many settings but an important dataset-dependent exception.

  • Experimental setup: Exact Gaussian-process inference is compared with ‘gold standard’ MCMC inference for finite Bayesian networks using ReLU units and variance corrections.The networks use three hidden layers and 50 units per layer in the reported experiments.
  • Fixed hyperparameters: The log-marginal likelihood estimates for the small XOR problem agree to within a single nat between annealed importance sampling and Gaussian-process computation.The difference is described as negligible from a model-selection perspective.
  • Fixed hyperparameters: The predictive log-likelihood correspondence is close but not exact, while Bayesian-network marginal function values are slightly more concentrated than Gaussian-process values.These comparisons use randomly generated four-dimensional regression data and include predictive latent-function marginals.
  • Learnt hyperparameters: The Bayesian-network and Gaussian-process posteriors agree very closely on the 100-point Snelson dataset.The comparison uses hyperparameters learned from the Gaussian-process kernel.
  • Summary and discussion: Across six datasets, the Bayesian network agrees closely with the Gaussian process on five, with the yacht dataset as the exception.The yacht comparison is especially relevant because the dataset has the highest dimensionality among those considered.
  • Summary and discussion: The comparison is computationally expensive, restricting experiments to relatively small, low-dimensional datasets and limiting the finite network sizes considered.The authors therefore recommend routinely comparing Bayesian deep-learning results with Gaussian processes using the studied kernels.

6. Proof of the main theorem

The proof reduces Gaussian-process convergence to finite-dimensional linear projections, then establishes their convergence using exchangeable-array central limit theorems and layerwise moment conditions. Consistency of these projections, followed by adding Gaussian biases, yields convergence of the full network function distribution to the stated Gaussian process.

  • Proof strategy: The proof represents each projection as a sum of exchangeable summands for fixed network width.The network is extended to infinite width with finite fan-out, and the weights are reparameterized using standard normals without changing final function distributions.
  • Proof strategy: Convergence of random functions is reduced to convergence of finite-dimensional marginals and all finite linear projections.The Cramér–Wold device turns the problem into convergence of real-valued projected variables.
  • Exchangeable CLT: De Finetti’s theorem supplies conditional independence, enabling central limit theorems for exchangeable sequences and arrays.The adapted central limit theorem requires mean-zero variables, finite variance and third moment, a limiting variance, and additional dependence conditions.
  • Exchangeable CLT: The proof establishes the required limiting variance and applies the adapted CLT to show each projection converges to N(0, σ2(µ, L, α)[∗]).The limiting variance is connected to the Gram matrix implied by the recursive kernel.
  • Conclusion of proof: Consistency of finite linear projections is sufficient for Gaussian-process convergence, and normally distributed biases are then added to obtain the final result.The argument concludes after combining projection convergence with the finite-dimensional convergence criterion.

7. Desirability of Gaussian process behaviour and methods to avoid it

The paper examines when Gaussian-process behaviour is desirable and what it omits relative to deep Bayesian neural networks. It notes that avoiding Gaussian limits requires violating central-limit-theorem conditions, while learned representations remain a key distinction.

  • Desirability: Gaussian processes with fixed kernels do not learn hierarchical representations or hidden features.The paper contrasts this flat representation with the hierarchical representations regarded as important to deep learning.
  • Desirability: Learning a few Gaussian-process hyperparameters is presented as unlikely to provide the benefits of full representation learning.Using substantially more hyperparameters would move beyond the paper’s scope.
  • Related alternatives: Representation-learning models composed with Gaussian processes differ from this paper’s paradigm because the paper assumes probabilistic model complexity without problem-specific structure.The contrasted alternatives include structured networks composed with Gaussian processes.
  • Methods to avoid Gaussian behaviour: Avoiding marginal Gaussian-process behaviour requires making at least one central-limit-theorem condition substantially fail.The paper identifies independence, bounded variance and many summands as chief conditions whose violation can prevent Gaussian behaviour.

8. Conclusions

The paper extends Gaussian-process convergence results from shallow to deep fully connected networks and finds close agreement between several finite Bayesian networks and their Gaussian-process analogues. It recommends routine GP comparisons and considering GP inference when that behaviour is desired.

  • Contributions: Under broad conditions, increasingly wide deep feedforward networks converge in distribution to Gaussian processes.The result formalizes and extends Neal’s earlier shallow-network result to networks with more than one hidden layer.
  • Empirical findings: Maximum mean discrepancy suggests Gaussian-process behaviour across models with sizes comparable to networks used in the literature.The empirical study uses MMD to assess distance between finite networks and their Gaussian-process analogues.
  • Empirical findings: Several finite Bayesian neural-network experiments show close agreement with corresponding Gaussian-process models, suggesting similar results in parts of the Bayesian deep-learning literature.The conclusion is stated as a likelihood rather than a universal equivalence.
  • Implications: The paper recommends routinely comparing Bayesian neural networks with Gaussian-process analogues.When GP behaviour is desired, exact or approximate GP inference is proposed as an alternative to neural-network inference.

Appendix A. Adapting the exchangeable CLT of Blum et al. 1958.

The appendix adapts an exchangeable central limit theorem to the proof’s variance-scaled triangular-array setting. It separates the zero-variance case from the positive-variance case and transfers convergence through asymptotically negligible differences.

  • CLT adaptation: The appendix develops an intermediate lemma to prove the adapted central limit theorem for sequences of exchangeable sequences.The proof embeds a non-triangular array into a larger triangular array by filling extra positions with standard normal variables.
  • Zero-variance case: When the limiting variance is zero, the normalized sum converges in probability to zero, equivalently converging in distribution to a degenerate normal.The argument uses uncorrelated distinct elements and Chebyshev’s inequality.
  • Positive-variance case: When the limiting variance is nonzero, standardization produces unit-variance variables satisfying the original exchangeable-CLT conditions.The proof verifies the mean, variance, moment, and dependence conditions term by term.
  • Transfer to target sequence: The standardized sum converges to N(0, σ2(µ, L, α)[∗]), and the difference between the standardized and target sums converges surely to zero.A general convergence result then transfers convergence in distribution to the target sequence.

Appendix B. Details of the proof of Theorem 4

The proof of Theorem 4 proceeds inductively across hidden layers, establishing Gaussian convergence of finite projections and then extending it to the network’s finite-dimensional distributions.

  • Proof strategy: The proof inductively shows that hidden units in each layer converge in distribution under the metric ρ.The induction treats a fixed number of hidden layers and propagates convergence layer by layer.
  • Proof strategy: Proposition 14 derives Gaussian convergence for finite linear combinations of pre-nonlinearities at the next layer.It assumes convergence at the preceding layer and applies a central-limit argument to the next layer’s projections.
  • Finite-dimensional convergence: The Cramér–Wold device converts convergence of finite projections into convergence of finite-dimensional marginal distributions.Independent bias terms are then incorporated to obtain convergence for pre-activations.
  • Conclusion: The resulting random functions converge in distribution to a centred Gaussian process with covariance given by the recursion in Lemma 2.This is the stated limiting process as network width tends to infinity.

B.1 Proofs of main lemmas and corollaries

The main lemmas establish convergence of layerwise covariance quantities by combining Gaussian limits, continuous mappings, moment bounds, and uniform integrability.

  • Covariance limits: Lemma 17 propagates convergence of products of post-nonlinearities and identifies the limiting covariance recursion.The limiting covariance without the bias term is represented by K(L).
  • Covariance limits: Continuity of the activation and product maps transfers convergence from pre-nonlinearities to post-nonlinearities and their two-way products.The continuous mapping theorem supplies both convergence steps.
  • Expectation convergence: Uniform integrability ensures that convergence in distribution also yields convergence of the relevant expectations.Theorem 3.5 is applied after uniform integrability is established through Lemma 21.
  • Conclusion: These results yield convergence of the variance and higher-order terms needed to complete the recursive Gaussian-process covariance analysis.The limiting variance is determined by the behaviour of the transformed layer variables.
  • Moment bounds: Lemma 16 proves that the required fourth-moment quantities remain bounded independently of network width.The proof reduces the bound to fourth moments of earlier-layer activations and uses the linear envelope condition.

B.2 Proofs of auxiliary results

The auxiliary results provide moment and integrability controls that support the central-limit and expectation-convergence steps in the main proof.

  • Mixed-moment control: Lemma 18 bounds mixed moments of four random variables using their eighth moments and a correlation-based inequality.Hölder-type arguments control lower-order absolute moments.
  • Gaussian moment calculations: Lemma 19 evaluates moments of weighted sums of independent standard normal variables using Gaussianity and known standard-normal moments.The weighted sum is reduced to a scalar standard normal variable.
  • Moment bounds: Lemma 20 establishes eighth-moment bounds for layer activations that are uniform in width and neuron index.The base case follows from Gaussian first-layer distributions, while induction handles deeper layers.
  • Inductive application: The auxiliary bounds support the inductive proof by controlling expanded sums whose term counts grow with preceding-layer width.The network’s 1/h^4 scaling offsets the upper bound on the number of terms.
  • Uniform integrability: Lemma 21 uses bounded eighth moments to prove uniform integrability for the relevant four-way products.A uniformly bounded higher moment supplies the required integrability criterion.
Loading 1804.11271v2…