PaperlayerView exact revision on arXiv

Source-linked AI summary

A Gaussian Process perspective on Convolutional Neural Networks

Anastasia Borovykh

arXiv:1810.10798v2stat.MLcs.LG

TL;DR

The paper addresses limited understanding of when CNN outputs approach multivariate normal distributions, despite established GP convergence results for fully connected networks. It analyzes CNNs through a GP perspective using a one-layer Lyapunov-type bound and numerical discrepancy tests; those tests find relatively quick GP-like behavior in some settings, while rigorous convergence rates for dependent layer summands remain unresolved.

  • Problem

    Although fully connected networks' convergence to GPs is well studied, little is known about when convolutional network outputs approach a multivariate normal distribution.

    Source: §1 · p. 1, §1.1 · p. 2

  • Method

    The paper derives a one-layer CNN result using a Lyapunov-type bound and numerically measures discrepancy between CNN and GP outputs.

    Source: §1.1 · p. 2, §4 · p. 9

  • Results

    Numerical tests show CNN outputs approach GP behavior relatively quickly at small filter widths, with discrepancy depending on activation and input setting.

    Source: §1.1 · p. 2, §4 · pp. 9–10

  • Takeaways & Limitations

    The GP perspective offers insight into CNN output distributions and can support analytical posterior uncertainty estimates without computationally intensive MCMC methods.

    Source: §5 · p. 12

    A rigorous convergence rate for the dependent variables summed within each layer remains to be proved.

    Source: §5 · p. 12

Abstract

from arXiv · show

In this paper we cast the well-known convolutional neural network in a Gaussian process perspective. In this way we hope to gain additional insights into the performance of convolutional networks, in particular understand under what circumstances they tend to perform well and what assumptions are implicitly made in the network. While for fully-connected networks the properties of convergence to Gaussian processes have been studied extensively, little is known about situations in which the output from a convolutional network approaches a multivariate normal distribution.

1 Introduction

The introduction frames CNNs as successful across several tasks but notes that their inner workings remain uncertain, motivating a Bayesian study of their limiting behavior. It positions the paper as extending GP convergence analysis to convolutional networks, whose compositional structure differs from fully connected networks.

1 Introduction

  • CNNs have gained popularity through success in image and speech classification and forecasting, while many of their inner workings remain unknown.A Bayesian perspective induces a distribution on the output function, which can be studied through limiting behavior.Source: §1 · p. 1

1.1 Our contributions

  • A generalized CLT in the form of a Lyapunov-type bound yields a one-layer result for CNNs, whose summands are uncorrelated and not necessarily identically distributed.Numerical study finds that outputs approach a GP relatively quickly even at relatively small filter widths, although the CLT does not fully apply to deeper networks.Source: §1.1 · p. 2

1.2 Related work

  • Prior convolutional GP work used convolutional structure to construct kernels, whereas this paper studies whether CNNs approach GP behavior by construction.The related work also connects the resulting recursive kernel to deep and additive kernels.Source: §1.2 · p. 2
  • Earlier GP-convergence studies addressed one-hidden-layer and deep fully connected networks, including convergence of finite-width networks and output discrepancy.These studies provide context for extending the question to convolutional networks.Source: §1.2 · pp. 2–3

2 Background

The background defines Gaussian processes through finite-dimensional Gaussian distributions and describes their covariance-based prediction. It then recalls why wide fully connected networks approach GP priors, while finite-width dependencies can make deeper outputs deviate from Gaussian behavior.

2.1 Gaussian processes

  • A function is a Gaussian process when every finite set of its values has a multivariate Gaussian distribution, specified by its mean and covariance functions.The covariance function encodes similarity between inputs and defines entries of the covariance matrix.Source: §2.1 · p. 3
  • With a Gaussian GP prior and Gaussian likelihood, the posterior remains a GP and yields a predictive distribution.The background sets this in the context of training data and a Gaussian observation-noise model.Source: §2.1 · p. 3

2.2 Neural networks and Gaussian processes

  • In a fully connected network with Gaussian weight and bias initialization, each unit sums independent identically distributed terms, so the CLT gives Gaussian outputs as width M tends to infinity.Applying the multivariate CLT to function values at finitely many inputs gives a multivariate Gaussian limit.Source: §2.2 · pp. 3–4
  • At finite width, units may be uncorrelated without being independent, so dependencies introduced across layers can make outputs deviate from Gaussian behavior.The text notes that products of weights and previous-layer activations are not necessarily multivariate normal, so zero covariance does not establish independence.Source: §2.2 · p. 4

3 Convolutional neural networks and Gaussian processes

The paper studies when convolutional-network outputs approach Gaussian processes, using a central-limit argument and a recurrent covariance kernel to characterize their behavior. Its kernel averages correlations over receptive fields, producing an additive, depth-dependent structure and greater flattening than in fully connected networks.

3.1 Central limit theorem

  • Because convolutional summands are independent but not necessarily identically distributed, the ordinary multivariate central limit theorem used for fully connected networks does not directly apply.A Lyapunov-type bound that allows non-identically distributed summands is presented as suitable for this setting.Source: §3.1 · p. 6

3.2 Gaussian process convergence

  • As filter size tends to infinity, the first-layer output approaches a multivariate normal distribution; later-layer GP convergence is hypothesized and checked numerically.The later-layer argument assumes multivariate-normal previous-layer outputs, but products of weights and those outputs need not be multivariate normal.Source: §3.2 · pp. 6–7

3.3 Kernel specification

  • The convolutional kernel sums correlations across a receptive field, and its input dependence grows exponentially with depth.The filter size and layer count together determine how many input terms contribute to the kernel.Source: §3.3 · p. 7
  • If the CNN output prior is sufficiently close to a GP, its covariance has a naturally arising additive-kernel form whose base kernel depends on the activation function.The number of summed patches depends on filter size and depth.Source: §3.3 · p. 7
  • Averaging previous-layer kernels over receptive fields makes the convolutional kernel flatter, amplifying the flattening that also depends on depth in fully connected networks.The illustration uses a filter size of two and two distinct angles.Source: §3.3 · p. 8, Figure 2 · p. 8

4 Numerical results

The numerical experiments compare CNN output distributions with corresponding Gaussian processes across activations, input dependencies, and network depth. They find that discrepancy depends on activation and depth, and that posterior behavior is close when the priors correspond.

Numerical results

  • The numerical study focuses mainly on time-series forecasting, while the authors state that its conclusions generalize to other prediction tasks.Source: §4 · p. 9
  • The experiments estimate discrepancy with unbiased squared MMD, comparing outputs from 500 CNNs against 500 GP samples using the depth-corresponding kernel.The kernel is computed by the recurrence in (6), with a Monte Carlo evaluation of the expected value in (5) using 1000 simulations.Source: §4 · p. 9
  • For i.i.d. inputs, linear CNNs converge quickly but discrepancy increases with depth; ReLU convergence is slower in depth, while hyperbolic tangent discrepancy remains low across depths.The ReLU reaches GP behavior at small filter sizes, while sufficiently many layers and small filters can avoid that behavior.Source: §4 · p. 9, Figure 3 · p. 10
  • The analytical recurrent kernel assumes sufficiently normal layer outputs, a condition attained only in the limit, yet its output-prior discrepancy from Monte Carlo appears small.The authors therefore use the analytical expression to mimic the deep CNN's recurrent relation.Source: §4 · p. 9
  • For autoregressive inputs, linear and ReLU CNN priors show substantially greater GP discrepancy than bounded hyperbolic tangent priors, with divergence increasing across layers.The experiments use an AR(2) series with finite covariance and d = 50.Source: §4 · p. 10, Figure 4 · p. 11

Posterior discrepancy

  • When CNN and GP priors correspond, their posteriors are also close in the AR(3) forecasting example.The CNN comparison uses an ensemble of 100 randomly initialized networks trained by mean squared error, which the authors note is not fully valid for uncertainty estimation.Source: §4 · p. 11

5 Conclusion and discussion

The paper extends the Gaussian-process perspective to convolutional networks, deriving a one-layer CLT-like discrepancy bound for non-identically distributed inputs and examining deeper networks empirically. It concludes that sufficiently close priors support GP-based uncertainty estimates, while rigorous convergence rates for dependent layer summands remain open.

5 Conclusion and discussion

  • A Lyapunov-type bound yields a one-layer CLT-like result for CNN–GP discrepancy with non-identically distributed inputs.Source: §5 · p. 11
  • For deeper layers, dependent multivariate-normal layer outputs mean the general CLT for independent variables no longer applies.Source: §5 · p. 11
  • When CNN and GP priors are sufficiently close, GP analytical properties can provide CNN uncertainty estimates without computationally intensive options such as MCMC.Source: §5 · pp. 11–12
  • A rigorous convergence rate for dependent variables in each layer's summation remains an open research question.Source: §5 · p. 12
  • The authors describe this as, to their knowledge, the first study of CNNs from the GP perspective, using that framework to examine when outputs can be close to or deviate from Gaussian distributions.Source: §5 · p. 12
Loading 1810.10798v2…