Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes

Roman Novak; Lechao Xiao; Jaehoon Lee; Yasaman Bahri; Greg Yang; Jiri Hron; Daniel A. Abolafia; Jeffrey Pennington; Jascha Sohl-Dickstein

Source-linked AI summary

Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes

Roman Novak, Lechao Xiao, Jaehoon Lee, Yasaman Bahri, Greg Yang, Jiri Hron, Daniel A. Abolafia, Jeffrey Pennington, Jascha Sohl-Dickstein

arXiv:1810.05148v4Oct 11, 2018stat.MLcs.AIcs.LGcs.NE

TL;DR

The paper addresses the limited understanding of CNNs through the previously established neural-network/Gaussian-process equivalence for fully connected networks. It derives the corresponding infinite-channel Bayesian CNN-GPs, including pooling, and introduces Monte Carlo kernel estimation for otherwise infeasible cases. The resulting CNN-GPs achieve state-of-the-art performance among GPs without trainable kernels, while carefully tuned finite SGD-trained CNNs can outperform them, and equivariance disappears from the infinite-channel Bayesian treatment without pooling.

Problem
Previous NN-GP equivalence results covered fully connected networks, leaving the correspondence and analytical use of NN-GPs for CNNs insufficiently developed.
Method
The paper analytically derives NN-GPs corresponding to infinitely wide Bayesian CNNs with and without pooling, and uses Monte Carlo sampling when kernel evaluation is computationally infeasible.
Results
CNN-GPs achieve state-of-the-art results for GPs without trainable kernels on CIFAR10, while carefully tuned finite SGD-trained CNNs can significantly outperform corresponding NN-GPs.
Takeaways & Limitations
Without pooling, CNN-GPs are identical to LCN-GPs, so translation equivariance plays no role in the infinite-channel Bayesian limit even though it benefits finite SGD-trained CNNs.
Takeaways & Limitations
The paper does not evaluate finite-width Bayesian networks, and a large-scale comparison between CNNs and CNN-GPs with pooling was computationally infeasible.

Abstract

from arXiv · show

There is a previously identified equivalence between wide fully connected neural networks (FCNs) and Gaussian processes (GPs). This equivalence enables, for instance, test set predictions that would have resulted from a fully Bayesian, infinitely wide trained FCN to be computed without ever instantiating the FCN, but by instead evaluating the corresponding GP. In this work, we derive an analogous equivalence for multi-layer convolutional neural networks (CNNs) both with and without pooling layers, and achieve state of the art results on CIFAR10 for GPs without trainable kernels. We also introduce a Monte Carlo method to estimate the GP corresponding to a given neural network architecture, even in cases where the analytic form has too many terms to be computationally feasible. Surprisingly, in the absence of pooling layers, the GPs corresponding to CNNs with and without weight sharing are identical. As a consequence, translation equivariance, beneficial in finite channel CNNs trained with stochastic gradient descent (SGD), is guaranteed to play no role in the Bayesian treatment of the infinite channel limit - a qualitative difference between the two regimes that is not present in the FCN case. We confirm experimentally, that while in some scenarios the performance of SGD-trained finite CNNs approaches that of the corresponding GPs as the channel count increases, with careful tuning SGD-trained CNNs can significantly outperform their corresponding GPs, suggesting advantages from SGD training compared to fully Bayesian parameter estimation.

1 INTRODUCTION

This work extends the neural-network/Gaussian-process equivalence from fully connected networks to deep CNNs, including architectures with pooling, and uses the correspondence to analyze topology and training. It shows that infinite-channel Bayesian CNNs can differ qualitatively from finite CNNs trained with SGD, especially regarding equivariance and performance.

Motivation: Previous NN-GP work established the correspondence for fully connected networks but did not extend it to CNNs or use it to analyze equivalent neural networks.The paper positions its contribution as both an architectural extension and an analysis of the induced function prior.
Contributions: The paper proves that fully Bayesian CNNs with infinitely many channels correspond to NN-GPs with and without pooling, arbitrary striding, and same or valid padding.It also proves convergence as the number of channels in hidden layers approaches infinity simultaneously.
Bayesian CNNs: Without pooling, CNN-GPs and LCN-GPs are identical despite CNNs having weight sharing and translation equivariance.The LCN has the same local connectivity pattern as a CNN but lacks weight sharing and translation equivariance.
Empirical findings: On CIFAR10, CNN-GPs achieve state-of-the-art results among GPs without trainable kernels, while carefully tuned finite SGD-trained CNNs can significantly outperform their corresponding NN-GPs.In some settings, finite CNN performance approaches the corresponding GP as channel count increases.
Methods: The paper introduces Monte Carlo estimation of NN-GP kernels when analytic evaluation is computationally infeasible, including CNNs with pooling.This complements the closed-form covariance calculations for architectures where the analytic kernel is tractable.
Scope: The study does not evaluate finite-width Bayesian networks or make claims about their performance relative to infinite-width GPs or finite SGD-trained networks.Performance of CNN-GPs with pooling on the full dataset was initially constrained by computational limitations.

2 MANY-CHANNEL BAYESIAN CNNS ARE GAUSSIAN PROCESSES

The paper proves that deep CNNs with infinitely many hidden-layer channels converge to Gaussian processes, with covariance obtained through deterministic layerwise transformations. The result covers pooling, striding, and padding variants under stated assumptions on the nonlinearity.

Network setup: The analysis considers deep convolutional networks with circular padding, Gaussian filters and biases, and hidden-layer channel counts tending to infinity.The main presentation uses one-dimensional convolutions without pooling, while the formalism extends to higher dimensions, striding, valid padding, and intermediate pooling.
Layerwise correspondence: Each convolutional layer is conditionally Gaussian given the previous layer’s activation covariance, because its pre-activations are linear transformations of Gaussian parameters.The covariance is represented by an affine convolutional operator applied to the previous activation covariance.
Infinite-channel limit: As channel counts increase, activation covariances become deterministic through a law-of-large-numbers argument, while pre-activations remain stochastic.The limiting covariance is obtained by repeatedly applying the covariance map C followed by the convolutional operator A.
Convergence result: The top-layer pre-activations converge in distribution to a multivariate normal under absolute continuity and exponential derivative-growth conditions on the nonlinearity.Only hidden-layer channels are taken to infinity; the top-layer output channel count remains fixed.

3 TRANSFORMING A GP OVER SPATIAL LOCATIONS INTO A GP OVER CLASSES

The paper converts the spatially indexed Gaussian process produced by a convolutional network into a class-indexed GP through vectorization or deterministic projection. These readouts preserve GP equivalence, while pooling-based projection can improve performance by incorporating spatial covariances and translation invariance.

Class-indexed GP: In the infinite-channel limit, CNN outputs converge to jointly Gaussian class predictions, with covariance obtained by transforming the spatial covariance of the final convolutional layer.The resulting GP is indexed by input samples and target classes.
Vectorization: Vectorization flattens the final convolutional feature map before a fully connected readout, but ignores pixel-pixel covariances and corresponds to a locally connected network GP.This strategy can be computed using only matching spatial covariance terms.
Projection: Projection with a deterministic vector collapses spatial dimensions while preserving the GP equivalence and can use global average pooling to incorporate all pixel-pixel covariances.Global average pooling makes the kernel translation invariant, though the approach has higher computational demands.
Empirical comparison: MC-CNN-GP with pooling consistently outperforms the other dimensionality-collapsing strategies on the reported CIFAR10 validation subset.The caption attributes this performance to translation invariance of the kernel.
Empirical comparison: CNN-GPs generally outperform FCN-GPs, while shallow CNN-GPs can miss nonlinear interactions between spatially distant pixels.Zero padding increasingly outperforms no padding with depth; at depth 15, no-padding CNN-GP becomes equivalent to center-pixel selection.

4 MONTE CARLO EVALUATION OF INTRACTABLE GP KERNELS

The paper introduces Monte Carlo estimates for NN-GP kernels that are analytically impractical, including CNNs with pooling. Increasing effective sample size improves kernel and validation-accuracy convergence, while finite-width estimates retain bias and variance trade-offs.

Monte Carlo kernel estimation: Monte Carlo-GP kernels estimate analytically impractical or unknown NN-GP kernels by averaging uncentered activation covariances from random finite-width networks.The method instantiates M networks with n hidden channels and uses their empirical activation covariances as the kernel estimate.
Convergence: Validation accuracy increases with n × M, while distance to the exact kernel decreases as Monte Carlo estimates converge.Here n × M denotes channel count multiplied by the number of samples; the dark accuracy band reflects ill-conditioning of KL+1.
Convergence: The Monte Carlo kernel converges to the analytic kernel as width n approaches infinity for the nonlinearities studied.The stated conditions include ReLU and erf nonlinearities.
Estimator trade-offs: Mn acts as the effective sample size, while increasing M and reducing n can lower memory cost at the expense of compute time and bias.
Monte Carlo kernel estimation: CNN-GP evaluation with pooling becomes practical through the Monte Carlo method.

5 DISCUSSION

Without pooling, Bayesian CNNs with infinitely many channels have the same NN-GP as locally connected networks, so weight sharing and equivariance do not alter the GP. Finite CNNs trained with SGD can nevertheless outperform their corresponding CNN-GPs under suitable tuning.

Bayesian CNNs and LCNs: Without pooling, the NN-GPs of CNNs and locally connected networks are identical despite the latter lacking weight sharing and translation equivariance.The equivalence follows because these predictions depend on sample-sample rather than pixel-pixel covariances.
Pooling and invariance: Pooling is the only CNN-GP mechanism described here that leverages pixel-pixel covariances and enables translation-invariant predictions, modulo edge effects.
Finite-width SGD training: Finite-width SGD-trained CNNs often approach CNN-GP performance, suggesting equivariance may not benefit those networks in some settings without pooling.
Finite-width SGD training: The best finite CNN significantly outperforms the best CNN-GP, especially for ReLU networks trained with a large learning rate.The paper identifies this gap as specific to CNNs rather than FCNs or LCNs.
Open questions: The paper conjectures that equivariance contributes to well-tuned finite-channel SGD CNN performance, but leaves stochastic optimization, finite width, and weight-sharing effects to future work.

6 CONCLUSION

The paper derives infinitely wide Bayesian CNNs as Gaussian processes, computes their covariances analytically or by Monte Carlo, and evaluates their performance against finite SGD-trained networks. CNN-GPs achieve state-of-the-art results among CIFAR10 GPs without trainable kernels, while carefully tuned finite SGD-trained CNNs can perform better.

Contributions: Infinitely wide Bayesian multi-layer CNNs correspond to Gaussian processes whose covariance is computable in closed form or by Monte Carlo sampling.The Monte Carlo method addresses architectures, such as CNNs with pooling, whose analytic evaluation is computationally infeasible.
Results: CNN-GP achieves state-of-the-art results on CIFAR10 among GPs without trainable kernels.It can also perform competitively with equivalent-architecture CNNs that fit the training set.
Results: Carefully tuned finite SGD-trained CNNs can significantly outperform their corresponding infinite Bayesian CNN-GPs, at least without pooling.The comparison concerns models with matching architectural hyperparameters and highlights a performance difference between finite SGD training and Bayesian inference.
Limitations: Large-scale CNN-versus-CNN-GP comparisons with pooling were not computationally feasible, leaving their relative performance an open question.Only one CNN-GP model with pooling was evaluated in that comparison.
Results: Underfitting and training choices affect generalization: allowing SGD-trained CNNs to underfit can improve performance, while small-learning-rate networks constrained to 100% training accuracy can be outperformed by CNN-GPs.The reported correspondence between finite CNNs and CNN-GPs is conditioned on the neural network fitting the training set to 100%.

Appendices

The appendices describe notation, convergence behavior, signal-propagation connections, pooling and striding extensions, Gaussian-process inference, and alternative infinite-channel limits. They also identify convergence and deep-network pathologies that constrain practical inference.

Monte Carlo convergence: MC-GP validation accuracy increases with n × M, while distance to the exact or best-available GP kernel decreases.Here n × M denotes width times number of Monte Carlo samples; shared weights converge more slowly because they use fewer independent random parameters.
Large-depth behavior: As depth increases, good CNN-GP and FCN-GP performance concentrates near the critical line separating ordered and chaotic regimes.The reported plots vary weight and bias variances and use the erf nonlinearity.
Large-depth behavior: Deep GP kernels can become pathological when covariance iterates converge to a fixed point, because outputs become asymptotically identically correlated and lose information about training inputs.This makes learning and inference infeasible in both chaotic and ordered regimes.
Large-depth behavior: Choosing hyperparameters on the critical line slows convergence to the fixed point from exponential to polynomial rates, enabling very deep networks and deep NN-GP inference.The critical line separates two untrainable phases in the described signal-propagation analysis.
Architecture extensions: Average pooling and strided convolution are represented by linear operators applied to Gaussian pre-activations before the pointwise nonlinearity.Strided convolution corresponds to subsampling, while average pooling uses uniform weights over each pooling window.
GP inference: GP posterior prediction for regression is analytically tractable, with the main computational cost typically coming from inversion of the training covariance matrix.Approximate GP inference methods can be used when matrices are too large.
Infinite-channel limits: The sequential infinite-channel limit provides intuition for constructing the GP but does not guarantee convergence of large finite Bayesian CNNs.The simultaneous limit supplies the rigorous convergence proof, whereas the sequential argument relies on successive convergence in distribution.
Infinite-channel limits: The sequential and simultaneous approaches yield identical Gaussian processes despite differing proof strategies.The appendix notes that the simultaneous proof is more involved and makes stronger assumptions.

F.5 PROOF OF THEOREM F.6

The proof establishes Gaussian convergence for dense outputs and extends the analysis to global average pooling, subsampling, and vectorization followed by a dense layer.

Proof strategy: The exponent is a linear combination of independent Gaussian random variables, so integrating their characteristic function yields the required convergence expression.The argument uses Gaussian characteristic functions together with continuity near the limiting covariance.
Global average pooling / subsampling: The same characteristic-function framework handles a deterministic linear functional applied channel-wise after the final convolutional layer.This covers global average pooling and subsampling as stated in the proof remark.
Global average pooling / subsampling: After global average pooling or subsampling, the resulting outputs are independently Gaussian under the stated construction.The proof identifies this as the consequence of applying the linear functional before the final dense layer.
Vectorization and a dense layer: For vectorization followed by a dense layer, the absence of weight sharing makes the pixel index range over all d positions rather than the filter support.The dense weights connect each output to each channel-pixel pair.
Vectorization and a dense layer: The characteristic-function argument shows the dense-layer outputs converge weakly to a mean-zero Gaussian with covariance involving the mean trace operator.The mean trace computes the average of diagonal entries of the pixel-by-pixel covariance matrix.

F.6 PROOF OF THEOREM F.5

The proof uses induction and concentration to establish convergence of covariance operators while preserving positive semidefiniteness, and identifies extensions involving linear operators, pooling, normalization, and non-Gaussian priors.

Inductive convergence: The induction proves convergence of random covariance matrices to their infinite-width limits in the sup norm, with finite dimensionality yielding convergence in other norms.The norm is characterized by the maximum absolute covariance entry over inputs and pixel indices.
Inductive convergence: Positive semidefiniteness is preserved by the covariance operator, and the proof controls neighborhoods of the limiting covariance within the positive-semidefinite cone.The argument uses closure of the cone and bounds the operator near the limiting kernel.
Extensions: The framework suggests extensions to max-pooling, dropout, normalization, and sub-Gaussian weight and bias distributions, but these are listed as directions rather than established results.The proof remark explicitly presents these as ways to strengthen or extend the theorem.
Extensions: A deterministic channel-wise linear operator can be inserted after convolution, with the convergence proof carrying over after replacing the covariance operator by its transformed version.The outputs remain independently Gaussian after applying the operator.

F.7 PROOF OF THEOREM F.4

This proof establishes moment bounds and convergence for nonlinear covariance terms under exponentially bounded activations, while noting that the basic finite-sample rate can be weak in practice.

Moment bounds: Exponentially bounded nonlinearities have finite Gaussian moments C_k(φ, R), providing the integrability needed by the convergence proof.The proof bounds C_k using the supremum of Gaussian expectations over variances between 1/R and R.
Activation conditions: Absolute continuity with an exponentially bounded derivative implies that the activation itself is exponentially bounded and belongs to the required function class.The argument also controls polynomial factors involving Gaussian gradients.
Continuity bounds: The proof applies the Mean Value Theorem and norm bounds to control changes in the nonlinear covariance functional.Exponential bounds on the activation and its gradient provide the needed domination.
Finite-sample convergence: Chebyshev’s inequality yields a 1/n decay bound for estimating nonlinear covariance terms from finite samples.The variance is bounded by the second moment of the product of activation values.
Finite-sample convergence: In practice, the 1/n Chebyshev rate is often too weak, although linearly bounded activations permit an exponential decay bound via concentration inequalities.The stronger result is supplied by a subsequent lemma under the linear-growth condition.

G EXPERIMENTAL SETUP

The experiments compare CNN and locally connected architectures across pooling choices and channel counts under controlled CIFAR10 preprocessing and tuned optimization settings, including Monte Carlo GP estimates for pooled CNNs.

Data and architecture: The study uses 3 × 3 stride-1 filters without dilation, normalized inputs, regression targets, and no data augmentation.Inputs lie on a sphere after zero-mean, unit-variance normalization, while labels use centered class vectors.
Optimization: Neural networks are trained with Adam and mean squared error, with learning-rate reductions triggered by insufficient training-loss improvement.Training may stop after prolonged lack of progress.
Data and architecture: CIFAR10 experiments use 500 training examples and 4000 validation examples, with images bilinearly downsampled to 8 × 8 pixels.All models have three hidden layers, erf nonlinearities, and no valid padding.
Model selection: The hyperparameter search varies architecture, pooling, channel count from 2^0 through 2^12, learning rate, weight decay, and batch size.For each architecture-pooling-channel configuration, the lowest-validation-loss model among training-accurate candidates is selected.
GP evaluation: For pooled CNNs, the GP is estimated by Monte Carlo using n = 2^12 filters and M = 2^6 samples, while GP inference uses diagonal regularization starting at 10^-10.Locally connected models with pooling use an appropriately rescaled CNN-GP kernel.

G.2 MONTE CARLO EVALUATION OF INTRACTABLE GP KERNELS

The experiments estimate otherwise intractable GP kernels using Monte Carlo network samples and evaluate their classification performance under varied configurations and numerical regularization.

Monte Carlo GPs vary the channel count n and network-instantiation count M over powers of two from 2^0 through 2^10.
The CNN-GP pooling kernel uses K_210,210 instead of the exact K_∞ because computing the latter is impractical.
Inputs are rescaled to 31 × 31 so that depth 15 collapses the unpadded spatial dimension to a 1 × 1 patch.
For pooled MC-CNN-GPs, samples use n = 16 filters, depths up to 31, and M selected independently by validation accuracy.
Experiments use CIFAR10 subsets of 500 training and 1000 validation examples, with 64-bit computation and random-chance accuracy reported after numerical failure.

G.5 CNN-GP ON FULL DATASETS

The full-dataset experiments compare CNNs and FCNs across architectures, nonlinearities, depths, widths, pooling choices, and optimization settings on MNIST, Fashion-MNIST, and CIFAR10.

Experiments use full MNIST and Fashion-MNIST splits of 50000/10000/10000 and CIFAR10 splits of 45000/5000/10000 for training, validation, and testing.
The configuration grid varies CNN versus FCN architecture, erf versus ReLU nonlinearity, depth, and weight and bias variances.
Neural networks are trained for 2^18 updates with batch size 128, while learning rates and weight decay are tuned using validation accuracy.
CNNs compare no pooling with a single global average pooling layer before the final fully connected layer.
Widths or channel counts are powers of two, ranging from 2^1 to 2^9 and reaching 2^10 for pooled CNNs in Figure 6(a).
Certain large configurations could not be evaluated because of time and compute limitations, although the authors believe this did not qualitatively affect the results.