PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Convolutional Networks as shallow Gaussian Processes

Adrià Garriga-Alonso, Carl Edward Rasmussen, Laurence Aitchison

arXiv:1808.05587v2stat.MLcs.LG

TL;DR

The paper tackles the computational difficulty of Bayesian inference in CNNs by deriving an exact GP equivalent for infinitely wide convolutional and residual networks. It develops an efficiently computable recursive kernel and evaluates it on MNIST, where the 32-layer ResNet GP achieves 0.84% classification error. The approach is scoped to infinite-filter limits and retains computational challenges for large datasets and certain architectural choices.

  • Problem

    Bayesian inference in CNNs is difficult because their large parameter counts require highly approximate variational inference or storage of many posterior samples.

  • Method

    The paper derives the GP representation and recursive kernel of CNNs and residual CNNs in the limit of infinitely many convolutional filters, exploiting diagonal covariance structure for efficient computation.

  • Results

    0.84% classification error was obtained on MNIST by the kernel equivalent to a 32-layer ResNet, without computing gradients or augmenting the training set.

  • Takeaways & Limitations

    The resulting GP outperforms previous GP approaches without parametric neural-network components on handwritten digit classification while using the structure of CNN architectures.

  • Takeaways & Limitations

    The approach assumes infinitely many filters, and removing pooling was necessary to preserve the efficiency gains because pooling requires off-diagonal filter covariances.

Abstract

from arXiv · show

We show that the output of a (residual) convolutional neural network (CNN) with an appropriate prior over the weights and biases is a Gaussian process (GP) in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a CNN, the equivalent kernel can be computed exactly and, unlike "deep kernels", has very few parameters: only the hyperparameters of the original CNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GPs with a comparable number of parameters.

1 INTRODUCTION

The paper addresses the difficulty of Bayesian inference in CNNs by deriving an exact Gaussian-process representation for infinitely wide convolutional and residual networks, with efficient kernel computation and strong MNIST performance.

  • CNNs provide powerful pattern recognition, but Bayesian inference is difficult because their large parameter counts require approximate variational methods or many posterior samples.GPs offer exact posterior uncertainty, motivating a combination of CNN pattern recognition with GP inference.
  • Infinitely wide CNNs and convolutional residual networks with suitable random weights and biases have equivalent GP representations.The result extends infinite-width GP equivalence from fully connected networks to architectures that exploit image structure.
  • The induced kernel can be computed efficiently because only patch variances are needed and all-location variances can be computed simultaneously as a convolution.Independent zero-mean downstream weights decorrelate contributions from different locations.
  • 0.84% error rate was obtained on MNIST without gradients or training-set augmentation, setting a new record for nonparametric GP-based methods.The result demonstrates the performance benefit of adding translation-invariant structure to the GP prior.

2 GP BEHAVIOUR IN A CNN

The CNN is formulated as a sequence of convolutional transformations with Gaussian priors, and its infinite-channel limit is shown to produce jointly Gaussian feature maps indexed by inputs.

  • 2.1 A 2D CONVOLUTIONAL NETWORK PRIOR: The construction applies to 2D convolutions and extends to nD, dilated, and upconvolutions because these can be represented as linear transformations with tied coefficients.
  • 2.1 A 2D CONVOLUTIONAL NETWORK PRIOR: The network represents an image as a matrix of flattened channels and applies convolutional linear transformations recursively across hidden layers.The final activations provide the non-spatial outputs used for classification or regression.
  • 2.1 A 2D CONVOLUTIONAL NETWORK PRIOR: The convolutional filters and biases receive independent Gaussian priors, with weight variance scaled by the number of input channels to maintain activation variance.An additional division by filter size recovers the initialization scheme associated with He et al. (2016a).
  • 2.2 ARGUMENT FOR GP BEHAVIOUR: As the number of channels tends to infinity layer by layer, the multivariate central limit theorem makes the feature maps jointly Gaussian.The proof also establishes that different feature maps are identically distributed and independent through zero-mean independent downstream weights.
  • 2.2 ARGUMENT FOR GP BEHAVIOUR: The base case is Gaussian because each feature-map element is a linear combination of shared Gaussian biases and filters, while feature maps are iid conditioned on the data.The argument extends from pairs of inputs to finitely many input points.

3 THE CONVNET AND RESNET KERNELS

The ConvNet kernel avoids representing full feature-map covariances by propagating only diagonal covariances, enabling exact computation with cost close to a one-filter-per-layer CNN. The construction also extends to residual connections while preserving Gaussian-process behavior.

  • 3.1 GP MEAN AND COVARIANCE: The kernel computation tracks only covariances between corresponding feature-map locations, rather than all pairs of locations.This reduces the per-layer quantity from a full covariance representation to a diagonal covariance vector.
  • 3.2 COVARIANCE OF THE ACTIVITIES: For Gaussian activations, the elementwise covariance can be computed in closed form for several nonlinearities, including ReLU and erf.The recursion therefore combines diagonal covariance propagation with closed-form nonlinear activation expectations.
  • 3.3 EFFICIENCY OF THE CONVNET KERNEL: Diagonal covariances at layer ℓ + 1 depend only on diagonal covariances at layer ℓ, so the kernel recursion can be evaluated layer by layer.The required output covariance propagates backward through the network to corresponding input locations.
  • 3.3 EFFICIENCY OF THE CONVNET KERNEL: The resulting kernel has computational cost within a constant factor of a forward pass through the equivalent CNN with one filter per layer.The particular kernel form also permits the required quantities to be computed as convolutions.
  • 3.4 KERNEL FOR A RESIDUAL CNN: Skip connections between layers preserve GP behavior because the induction argument requires only that previous activations are iid Gaussian.The resulting construction is equivalent to pre-activation residual shortcuts.

4 EXPERIMENTS

Experiments evaluate CNN-equivalent GP kernels on MNIST, test convergence toward the infinite-channel limit, and measure computational cost. The 32-layer ResNet GP achieves strong performance, while kernel construction dominates runtime.

  • MNIST evaluation: The MNIST setup uses 50,000 training examples, 10,000 validation examples, and one-hot regression targets with values 1 and −1.Classification is reframed as multi-output regression because the GP classification likelihood is non-conjugate.
  • MNIST evaluation: Kernel training computes an N×N training kernel matrix, solves Kxx^-1Y, and predicts test outputs from Kx*xKxx^-1Y.The test kernel matrix contains similarities between each test example and all training examples.
  • Model configurations: The ResNet GP is a modified 32-layer residual architecture with an initial 3 × 3 convolution and final dense layer instead of average pooling.Pooling was removed because its output variance requires off-diagonal filter covariances, preventing the stated efficiency gains.
  • Finite-width behavior: Finite 32-layer ResNets approximate the limiting output distribution well around 100 channels, while empirical moments match closely even with 30 channels.The comparison samples 10,000 networks with 3, 10, 30, and 100 first-layer channels; typical ResNets use 64–192.
  • Computational efficiency: Computing the ResNet kernel for training, validation, and test took 3h 40min on two Tesla P100 GPUs, versus 43.25 ± 8.8 seconds for inversion and evaluation.Asymptotically, kernel construction costs O(N^2LD), while kernel inversion costs O(N^3).

5 RELATED WORK

The paper distinguishes its shallow CNN-induced GP from convolutional, deep-kernel, infinite-filter-size, and mean-field approaches. Its kernel restricts patch interactions and retains computational and performance advantages described in the comparison passages.

  • Convolutional GPs: Unlike van der Wilk et al.’s convolutional kernel, this work compares corresponding patch pairs rather than all cross-image patch pairs.The restricted pairing makes the CNN kernels cheaper to compute and is associated with superior Table 1 performance despite an approximate likelihood.
  • Deep GPs: Deep GPs stack multiple GPs with convolutional kernels, whereas this paper confines the hierarchy to the kernel and produces a shallow GP.The distinction concerns where the hierarchy is represented: in stacked functions versus in the kernel definition.
  • Deep kernels: Deep kernel learning preprocesses inputs with an optimized deep neural-network feature extractor before applying a classic GP kernel.The neural-network parameters are optimized by likelihood-based gradient ascent.
  • Other limits: The infinite-network analysis of Borovykh takes the limit over filter size rather than filter number, making it inapplicable to finite-dimensional real data.This differs from the infinite-filter limit used for the CNN-to-GP equivalence here.
  • Mean-field analyses: Mean-field analyses seek stable random initializations by tracking single-example variances, while GP kernels additionally require covariances across training and test examples.The two research directions therefore differ in both focus and required covariance information.

6 CONCLUSIONS AND FUTURE WORK

The paper establishes that infinitely wide Bayesian CNNs correspond to GPs with recursive kernels and reports strong handwritten-digit performance. It identifies broader applications and future work on scalable approximations.

  • Conclusions: Deep Bayesian CNNs with infinitely many filters are equivalent to Gaussian processes with recursive kernels.The paper also derives the GP-equivalent kernel for CNNs.
  • Conclusions: On handwritten digit classification, the proposed kernel outperforms previous GP approaches that do not incorporate a parametric neural network into the kernel.The conclusion frames this result as evidence of practical relevance for structured neural architectures.
  • Future work: Future directions include applying GP CNNs to adversarial examples, lifelong learning, and k-shot learning, and developing efficient multi-layered inducing-point approximations.These are stated as intended applications and improvement directions rather than demonstrated results in this section.

7 APPENDIX

The appendix extends convergence proofs to convolutional networks, addressing sequential and simultaneous width limits through exchangeable central-limit arguments and moment bounds. It also reports calibration experiments and clarifies when kernel rescaling leaves Gaussian-likelihood classification unchanged.

  • 7 APPENDIX: The main-text proof takes channel limits sequentially, whereas an alternative takes all layers to infinity simultaneously and introduces additional technical difficulties.Sequential limits simplify the argument because infinitely wide previous layers provide exactly Gaussian inputs, while simultaneous limits are described as more practically relevant.
  • 7 APPENDIX: The proof extends Matthews et al.'s framework from fully connected networks to CNNs by letting channel-width functions approach infinity.The construction adapts their theorem to convolutional channels and shows convergence of CNN outputs to Gaussian processes.
  • 7 APPENDIX: The convergence proof uses the Cramér–Wold device, an exchangeable central limit theorem, and verification of the theorem's moment conditions.When previous layers remain finite, channels are exchangeable but not independent, so a standard CLT cannot be applied directly.
  • 7 APPENDIX: Moment conditions are established by bounding eighth moments inductively, using linear-envelope bounds on the nonlinearities and uniform-integrability results.The argument handles unbounded but linearly enveloped nonlinearities and controls the expanded sums through the inductive moment hypothesis.
  • 7 APPENDIX: The uncertainty experiment uses a RobustMax likelihood with 1000 randomly selected inducing points to compare calibration for the proposed and RBF kernels.The appendix states that both kernels have similar calibration, while the figure plots predicted label probabilities against corresponding test-set proportions and shows training-bin proportions as bars.
  • 7 APPENDIX: Multiplicative rescaling of kernels does not change classification results under the Gaussian likelihood because layerwise weight factors can be taken outside the nonlinearity.This follows from the stated equivalence of kernels that differ only by multiplication in that likelihood setting.
Loading 1808.05587v2…