PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhanced Convolutional Neural Tangent Kernels

Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon S. Du, Wei Hu, Ruslan Salakhutdinov, Sanjeev Arora

arXiv:1911.00809v1cs.LGcs.CVcs.NEstat.ML

TL;DR

Kernel regression makes standard data augmentation difficult because training time is quadratic in the number of training images. The paper introduces LAP for CNN-GP and CNTK and combines it with patch-based preprocessing; CNN-GP reaches 89% accuracy on CIFAR-10, matching AlexNet.

  • Problem

    Kernel regression has quadratic training time in the number of training images, making it unclear how to incorporate data augmentation efficiently.

    Source: §1 · p. 2

  • Method

    The paper introduces LAP for CNN-GP and CNTK, connecting pooling-based symmetry to translation augmentation, and combines CNN-GP with Coates-style patch preprocessing.

    Source: §1 · p. 2, §6.2 · p. 11

  • Results

    89% accuracy on CIFAR-10 is achieved by CNN-GP with LAP and horizontal flips after Coates-style preprocessing, matching AlexNet.

    Source: §1 · p. 2, §7 · p. 13

  • Takeaways & Limitations

    The resulting approach is reported as the strongest classifier that is not a trained neural network.

    Source: §7 · p. 13

    The theoretical results use circular padding to avoid boundary effects.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Recent research shows that for training with $\ell_2$ loss, convolutional neural networks (CNNs) whose width (number of channels in convolutional layers) goes to infinity correspond to regression with respect to the CNN Gaussian Process kernel (CNN-GP) if only the last layer is trained, and correspond to regression with respect to the Convolutional Neural Tangent Kernel (CNTK) if all layers are trained. An exact algorithm to compute CNTK (Arora et al., 2019) yielded the finding that classification accuracy of CNTK on CIFAR-10 is within 6-7% of that of that of the corresponding CNN architecture (best figure being around 78%) which is interesting performance for a fixed kernel. Here we show how to significantly enhance the performance of these kernels using two ideas. (1) Modifying the kernel using a new operation called Local Average Pooling (LAP) which preserves efficient computability of the kernel and inherits the spirit of standard data augmentation using pixel shifts. Earlier papers were unable to incorporate naive data augmentation because of the quadratic training cost of kernel regression. This idea is inspired by Global Average Pooling (GAP), which we show for CNN-GP and CNTK is equivalent to full translation data augmentation. (2) Representing the input image using a pre-processing technique proposed by Coates et al. (2011), which uses a single convolutional layer composed of random image patches. On CIFAR-10, the resulting kernel, CNN-GP with LAP and horizontal flip data augmentation, achieves 89% accuracy, matching the performance of AlexNet (Krizhevsky et al., 2012). Note that this is the best such result we know of for a classifier that is not a trained neural network. Similar improvements are obtained for Fashion-MNIST.

1 Introduction

The paper adapts CNN-GP and CNTK to incorporate data augmentation efficiently, introducing Local Average Pooling (LAP) to model small translations. With LAP, horizontal flips, and Coates-style preprocessing, CNN-GP reaches 89% accuracy on CIFAR-10, matching AlexNet.

1 Introduction

  • CNNs with infinite width correspond to CNN-GP when only the last layer is trained and CNTK when all layers are trained.The paper also derives a CNN layer corresponding to LAP and observes improved performance on certain architectures.Source: §1 · pp. 1–2
  • Kernel regression makes ordinary augmentation difficult because its training time is quadratic in the number of training images.Unlike SGD-based deep learning, kernel training cannot trivially add transformed images on the fly.Source: §1 · p. 2
  • GAP in CNN-GP or CNTK is equivalent to full translation augmentation with wrap-around boundaries.The equivalence follows from imposing a symmetry constraint during the kernel computation.Source: §1 · p. 2
  • GAP's full-translation augmentation can create unrealistic images, motivating LAP's assumption of label invariance only to small translations.The paper also proposes BBlur, a CNN pooling layer corresponding to LAP, and reports benefits in experiments.Source: §1 · p. 2
  • 81% on CIFAR-10, 3% above the best previous kernel predictor, is achieved by CNN-GP with LAP.LAP consistently improves CNN-GP and CNTK on CIFAR-10 and Fashion-MNIST.Source: §1 · p. 2
  • 89% accuracy on CIFAR-10, matching AlexNet, is achieved by CNN-GP with LAP, horizontal flips, and Coates-style preprocessing.The preprocessing uses randomly sampled training-image patches as convolutional filters.Source: §1 · p. 2

2 Related Work

Related work situates the paper at the intersection of data augmentation, neural-network kernels, and pooling. It connects GAP to data augmentation and combines Coates et al.'s patch-based approach with CNTK and LAP.

2 Related Work

  • Data augmentation has been studied for neural networks and kernel methods, including theoretical accounts of feature averaging and variance regularization.The cited work also reports provable performance improvement in certain settings.Source: §2 · p. 3
  • CNN-GP represents networks with only the last layer trained, while CNTK represents networks trained across all layers.Jacot et al. named the neural tangent kernel, and Arora et al. proved equivalence between polynomially wide networks trained by gradient descent and NTK prediction.Source: §2 · p. 3
  • Prior work proposed GAP in CNNs and conjectured that it reduces final-layer parameters and improves robustness to spatial translations.This paper studies GAP through CNN-GP and CNTK and connects it to data augmentation.Source: §2 · p. 3
  • Prior work combined CNTK with LAP and Coates et al.'s approach to achieve the strongest reported performance for classifiers that are not trained neural networks.The paper describes Coates et al.'s approach as one of the best-performing CIFAR-10 approaches preceding modern CNNs.Source: §2 · p. 3

3 Preliminaries

The preliminaries establish notation and describe CNNs, CNN-GP, CNTK, and the augmentation schemes used in the paper. They also specify circular padding for theoretical results to avoid image-boundary effects.

3.1 Notation

  • The notation section defines boldface vectors, matrices, and tensors, along with indexed entries and tensor coordinates.It also uses ⊙ for pointwise products, ⊗ for tensor products, and diag(·) for diagonal matrices.Source: §3.1 · p. 3

3.2 CNN, CNN-GP and CNTK

  • CNN outputs may be defined with or without GAP, and CNN-GP and CNTK likewise have output kernels with or without GAP.The section introduces image dimensions, filter size, and the intermediate CNN representation used in these definitions.Source: §3.2 · p. 4
  • The theoretical results use circular padding to avoid boundary effects; circular and zero padding differ only at image boundaries.Circular padding wraps out-of-range image coordinates, whereas zero padding assigns them zero.Source: §3.2 · p. 4

3.3 Data Augmentation Schemes

  • The augmentation schemes are full translations and horizontal flips, with translated images retaining their original labels.The translation scheme constructs examples across image positions; horizontal-flip augmentation trains on both flipped and original images.Source: §3.3 · p. 5

4 Equivalence Between Augmented Kernel and Data Augmentation

The section formalizes when kernel augmentation matches training on an augmented dataset. For CNN-GP and CNTK with GAP, this equivalence connects the kernels to full-translation augmentation and offers a more efficient route to kernel regression.

4 Equivalence Between Augmented Kernel and Data Augmentation

  • With circular padding and GAP, CNN-GP and CNTK use the augmented kernel for the group of all image translations.The corresponding GAP kernels match predictions from full-translation-augmented data.Source: §4 · p. 6
  • An equivariant kernel’s augmented-kernel prediction equals prediction using the original kernel on the augmented dataset.The equivalence applies to a group G and dataset (X, y).Source: §4 · p. 6
  • Horizontal-flip augmentation is represented by the group containing the flip and identity, yielding an equivalent augmented-dataset prediction.The corollary applies to GAP kernels Σ and Θ.Source: §4 · p. 6
  • Augmented kernels keep the kernel matrix at the original dataset size, avoiding the larger matrix needed for explicit full-translation augmentation.For flips, the augmented dataset doubles in size, while the augmented-kernel matrix remains the original size.Source: §4 · p. 6
  • Full-translation augmentation can produce unrealistic images, whereas local translation augmentation produces more realistic ones.The figure illustrates randomly sampled CIFAR-10 images under the two schemes.Source: Figure 1 · p. 7

5 Local Average Pooling

Local Average Pooling (LAP) incorporates restricted translations into the kernel, avoiding the computational infeasibility of naively augmenting data locally. Its formula interpolates between no pooling and GAP, and it corresponds to a box-blur layer in CNNs.

5 Local Average Pooling

  • LAP restricts translations to offsets within [−c, c] on each axis, where c controls the allowed translation distance.The restriction is motivated by full translation augmentation’s potential to create unrealistic images; the derivation assumes P = Q and first treats CNTK.Source: §5 · p. 7
  • Naive local translation augmentation is computationally infeasible even for moderate c, motivating a kernel formula for LAP.The CNTK formula is stated under circular padding; the authors say it generalizes straightforwardly to CNN-GP.Source: §5 · pp. 7–8
  • With zero padding, LAP becomes GAP when c = P and becomes no pooling when c = 0; the formula incurs no extra computational cost.The formula is well-defined for all padding schemes, and its weights can be calculated in constant time.Source: §5 · p. 8
  • In CNNs, the box-blur layer BBlur corresponds to LAP, and standard average pooling with size 2c + 1 and stride 1 is equivalent to BBlur.The paper reports experimental evaluation of BBlur on CNNs.Source: §5 · p. 8

6 Experiments

Experiments evaluate CNTK and CNN-GP on CIFAR-10 and Fashion-MNIST, then test whether Coates-style preprocessing improves CIFAR-10 performance and whether BBlur helps a finite CNN. LAP’s best setting varies by dataset, while preprocessed CNN-GP reaches AlexNet-level accuracy on CIFAR-10.

6 Experiments

  • Experiments use the full CIFAR-10 and Fashion-MNIST training and test sets, 3×3 stride-1 convolutions, and standard image normalization.The CNTK and CNN-GP depth study tests 5, 8, 11, or 14 convolutional layers; its tables report test accuracy with and without horizontal flips.Source: §6 · p. 8, §6.1 · p. 9

6.1 Ablation Study on CIFAR-10 and Fashion-MNIST

  • The best LAP setting is c=12 or 16 on CIFAR-10 and c=4 on Fashion-MNIST, consistently across tested depths and kernel families.The authors attribute the difference tentatively to greater translation tolerance in real-life images than centered clothing images.Source: §6.1 · pp. 10–11
  • Horizontal-flip augmentation is less effective on Fashion-MNIST than on CIFAR-10, possibly because many clothing images are nearly symmetric or baseline accuracy is already high.Reported CNTK accuracies for c=0 and c=32 on CIFAR-10 are close to Arora et al. [2019]; CNN-GP results at c=0 are close to Novak et al. [2019] on CIFAR-10 and Fashion-MNIST.Source: §6.1 · p. 11

6.2 Improving Performance on CIFAR-10 via Additional Pre-processing

  • The Coates-style preprocessing uses 2,048 normalized, mean-subtracted 5×5 training-image patches as convolutional filters, then applies ZCA and paired horizontal-flip filters.The output feature dimension for a 32×32 CIFAR-10 image is 8192×28×28; patch-count effects beyond 2,048 were left for future study because of computational limits.Source: §6.2 · p. 11
  • 88.92% on CIFAR-10 with 2,048 image patches matches AlexNet using additional preprocessing with CNTK or CNN-GP.The reported result improves on Coates et al. [2011] alone, which reached 84.2% with 256,000 patches or 83.3% with 32,000; with horizontal flips, those results were 85.6% and 85.0%, respectively. Further gains from using more patches are conjectured but untested because of computational limits.Source: §6.2 · pp. 11–12, Table 6 · p. 12, Table 5 · p. 12

6.3 Experiments on CNN with BBlur

  • BBlur with an appropriate c outperforms GAP on a 10-layer CIFAR-10 CNN, even with a fixed final fully connected layer.The experiment also found that GAP could improve CNN performance in this setting.Source: §6.3 · p. 12, Figure 2 · p. 13

7 Conclusion

The paper combines Local Average Pooling (LAP) with Coates et al.’s preprocessing technique to improve CNN-GP and CNTK image-classification performance. The resulting CNN-GP matches AlexNet on CIFAR-10 at 89% accuracy; the conclusion also identifies further CNN techniques to study through these kernels.

7 Conclusion

  • With LAP and Coates et al.’s preprocessing, CNN-GP achieves 89% accuracy on CIFAR-10, matching AlexNet and described as the strongest classifier that is not a trained neural network.Source: §7 · p. 13
  • LAP is derived for CNTK and CNN-GP from the connection between full translation data augmentation and Global Average Pooling (GAP), and consistently improves image-classification performance.The preprocessing technique is attributed to Coates et al. [2011].Source: §7 · p. 13
  • The paper proposes studying batch normalization, residual layers, and other modern CNN components through CNTK and CNN-GP.Source: §7 · p. 13

A Formal Definitions of CNN, CNN-GP and CNTK

This section establishes notation and formalizes CNNs, CNN-GP, and CNTK. It describes convolutional network components and recursive kernel definitions.

A Formal Definitions of CNN, CNN-GP and CNTK

  • The notation includes the activation function and its derivative, Gaussian distributions, and standard vector and matrix operations used in the definitions.ReLU is given as an example activation, with σ(z) = max{z, 0}.Source: A Formal Definitions of CNN, CNN-GP… · p. 16
  • The section defines convolution for filters and images and specifies that the input values used depend on the padding scheme.Source: A Formal Definitions of CNN, CNN-GP… · p. 16
  • The section formalizes CNN, CNN-GP, and CNTK, starting from an input image with an initial channel count and defining intermediate outputs layer by layer.The convolutional filters and biases use Gaussian initialization.Source: A Formal Definitions of CNN, CNN-GP… · p. 16
  • The kernel definitions follow similar patterns to NTK definitions, with an additional trace over patches representing convolution before the kernel value is defined recursively.Source: A Formal Definitions of CNN, CNN-GP… · p. 17

B Additional Definitions and Proof of Theorem 4.1

This appendix defines a group of operators and gives proof details for Theorem 4.1. Under the stated invertibility and equivariance conditions, it relates coefficients for the group-augmented kernel to the original coefficients.

B Additional Definitions and Proof of Theorem 4.1

C Equivalence Between LAP and Box Blur Layer.

This section relates a CNN box-blur layer to LAP. For CNTK, the two are equivalent; the CNN-GP equivalence is stated to follow similarly.

C Equivalence Between LAP and Box Blur Layer.

D Setting of the Experiment in Section 6.3

The experiments use an 80-epoch training schedule with specified optimization settings, circular padding, and three repetitions; Figure 2 distinguishes average from best test accuracy.

Loading 1911.00809v1…