PaperlayerView exact revision on arXiv

Source-linked AI summary

Properties of the After Kernel

Philip M. Long

arXiv:2105.10585v3cs.LGcs.AIcs.NE

TL;DR

The paper examines the neural tangent kernel (NTK), whose definition refers to a neural network at initialization. The study trains VGG-like and fully connected networks on binary MNIST and CIFAR10 tasks, then evaluates after kernels using hard-margin SVM accuracy, globality, invariance, conjugate projections, and learning-rate comparisons. After kernels often outperform initialization kernels after a few epochs; VGG-like after kernels are more global, conjugate projections are architecture-dependent, and larger learning rates yield better kernels with greater globality and invariance.

  • Problem

    The paper examines the neural tangent kernel (NTK), whose definition refers to a neural network at initialization.

    Source: §1 · p. 1

  • Method

    The study trains VGG-like and fully connected networks on binary MNIST and CIFAR10 tasks, then evaluates after kernels using hard-margin SVM accuracy, globality, invariance, conjugate projections, and learning-rate comparisons.

    Source: §1 · pp. 2–3

  • Results

    After kernels often outperform initialization kernels after a few epochs; VGG-like after kernels are more global, conjugate projections are architecture-dependent, and larger learning rates yield better kernels with greater globality and invariance.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Kernel quality and invariance evolve rapidly during training, and the resulting after-kernel properties depend on architecture and learning rate rather than initialization alone.

    Source: §1 · pp. 2–3, §4 · p. 20

    The study reports that after kernels are not strongly invariant to small input perturbations in general, and data augmentation did not make them more invariant.

    Source: §4 · p. 20

Abstract

from arXiv · show

The Neural Tangent Kernel (NTK) is the wide-network limit of a kernel defined using neural networks at initialization, whose embedding is the gradient of the output of the network with respect to its parameters. We study the "after kernel", which is defined using the same embedding, except after training, for neural networks with standard architectures, on binary classification problems extracted from MNIST and CIFAR-10, trained using SGD in a standard way. For some dataset-architecture pairs, after a few epochs of neural network training, a hard-margin SVM using the network's after kernel is much more accurate than when the network's initial kernel is used. For networks with an architecture similar to VGG, the after kernel is more "global", in the sense that it is less invariant to transformations of input images that disrupt the global structure of the image while leaving the local statistics largely intact. For fully connected networks, the after kernel is less global in this sense. The after kernel tends to be more invariant to small shifts, rotations and zooms; data augmentation does not improve these invariances. The (finite approximation to the) conjugate kernel, obtained using the last layer of hidden nodes, sometimes, but not always, provides a good approximation to the NTK and the after kernel. Training a network with a larger learning rate (while holding the training error constant) produces a better kernel, as measured by the test error of a hard-margin SVM. The after kernels of networks trained with larger learning rates tend to be more global, and more invariant to small shifts, rotations and zooms.

1 Introduction

The paper studies how neural-network kernels change after training, finding that after kernels can outperform initialization kernels and acquire architecture- and learning-rate-dependent invariances.

Conceptual framing

  • The after kernel captures post-training features, contrasting with the NTK’s initialization-fixed features under the linearized training regime.The NTK is defined from the network’s parameter Jacobian at initialization, whereas the after kernel uses the analogous embedding after training.Source: §1 · p. 1

Experimental setup

  • The study evaluates three binary problems, VGG-like and fully connected architectures, and larger VGG-like networks, averaging results over ten random runs.The experiments use 3-vs-8 and 5-vs-6 from MNIST and cats-vs-dogs from CIFAR10.Source: §1 · p. 2

Kernel quality

  • After a few epochs, hard-margin SVMs using after kernels can substantially outperform the finite NTK on fully connected MNIST and VGG-like CIFAR10 tasks.Most improvement appears early, although the trained neural network eventually catches up.Source: §1 · p. 2

Globality

  • After kernels are more global than initialization kernels for VGG-like networks, whereas fully connected networks do not show the same immediate increase in global structure.Globality is measured by invariance to swapping image quadrants, which preserves local statistics but disrupts global structure.Source: §1 · pp. 2–3

Invariances

  • After kernels are usually more invariant to one-pixel shifts, small rotations, and slight zooms, while data augmentation tends not to improve these invariances.Augmentation can improve test error despite failing to improve kernel invariance.Source: §1 · p. 3

Conjugate projection

  • The conjugate projection approximates the after kernel well for VGG-like networks but not for fully connected MNIST networks, and SVM accuracy can differ accordingly.On CIFAR10 cats-vs-dogs, the after kernel is initially more accurate before the conjugate projection catches up.Source: §1 · p. 3

Learning rate

  • Larger learning rates produce better after kernels at matched training error, with greater globality and invariance and a smaller gap from conjugate-kernel performance.The paper interprets this pattern as consistent with higher learning rates producing better high-level features.Source: §1 · p. 3

2 Preliminaries and Methods

The paper evaluates after-kernel methods for binary image classification using several neural-network architectures, training schedules, and invariance tests. It defines kernel-quality and representation comparisons through hard-margin SVM error, perturbation similarity, and kernel alignment.

Training protocol

  • Training varies across epochs and learning rates, with each experiment repeated ten times and averaged with confidence intervals.Epoch counts are 0, 1, 3, 10, 30, 100, and 200; learning-rate studies stop when training loss reaches at most 0.01.Source: §2 · pp. 5–6

Experimental setup

  • The experiments use binary MNIST and CIFAR10 tasks with VGG-like, fully connected, larger VGG-like, and unit-test networks.The VGG-like models use convolution, pooling, and fully connected layers; larger variants have approximately one million parameters.Source: §2 · pp. 4–5

Evaluation

  • Kernel quality is evaluated by training hard-margin SVMs on network-derived kernels and measuring test error.The same training data are used to train the neural networks and the SVM models.Source: §2 · p. 5

Invariance measurements

  • Translation, rotation, and zoom invariance are measured by averaging embedding cosine similarities between original and perturbed test images.Perturbations include one-pixel shifts, rotations by 1/4 radians, and cropping followed by resizing.Source: §2 · p. 5
  • Global-image sensitivity is assessed by swapping image quadrants and comparing embeddings, because local statistics remain similar while global structure is disrupted.This test is especially relevant for MNIST, where quadrant swaps materially alter global image properties.Source: §2 · p. 5

Kernel comparisons

  • The conjugate projection is compared with the NTK before and after training using Gram-matrix alignment computed from flattened test-data matrices.The alignment uses vec(G)·vec(H), with vec denoting matrix flattening.Source: §2 · pp. 5–6

3 Results

After a few training epochs, the after kernel often improves hard-margin SVM accuracy over the initial kernel, while training changes its global structure and transformation invariances. Learning rate also shapes kernel quality, globality, invariance, and the usefulness of the conjugate projection.

Kernel accuracy

  • After a few epochs, the after kernel substantially improves hard-margin SVM test error over the initial kernel for several architecture–dataset pairs, while the neural network later catches up.The improvement is rapid early in training; the initial kernel corresponds to zero epochs.Source: §3 · p. 6, Figure 2 · p. 7

Globality

  • Training decreases swap-invariance for VGG-like networks, indicating greater dependence on global image structure, while fully connected kernels remain less global overall.For fully connected networks, the results are consistent with learning local features before global features.Source: Figure 4 · p. 9, §3 · p. 10

Transformation invariance

  • After one epoch, invariance to shifts, zooms, and rotations rises, then generally declines toward values above the initial kernel; fully connected kernels remain less invariant.The VGG-like CIFAR10 after kernel becomes much less rotation invariant than at initialization.Source: §3 · p. 10, Figure 5 · p. 11

Conjugate projection

  • The conjugate projection is a good approximation to the after kernel, especially at initialization, but becomes less accurate after training; SVM accuracy is usually nearly comparable, except for VGG on MNIST 5-vs-6.Source: §3 · p. 10

Data augmentation

  • Data augmentation improves trained-model accuracy but does not remove the learning-rate trend toward lower shift, zoom, and rotation invariance.The reduced invariance at larger learning rates persists with augmentation and is stronger on CIFAR10.Source: §3 · pp. 10, 12, Figure 8 · p. 14

Learning rate

  • Holding training error constant, larger learning rates generally produce better kernels, more global features, and lower invariance to small image transformations.For VGG networks, after–conjugate alignment is highest at the lowest learning rates; for fully connected networks, the opposite holds.Source: §3 · p. 12

4 Discussion

The after kernel differs qualitatively from the finite NTK and improves rapidly during training, while larger learning rates produce better kernels even at fixed training error. Its invariance to small input perturbations remains limited, and several extensions remain open.

4 Discussion

  • The after kernel is often qualitatively different from the finite neural tangent kernel.Source: §4 · p. 20
  • The after kernel improves rapidly during training, and larger learning rates produce better kernels even when training error is held constant.The advantage appears in both hard-margin SVM accuracy and invariance properties.Source: §4 · p. 20
  • The after kernel is not strongly invariant to small input perturbations, and data augmentation does not improve these invariances.The degree of invariance can also degrade slowly during later training.Source: §4 · p. 20
  • The study leaves extensions to contrastive training, EfficientNet, visual transformers, network depth, NLP tasks, and initialization scale for future work.The authors note that the accompanying code could support these investigations.Source: §4 · p. 20

A The networks are sufficiently overparameterized

The appendix tests whether the studied networks are sufficiently overparameterized by varying width while holding architecture and training protocol fixed. Across the three datasets, increasing model size from approximately 60K to 100K parameters does not substantially reduce test error.

A The networks are sufficiently overparameterized

  • The width study scales convolutional channels in VGG-like networks and hidden-layer sizes in fully connected networks.The evaluated models contain approximately 60K, 80K, or 100K parameters.Source: A The networks are sufficiently… · p. 25
  • Increasing model size from approximately 60K to 100K parameters does not improve test error much across the three datasets.The comparison uses average test errors from ten runs under the study's training protocol.Source: A The networks are sufficiently… · p. 25, Figure 15 · p. 26

B Sanity check: augmentations improve accuracy

The appendix sanity check confirms that data augmentation improves the test accuracy of learned models. It compares models trained with and without augmentation.

B Sanity check: augmentations improve accuracy

Related papers

Loading 2105.10585v3…