PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing the Power of Infinitely Wide Deep Nets on Small-data Tasks

Sanjeev Arora, Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, Dingli Yu

arXiv:1910.01663v3cs.LGstat.ML

TL;DR

Kernel methods offer an analytically tractable view of infinitely wide networks but scale poorly with dataset size, motivating evaluation on low-data tasks. The paper evaluates NTK and convolutional NTK classifiers across small-data benchmarks and reports strong performance against finite networks and established classifiers. It concludes that NTKs are promising off-the-shelf methods for low-data settings, while key equivalence and generalization questions remain open.

  • Problem

    Kernel methods can be computationally inefficient on large datasets, leaving limited evidence about whether Neural Tangent Kernels are practical and effective for small-data tasks.

  • Method

    The paper evaluates fully connected NTK and convolutional NTK classifiers against finite neural networks and established classifiers across low-data benchmarks.

  • Results

    NTKs perform strongly across low-data tasks, including UCI classification, CIFAR-10, and few-shot image classification, outperforming corresponding finite networks or standard classifiers in reported comparisons.

  • Takeaways & Limitations

    Fully connected NTK classifiers are presented as easy-to-compute, no-GPU, off-the-shelf alternatives for many low-data settings.

  • Takeaways & Limitations

    Open questions include whether NTK SVMs correspond to an infinite-net architecture and what explains generalization in small-data settings; kernel construction scales quadratically with data points.

Abstract

from arXiv · show

Recent research shows that the following two models are equivalent: (a) infinitely wide neural networks (NNs) trained under l2 loss by gradient descent with infinitesimally small learning rate (b) kernel regression with respect to so-called Neural Tangent Kernels (NTKs) (Jacot et al., 2018). An efficient algorithm to compute the NTK, as well as its convolutional counterparts, appears in Arora et al. (2019a), which allowed studying performance of infinitely wide nets on datasets like CIFAR-10. However, super-quadratic running time of kernel methods makes them best suited for small-data tasks. We report results suggesting neural tangent kernels perform strongly on low-data tasks. 1. On a standard testbed of classification/regression tasks from the UCI database, NTK SVM beats the previous gold standard, Random Forests (RF), and also the corresponding finite nets. 2. On CIFAR-10 with 10 - 640 training samples, Convolutional NTK consistently beats ResNet-34 by 1% - 3%. 3. On VOC07 testbed for few-shot image classification tasks on ImageNet with transfer learning (Goyal et al., 2019), replacing the linear SVM currently used with a Convolutional NTK SVM consistently improves performance. 4. Comparing the performance of NTK with the finite-width net it was derived from, NTK behavior starts at lower net widths than suggested by theoretical analysis(Arora et al., 2019a). NTK's efficacy may trace to lower variance of output.

1 Introduction

The paper asks whether Neural Tangent Kernels can make infinitely wide networks practical and effective on small-data tasks despite kernel methods’ poor large-dataset scaling. It reports strong performance across UCI tasks and motivates NTKs as stable, low-tuning alternatives in this regime.

  • Motivation: Kernel regression can require quadratic time in the number of data points, making CNTK computation impractical for large datasets and difficult to combine with CIFAR-10 augmentation.The cited discussion identifies kernel construction as the practical bottleneck.
  • Motivation: NTKs may suit small datasets because kernel classifiers have stable convex solvers with minimal hyperparameter tuning.The motivation contrasts this with the tuning and variance issues associated with finite neural networks.
  • Contributions: On 90 UCI classification tasks, NTK predictors achieve superior, arguably strongest classification performance across several standard statistical tests.The tests include Friedman Rank, Average Accuracy, PMA, P90, and P95.
  • Contributions: On every UCI dataset, NN and NTK accuracy differ by at most 5%, while differences from classifiers such as RF can reach 20%.The paper interprets this similarity as evidence that NTK describes finite neural-network behavior in low-data settings.

2 Related Work

The paper builds on prior UCI comparisons that established Random Forests as a reference and on theory connecting infinitely wide networks to kernels. It also situates NTK alongside extensions to graph architectures and notes unresolved computational and equivalence issues.

  • Prior UCI comparisons: Random Forests were treated as a gold-standard reference after earlier UCI experiments found them strongest, with Gaussian-kernel SVM following.The paper adopts this comparison strategy for evaluating NTK.
  • Open scope: NTKs can be used in kernel SVMs, but equivalence to infinitely wide network training is currently known only for ridge regression.The paper reports trying both SVM and ridge-regression formulations.
  • Neural-network kernels: NTKs differ from earlier neural-network-induced kernels because all network layers are trained, rather than only the last layer.The paper identifies Jacot et al. (2018) as the first proposal of this formulation.
  • Theory and approximation: Arora et al. (2019a) established a non-asymptotic perturbation bound between gradient-descent-trained neural-network and NTK predictors.Prior empirical work also verified NTK as a good approximation to NN on small-scale data.
  • Extensions: Graph neural tangent kernels have been proposed and reportedly outperform corresponding graph neural networks on datasets with up to 5000 samples.This places NTK-style methods within a broader extension beyond fully connected and convolutional architectures.

3 Neural Network and Neural Tangent Kernel

The paper defines fully connected neural networks recursively and derives their infinite-width covariance and Neural Tangent Kernels analytically. The resulting kernel depends on the trained layers and can be used directly for downstream classification.

  • Neural Network: The network takes x ∈ R^d as input and recursively defines an L-hidden-layer fully connected architecture.The input is identified with g^(0)(x), and d_0 is set equal to d.
  • Neural Network: The hidden-layer weights are matrices, activations are coordinate-wise functions, and the scaling factor c_σ is explicit in the NTK parameterization.Experiments consider ReLU or ELU for NNs, while NTK kernels use ReLU-induced architectures.
  • Neural Network: All network weights are initialized as independent N(0,1) random variables, and the final layer maps the last hidden representation to the output.The parameter vector w contains all network parameters.
  • Infinite-width limit: As hidden widths tend to infinity, layer-output covariance kernels admit analytic recursive forms that remain valid during small-learning-rate gradient-descent training.This persistence under training supports the kernel description of the network trajectory.
  • Neural Tangent Kernel: The NTK is defined as a limiting gradient kernel, and training only a subset of layers yields a corresponding modified kernel.When only the final layer is trained, the resulting kernel is the NNGP kernel; the number of fixed layers can be tuned.

4 Experiments on UCI Datasets

On 90 UCI datasets, NTK was the strongest overall classifier across reported metrics and generally outperformed RF, Gaussian-kernel SVM, and finite neural networks in pairwise tests.

  • Overall performance: NTK was the best classifier under all reported Table 1 metrics, followed by Random Forests.The comparison included RF, Gaussian and polynomial kernel SVMs, and neural networks with He or NTK initialization.
  • Overall performance: NTK achieved Friedman Rank 28.34 and Average Accuracy 81.95%, versus RF at 33.51 and 81.56%.The NTK–RF difference was significant: −5.17 in Friedman Rank and +0.39% in Average Accuracy.
  • Overall performance: NTK achieved the best average PMA, while RF was 0.47% lower and other classifiers were below 94.6%.NTK, NTK-initialized NN, and RF also had relatively small standard deviations of 5.17%, 5.89%, and 5.30%.
  • Pairwise comparisons: NTK outperformed RF on 42 datasets versus 40, with a statistically significant mean accuracy difference of 2.51%.The comparison counted a classifier as better when its accuracy was at least 0.1% higher.
  • Pairwise comparisons: NTK outperformed Gaussian-kernel SVM on 43 datasets versus 34, with a statistically significant mean accuracy difference of 2.22%.On five datasets with fewer than 200 samples, NTK exceeded Gaussian-kernel SVM by at least 11% accuracy.
  • Pairwise comparisons: NTK outperformed He-initialized NN on 50 datasets versus 27, with a statistically significant mean difference of 1.96%.Against NTK-initialized NN, no dataset showed a significant advantage for either classifier, although NTK was often better.

5 Experiments on Small CIFAR-10 Dataset

The study evaluates CNTK against ResNet-34 on subsampled CIFAR-10, finding CNTK stronger across small training-set sizes but not at the largest tested size.

  • Experimental setup: CNTK consistently outperforms ResNet-34 when trained on small subsampled CIFAR-10 datasets.The experiments vary the training size and CNTK depth, using the full CIFAR-10 test set for evaluation.
  • Results: 36.57% versus 33.15% is the largest reported gap, at n = 320 with 14-layer CNTK and ResNet-34, respectively.
  • Results: 15.33% versus 14.59% is the smallest improvement, at n = 10 for CNTK and ResNet-34, respectively.
  • Results: At n = 1280, ResNet-34 outperforms CNTK, while 14-layer CNTK is best among CNTK variants at every tested training size.

6 Experiments on Few-shot Learning

The few-shot experiments test CNTK as a replacement for linear SVMs on ResNet-50 features for VOC07 classification, with gains depending on feature level and CNTK depth.

  • Experimental setup: CNTK is evaluated as a drop-in replacement for linear SVM in few-shot VOC07 classification using ImageNet-trained ResNet-50 features.Features come from conv1 through conv5, and the number of positive examples per class varies from 1 to 8.
  • Evaluation protocol: The reported evaluation tunes C on the first 10 VOC07 classes and reports selected-C performance on the other 10 classes.
  • Results: Tables 3–9 show CNTK consistently outperforming the linear classifier, including on every randomly sampled training set.
  • Feature-level comparisons: CNTK improves over linear classification by about 1% with Conv5 features and by around 2%, sometimes 3%–4%, with Conv3 and Conv4 features.
  • CNTK depth: CNTK often performs best with a single convolutional layer, but three layers are best for Conv3 features when k = 7 or 8.

7 Conclusion

The paper concludes that NTKs perform strongly on low-data tasks and can serve as practical classifiers, while leaving theoretical explanations and extensions unresolved.

  • Conclusion: NTK achieves strong low-data performance, beating extensively tuned classic methods such as random forests.
  • Practical implication: Fully connected NTK classifiers are described as easy to compute without a GPU and suitable as off-the-shelf classifiers in many settings.
  • Practical implication: The paper plans code supporting drop-in replacement for SVMs and linear regression.
  • Open questions: Open questions include whether NTK SVMs correspond to infinite-net architectures, how small-data generalization works, and whether recurrent NTKs perform well on benchmarks.

A Additional Experimental Details on UCI

The UCI experiments use a low-data dataset selection and cross-validation protocol to compare NTK classifiers with neural-network baselines under tuned hyperparameters.

  • Dataset selection: The study retains preprocessed UCI datasets with fewer than 5000 samples to target the low-data regime.
  • Evaluation protocol: Performance is compared using 4-fold cross-validation with validation-based hyperparameter selection and balanced class splits.
  • NTK specification: NTK classifiers tune depth-related parameters and SVM cost across 105 combinations, fewer than the 500 combinations used for Gaussian-kernel SVM.
  • NN specification: Fully connected neural networks use 512 hidden nodes per layer, depths from 1 to 5, optional batch normalization, and learning rates of 0.1 or 1.
  • NN specification: Neural networks are trained for 2000 epochs, although they can reach zero training loss in fewer than 100 epochs.

B Additional Experimental Details on Small CIFAR-10 Datasets

The CIFAR-10 experiments evaluate classifiers across progressively larger low-data regimes, using repeated subsampling and comparisons between CNTK and ResNet-34 configurations.

  • CIFAR-10 sampling: 10 to 1280 samples were evaluated, with n/10 samples randomly chosen from each CIFAR-10 class.Each setting was repeated 20 times, reporting mean accuracy and standard deviation.
  • CNTK configuration: CNTK models used 5–14 convolutional layers, followed by global pooling and a fully connected layer.The kernels were normalized so each sample had unit length in feature space.
  • ResNet-34 baseline: ResNet-34 baselines used widths 64, 128, and 256 with learning rate 0.1, momentum 0.9, and weight decay 0.0005.Training used learning-rate decay at epochs 80 and 120 over 160 total epochs, reporting the best testing accuracy.

C Additional Results in Few-shot Learning

The VOC07 results compare linear SVM with CNTK across convolutional depths and feature-extraction stages, using the best regularization parameter C.

  • Feature stage comparisons: Table 7 compares linear SVM and CNTK using features extracted from conv5 of ResNet-50.Rows vary the number of convolutional layers used by CNTK across all VOC07 classes.
  • Feature stage comparisons: Table 8 compares linear SVM and CNTK using features extracted from conv4 of ResNet-50.The table varies CNTK convolutional depth across all VOC07 classes.
  • Feature stage comparisons: Table 9 compares linear SVM and CNTK using features extracted from conv3 of ResNet-50.The table varies CNTK convolutional depth across all VOC07 classes.
Loading 1910.01663v3…