Finite Versus Infinite Neural Networks: an Empirical Study

Jaehoon Lee; Samuel S. Schoenholz; Jeffrey Pennington; Ben Adlam; Lechao Xiao; Roman Novak; Jascha Sohl-Dickstein

Source-linked AI summary

Finite Versus Infinite Neural Networks: an Empirical Study

Jaehoon Lee, Samuel S. Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, Jascha Sohl-Dickstein

arXiv:2007.15801v2Jul 31, 2020cs.LGstat.ML

TL;DR

The paper asks how finite neural networks relate empirically to their infinite-width kernel limits, an area with many unresolved foundational questions. It performs a large controlled study of architectures, training choices, preprocessing, regularization, precision, and ensembling. Across these experiments, it finds substantial architecture- and practice-dependent differences, develops improved kernel and finite-network procedures, and reaches state-of-the-art CIFAR-10 results for the studied architecture-specific kernels.

Problem
Foundational empirical questions remain about how finite neural networks correspond to infinite-width kernel methods and which factors drive their performance.
Method
The paper conducts a large controlled empirical study comparing finite networks, NNGP and NTK kernels, and linearized networks across architectures, interventions, and datasets.
Results
Infinite kernels outperform finite networks for some architectures but underperform convolutional finite-width networks, while NNGP usually outperforms NTK and best-practice kernel ensembling achieves state-of-the-art CIFAR-10 results for the studied architecture classes.
Takeaways & Limitations
Kernel and finite-network performance depends strongly on architecture and training choices, so improved parameterization, regularization, preprocessing, and ensembling practices are useful for practical prediction.

Abstract

from arXiv · show

We perform a careful, thorough, and large scale empirical study of the correspondence between wide neural networks and kernel methods. By doing so, we resolve a variety of open questions related to the study of infinitely wide neural networks. Our experimental results include: kernel methods outperform fully-connected finite-width networks, but underperform convolutional finite width networks; neural network Gaussian process (NNGP) kernels frequently outperform neural tangent (NT) kernels; centered and ensembled finite networks have reduced posterior variance and behave more similarly to infinite networks; weight decay and the use of a large learning rate break the correspondence between finite and infinite networks; the NTK parameterization outperforms the standard parameterization for finite width networks; diagonal regularization of kernels acts similarly to early stopping; floating point precision limits kernel performance beyond a critical dataset size; regularized ZCA whitening improves accuracy; finite network performance depends non-monotonically on width in ways not captured by double descent phenomena; equivariance of CNNs is only beneficial for narrow networks far from the kernel regime. Our experiments additionally motivate an improved layer-wise scaling for weight decay which improves generalization in finite-width networks. Finally, we develop improved best practices for using NNGP and NT kernels for prediction, including a novel ensembling technique. Using these best practices we achieve state-of-the-art results on CIFAR-10 classification for kernels corresponding to each architecture class we consider.

1 Introduction

Infinite-width neural networks correspond to Gaussian processes and kernel methods, providing exact theoretical descriptions for broad classes of Bayesian and gradient-trained networks. This paper conducts an extensive empirical study to answer foundational questions about finite–infinite correspondence and improve practical performance.

Infinite-width Bayesian networks are described by NNGP kernels, while gradient-trained networks are described by NTK and weight-space linearization.
The correspondence between wide neural networks and kernel methods has supported advances in neural-network theory, kernel methods, Bayesian deep learning, active learning, and semi-supervised learning.
The study systematically investigates finite and infinite width networks to quantify performance drivers, uncover unexpected behaviors, and develop practices that improve both network types.

2 Experiment design

The experiments establish comparable finite-network, kernel, and linearized baselines across fully connected and convolutional architectures. They use controlled training and preprocessing choices on CIFAR-10, with selected claims evaluated on additional datasets.

Finite-network baselines use mini-batch gradient descent with a constant small learning rate and MSE loss for direct comparison with closed-form kernel predictions.
The study uses ReLU networks with critical initialization and compares fully connected and convolutional architectures under standard and NTK parameterizations.
Kernel experiments independently optimize diagonal regularization, while finite networks generally use the small-learning-rate base case unless otherwise specified.
Computing full CNN-GAP kernels for CIFAR-10 requires explicitly evaluating a 6 × 10^7-by-6 × 10^7 matrix, typically taking about 1200 GPU hours in double precision.
The experiments evaluate every intervention across architectures on CIFAR-10 and test several key claims on CIFAR-100 and Fashion-MNIST for robustness across datasets.

3 Observed empirical phenomena

The experiments show that finite-versus-infinite performance depends strongly on architecture, parameterization, optimization, regularization, preprocessing, width, and numerical precision. Centering, ensembling, and improved regularization can narrow or reverse performance gaps, while several effects depart from standard kernel or double-descent expectations.

Base finite networks: Infinite FCN and CNN-VEC networks outperform their respective finite-width networks, whereas finite CNN-GAP networks outperform their infinite counterparts in the base case.Architecture determines the relative performance of finite and infinite networks; infinite FCNs can remain ahead despite high learning rate, L2 regularization, or underfitting until ensembling is applied.
NNGP versus NTK: NNGP inference usually outperforms NTK inference across architectures and datasets, with the strongest fixed-kernel results attained by NNGP.NNGP kernels also require about half the memory and compute of corresponding NTKs.
Centering and ensembling: Ensembling reduces finite-network prediction variance, closes the FCN gap with kernels, and lets ensembled nonlinear CNNs significantly outperform the best kernel methods.The improvement is attributed to removing excess finite-width prediction variance through averaging.
Centering and ensembling: Centering significantly accelerates training and improves generalization for FCN and CNN-VEC models, but has little-to-no effect on CNN-GAP architectures.Centering subtracts each model's initial predictions from its evolving predictions.
Optimization and regularization: L2 regularization improves all architectures by 1-2%, while combining L2 with early stopping boosts finite CNN-VEC performance by 10%-15% and can surpass NNGP and NTK.Large learning rates also have architecture- and parameterization-dependent effects, including a +5% boost for CNN-VEC with standard parameterization.
Optimization and regularization: L2 regularization benefits NTK-parameterized finite networks by up to 6% more than standard-parameterized networks, motivating a layer-wise standard-parameterization regularizer with λ_l = λn_l.The modified regularizer is designed to match the penalty applied to the equivalent NTK-parameterized network.
Width dependence: Finite-network performance usually improves monotonically with width, but CNN-VEC with NTK parameterization has an intermediate optimal width, a non-monotonicity distinct from double descent.All widths in this comparison correspond to overparameterized models.
Kernel regularization and early stopping: Diagonal kernel regularization closely tracks early stopping, with the best validation performance usually obtained using early stopping and non-zero ε.The paper reports an approximate correspondence between stopping time and diagonal regularization strength.

4 Discussion

The discussion consolidates the study’s empirical contributions, caveats, and practical advances for finite and infinite neural networks.

The study quantified generalization, architecture dependence, finite–infinite deviations, numerical stability, augmentation, preprocessing, ensembling, topology, and linearization failure modes.
The normalized Gaussian Myrtle kernel lacks a corresponding finite-width neural network and was additionally tuned on CIFAR-10’s test set.
Data-augmentation ensembling improves kernel prediction accuracy and makes large augmented datasets computationally tractable, especially for NNGP.
The authors developed best practices that improve performance for both finite and infinite networks.
Further study of self-attention, normalization, residual connections, and systematic finite-width corrections remains an open direction.

Broader Impact

The paper frames neural-network theory as relevant to understanding and mitigating model bias and failure in safety-critical applications.

Understanding model biases matters because neural networks can perpetuate and exaggerate existing racial, gender, and other social biases.
Understanding model failure matters for human safety as neural networks increasingly drive cars and control the electrical grid.
The authors hope their empirical foundations will support safer and more just machine-learning models.

Supplementary Material

The supplementary material defines the paper’s experimental abbreviations and summarizes CIFAR-10 comparisons across finite networks and kernel methods.

The paper defines abbreviations including L2 for weight decay, LR for large learning rate, DA for data augmentation, Ens for ensembling, and ZCA for preprocessing.
It also distinguishes FCNs, CNN-VEC, and CNN-GAP by architecture and readout structure.
Table S1 compares nonlinear and linearized finite networks with NTK and NNGP methods, including interventions such as centering, learning rate, early stopping, ZCA, and ensembling.

C Experimental details

The experiments use Neural Tangents, standardized image datasets, MSE-based finite-network training, and regularized NNGP/NTK inference under controlled learning-rate and preprocessing settings.

All experiments use the Neural Tangents library built on JAX, with common settings followed by experiment-specific hyperparameters.
Finite-width networks are trained with MSE loss and mini-batch SGD at a constant learning rate.
L2 experiments add L2 regularization to the loss, with λ selected by validation-accuracy grid search.
Learning rates are expressed relative to the critical learning rate, with large-learning-rate runs allowing factors c > 1 while linearized networks prohibit c > 1.
Kernel inference grid-searches diagonal regularization, parameterized so ε is invariant to kernel scale.
Inputs are standardized per channel, regularized ZCA whitening is applied, and CIFAR augmentation uses random horizontal flips and four-pixel crops.

C.1 Hyperparameter configurations for all experiments

The experiments use grid searches over regularization and learning-rate factors, with distinct settings for linearized, centered, ensemble, and loss-comparison runs.

General selection: Hyperparameters are selected by validation accuracy, while reported results are generally test-set performance.The same validation procedure also determines underfitting and early-stopping measurement steps.
Grid searches: The main grid searches L2 strength from 0 through 10^-9 to 10^-3 and learning-rate factors from 2^-2 to 2^5.Linearized networks exclude learning-rate factors above 1 because training diverges; centered nonlinear runs use factors from 2^0 to 2^4.
Ensembling: Network ensembles use λ = 0 and c = 1 across 64 initialization seeds, while kernel ensembles use architecture-dependent predictor counts.The cited configuration specifies 50 predictors for FCN and CNN-VEC and begins a separate count for CNN-GAP.
Figure-specific settings: Learning-rate factors are c = 1 for nonlinear networks and c = 0.5 for linearized networks in the corresponding figure experiments.Shaded regions span minimum to maximum performance across 64 seeds, and solid lines show mean performance.
Data augmentation: Data-augmentation kernel ensembles use optimal ZCA regularization and 50 augmented draws for FCN and CNN-VEC versus 32 for CNN-GAP.The implementation takes the first ensemble instance from the nonaugmented training set.

D Noise model

The noise model estimates when finite-precision perturbations can make an otherwise positive-semidefinite kernel indefinite, using eigenvalue decay and machine-precision noise scales.

Precision scales: Float32 and float64 machine precisions are modeled with noise scales σ_n ≈ 10^-7 and σ_n ≈ 2 × 10^-16, respectively.These values are used as proxies for machine precision relative to kernel entries of order 1.
Regularization: A diagonal regularizer larger than the noise-induced minimum eigenvalue magnitude is needed to ensure positive definiteness when perturbations make the kernel indefinite.For CIFAR-10 with m = 50k, typical negative eigenvalues from float32 noise are around 4 × 10^-5 and 7 × 10^-14.
Eigenvalue scaling: For kernel entries of order 1, the smallest kernel eigenvalue scales as λ_min(K) ∼ C m^(1−α) under the observed power-law decay.Here m is dataset size, α is the eigenvalue-decay exponent, and C is dataset-size independent.
Noise mechanism: The smallest eigenvalue of a perturbed kernel can become negative when negative noise overwhelms the kernel’s smallest eigenvalue, breaking positive semidefiniteness.The model considers a worst-case perturbation affecting the smallest eigenvalue.
Critical dataset size: The model predicts a critical dataset size m* beyond which positive semidefiniteness can be broken for specified noise scale and eigenvalue decay.The critical size depends on the interaction between the kernel’s eigenvalue decay and perturbation magnitude.
Architecture dependence: FCN and CNN-VEC kernels with small α avoid worsening with dataset size under float32, whereas CNN-GAP conditioning increases precision requirements.Diagonal regularization can mitigate noise but may discard information in small-eigenvalue directions and reduce generalization strength.

E Data augmentation via kernel ensembling

The paper studies kernel ensembling by averaging predictions across independently transformed training sets, including stochastic data augmentation, and finds only small gains from inverse-variance weighting over simple averaging.

Ensemble construction: Each predictor produces μ(x*, D_i) for an unseen point from a training set D_i, and ensembling averages predictions across the sequence of training sets.The learning algorithms considered are NNGP and NTK inference.
Averaging methods: Naive averaging is used as a simple committee-machine ensemble, while a weighted alternative generalizes the average with weights satisfying the stated normalization condition.The paper favors simplicity over more sophisticated ensembling methods.
Weighting comparison: Inverse-variance weighting provides small improvements over naive averaging in simple bagging, indicating similar posterior variance across training-set draws.This comparison concerns NNGP/NTK predictors with MSE loss.
Data augmentation: Data augmentation generates each D_i by applying a stochastic transformation T to a clean training set, with identity retained at probability 1 − p.The experiments use horizontal flips and four-pixel random crops with augmentation fraction p = 0.5.

F ZCA whitening

ZCA whitening transforms inputs using covariance eigen-directions to remove feature correlations while preserving the original image’s spatial and chromatic structure; a nonzero regularizer yields ZCA-regularized preprocessing.

Whitening objective: Whitening seeks a transformation W that makes the transformed training features have identity covariance.The covariance is constructed from the training set, while W is applied to both training and test inputs.
ZCA choice: ZCA whitening rescales inputs along covariance eigen-directions without applying additional rotations or flips.This distinguishes ZCA from other whitening transformations.
Construction: The whitening transformation is computed by eigendecomposing the training covariance matrix into eigenvalues and eigenvectors.The diagonal matrix D contains eigenvalues, and U contains the covariance eigenvectors as columns.
Regularized ZCA: A nonzero regularization strength ϵ defines ZCA regularization preprocessing, while ϵ → 0 recovers whitening with identity covariance.The regularizer is reparameterized so its strength is invariant to input scale.
Image preprocessing: ZCA transformation preserves spatial and chromatic image structure, so image inputs are reshaped to retain their original image shape.Training and test features are standardized before and after whitening to obtain mean zero and variance of order 1.

G MSE vs Softmax-cross-entropy loss training of neural networks

The study compares finite neural networks with kernel methods under different training losses, optimization settings, parameterizations, preprocessing choices, and ensembling strategies. Crossentropy benefits classification, while centering, ensembling, NTK parameterization, and ZCA regularization can substantially affect finite-network and kernel behavior.

Optimization setup: Full-batch gradient descent gives the direct NTK correspondence, but the large CIFAR-10 study uses mini-batch SGD with batch sizes 100 for FCN and 40 for CNNs.These choices are practical approximations to the full-batch comparison setting.
Optimization setup: For the studied workloads, increasing batch size beyond the chosen values does not affect training dynamics, and the maximal useful batch size is relatively small.The batch-size study compares the selected settings with much larger batches.
Loss functions: Softmax-cross-entropy provides a clear classification benefit over MSE, although MSE-trained networks remain competitive.The comparison covers base networks with multiple interventions.
Kernel comparisons: NNGP often outperforms NTK on UCI regression RMSE, although this dominance is less consistent across tabular datasets than in image classification.The comparison varies depth, weight variance, and bias variance across standard 20-fold splits.
Finite-to-infinite correspondence: Centering accelerates finite-network training, while ensembling base networks matches kernel performance or exceeds it for nonlinear CNNs.The figures report validation and training dynamics for centering and performance comparisons for ensembles.
Regularization and width: NTK parameterization makes L2 regularization more helpful, while regularization toward initial weights provides no performance benefit and performance varies non-monotonically with width for CNN-VEC.ZCA regularization also helps finite-network training under both standard and NTK parameterizations.