Source-linked AI summary
Scaling description of generalization with number of parameters in deep learning
Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d'Ascoli, Giulio Biroli, Clément Hongler, Matthieu Wyart
TL;DR
Over-parametrized networks can fit training data while their test error improves with parameter count, a behavior that standard capacity arguments do not explain. The paper develops an NTK-based fluctuation framework, finding N^-1/4 output fluctuations and an asymptotic N^-1/2 test-error approach to a plateau, with a divergent jamming transition and an ensemble-based practical consequence.
Problem
Over-parametrized DNNs fit training data yet improve generalization as N increases, contrary to traditional overfitting expectations.
Method
The paper uses an NTK-based framework to analyze initialization-induced output fluctuations and their relationship to decision-boundary displacement in classification.
Results
N^-1/4 output fluctuations yield test-error convergence toward a plateau, while near N* the output norm appears to diverge and explain the test-error cusp.
Takeaways & Limitations
Given a fixed computational envelope, ensemble averaging of several networks slightly beyond N* can provide near-optimal generalization.
Takeaways & Limitations
Direct test-error fits can show an apparent exponent below 1/2 because the asymptote and corrections to scaling complicate estimation; cross-entropy dynamics may also build the jamming singularity slowly.
Abstract
from arXiv · showhide
Supervised deep learning involves the training of neural networks with a large number $N$ of parameters. For large enough $N$, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as $N$ grows past a certain threshold $N^{*}$. Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with $N$. We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations $\|f_{N}-\bar{f}_{N}\|\sim N^{-1/4}$ of the neural net output function $f_{N}$ around its expectation $\bar{f}_{N}$. These affect the generalization error $ε_{N}$ for classification: under natural assumptions, it decays to a plateau value $ε_{\infty}$ in a power-law fashion $\sim N^{-1/2}$. This description breaks down at a so-called jamming transition $N=N^{*}$. At this threshold, we argue that $\|f_{N}\|$ diverges. This result leads to a plausible explanation for the cusp in test error known to occur at $N^{*}$. Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, the smallest generalization error is obtained using several networks of intermediate sizes, just beyond $N^{*}$, and averaging their outputs.
Introduction
The paper addresses why over-parametrized deep networks can generalize well despite fitting training data, developing an NTK-based framework for parameter-dependent fluctuations and test error.
- Over-parametrized DNNs can fit training data yet generalize well without explicit regularization, challenging traditional capacity-based expectations.
- The test error decreases slowly as N grows beyond N*, but increases sharply near N*, producing a cusp-like behavior.
- The framework connects wide DNN training to the Neural Tangent Kernel, which becomes deterministic and time-independent as N→∞.
- N^-1/4 NTK fluctuations around their mean induce comparable finite-size fluctuations in the trained output function fN.
- Classification test-error changes are related to squared decision-boundary fluctuations, enabling predictions for generalization as N and ensemble size increase.
- Near the jamming transition, output fluctuations appear to diverge; the analysis suggests ||fN||∼(N−N*)^-1 and motivates ensemble averaging of intermediate-size networks beyond N*.
1 Setting
The paper studies fixed-depth fully connected DNNs for binary classification, trains them by gradient-based optimization of a square-hinge loss, and defines jamming through the smallest parameter count attaining minimum loss.
- The model uses fully connected networks with L layers and h neurons per layer, containing weights, biases, and a nonlinear activation such as ReLU.
- Binary predictions use sign f(x;θ) to assign labels ±1, with training minimizing a square-hinge cost based on the margin.
- Training uses a first-order method such as gradient descent and stops when the loss reaches its lowest possible value or the maximum running time.
- The jamming transition N* is the smallest parameter count for which training reaches the lowest possible loss.
- Experiments use parity classification on MNIST PCA inputs and CNNs on full CIFAR10 images, with full-batch training.
2 Numerical Results on MNIST
MNIST experiments show a cusp-like test-error maximum near jamming, followed by slow improvement with larger networks; averaging independent runs nearly removes this dependence beyond N*.
- The MNIST test error reaches a local maximum near the jamming transition and then slowly decreases as N increases.
- n=20 ensemble averaging produces nearly flat test error for N>N*, supporting reduced output variance as the source of improved generalization.
3 Relationship Between Variance and Generalization in Classification Tasks 1
The paper relates output-function variance to classification generalization through decision-boundary displacement, arguing that small fluctuations increase test error quadratically under smoothness assumptions.
- Writing fN=¯fN+δfN separates the learned function into its initialization-averaged output and a fluctuation around that average.
- For mean-square regression, output fluctuations contribute exactly ||δfN||^2 to the excess generalization error.
- 3.2 Classification task: The classification analysis assumes smooth random functions and compares decision boundaries B={f=0} and ¯B={¯f=0}.
- 3.2 Classification task: A small output perturbation displaces the boundary by δ(x0)=δf(x0)/||∇f(x0)|| plus higher-order terms.
- 3.2 Classification task: The typical boundary distance is well estimated by ||δf||µ/||∇f||µ over test points classified differently by f and ¯f.
- 3.2 Classification task: Under smooth-test-error assumptions, the average true test error increases quadratically with fluctuation size, with a positive prefactor near an optimal boundary.
4 Asymptotic generalization as n → ∞
As the number of averaged networks n grows, ensemble fluctuations decrease and the true test error approaches a constant with a 1/n correction under smooth-boundary assumptions.
- δ scales as 1/√n for the ensemble decision-boundary variation.The central limit theorem and law of large numbers support this asymptotic scaling.
- The true test error approaches its asymptotic value with a correction scaling as 1/n.
- The analysis assumes that true test error is a smooth function of the decision boundary.This holds when label-conditioned input distributions are smooth, such as with Gaussian input noise.
- A nonzero prefactor is expected generically, but it could vanish if the optimal decision boundary is degenerate.The authors do not expect this degeneracy generically when data include Gaussian noise.
- For n_in = 30, h = 60, L = 5, N = 16k, and P = 10k, the fitted asymptotic test error is 2.148%.The right-panel fit supports the relation between test-error increase and boundary variation, while the center panel supports δ ∼ 1/√n.
5 Asymptotic Generalization as N → ∞
In the over-parametrized regime, finite-width randomness in the NTK at initialization produces N^-1/4 output fluctuations, while classification generalization approaches an asymptotic plateau with N^-1/2 corrections. This scaling description is supported empirically, but requires correction terms for accurate finite-N fits.
- Finite-N fluctuations: N-independent initialization randomness remains, but training shrinks output fluctuations around the training points.The paper separates persistent randomness in the initial function from randomness generated by the training process.
- Finite-N fluctuations: N^-1/4 NTK fluctuations at initialization are the dominant source of finite-N fluctuations in the trained output function.The NTK becomes deterministic in the infinite-width limit, whereas its initialization fluctuations induce fluctuations of the same order in fN at convergence.
- Generalization scaling: N^-1/2 is the leading asymptotic order of the excess test error ϵN −¯ϵN caused by finite-N output fluctuations.The prediction is tested without fitting parameters and is confirmed asymptotically in Fig. 6.D.
- Generalization scaling: N^-1/2 also describes the difference between the mean finite-width test error ¯ϵN and its infinite-width value ¯ϵ∞.Feature learning and nonlinearity in the kernel-function relationship are both estimated as O(N^-1/2), smaller than the O(N^-1/4) output fluctuations.
- Generalization scaling: The predicted form ϵN −ϵ∞= B0N^-1/2 + B1N^-3/4 is consistent with observations, with B0 > 0 for MNIST fully-connected and convolutional networks.The positive leading coefficient agrees with finite N increasing output fluctuations, while Fig. 5 shows variance peaking at jamming and decaying afterward.
- Finite-size corrections: Direct test-error fits can yield an apparent exponent below 1/2 because the asymptote is fitted and correction-to-scaling terms are large.The large correction to scaling of ||∇fN||µ contributes to the discrepancy between direct fits and the asymptotic prediction.
6 Vicinity of the jamming transition
Near the jamming transition, the large-N generalization description fails because the network norm diverges, producing a cusp in test error. Finite learning times cut off this singularity.
- At N = N*, the test-error cusp is attributed to a divergence of ||fN||µ without regularization.The proposed scaling is ||fN|| ∼ (N − N*)−1.
- For N ≥ N*, the hinge-loss network can reach zero loss when every pattern satisfies yµf(xµ) > ϵm.The margin ϵm measures confidence above the classification threshold, while the unregularized norm leaves N* unaffected by its precise value.
- ||fN|| ∼ (N − N*)−1 is tested as an inverse power law near the transition.The transition location is defined asymptotically from the power-law behavior of ||fN,t|| as a function of N/N* − 1.
- Finite training times truncate the norm divergence and shift the apparent transition toward its t → ∞ value.The singularity becomes sharper as learning time increases.
- For cross-entropy, training slows without stopping, so the asymptotic norm is expected to diverge above N*, while finite-time singularities remain blurred.This differs from hinge loss, which stops in finite time in the over-parametrized regime.
7 Subleading Finite-Size Effect
Ensembling networks slightly above the jamming threshold can outperform using much wider individual networks under a fixed computational budget. The effect is linked to variance reduction and is reported as stronger for CNNs.
- Slightly over-parameterized networks with ensemble averaging are reported as more efficient than much larger networks under a fixed computational envelope.The ensemble can also achieve slightly better generalization than networks with N ≫ N*.
- Figure 7 shows that the network norm diverges just after jamming, while finite learning times cut off the divergence.The black reference line has slope −2, and N* is tuned to produce straight power-law curves.
- Figure 8 compares test and train error against the number of filters per convolutional layer using average and ensemble curves.The setup uses a three-convolutional-layer plus fully connected architecture trained on CIFAR10, with the transition at f ∈ {24, …, 28}.
- The ensembling benefit is described as stronger for CNN architectures.The paper presents this as support for the usefulness of controlling model size when averaging differently initialized networks.
8 Extension to Convolutional Networks
The framework is tested on a vanilla convolutional network trained on CIFAR10 while varying the number of channels per convolutional layer. Ensemble averaging improves performance near the jamming transition.
- The CIFAR10 experiment uses 50,000 training images, 10,000 test images, ten classes, and a three-convolutional-layer plus fully connected architecture.Each convolutional layer has f channels and the output is ten-dimensional.
- The number of channels varies from f = 2^1 to 2^11 across 20 independently initialized models, with learning rate 1/f.The jamming transition occurs just before f ∼ 2^8.
- 80.5–80.7% ensemble performance exceeds the approximately 77.5% average accuracy of the widest models.Peak performance is achieved by ensembling at f = 64, yielding approximately 80.7%.
9 Conclusion
The paper explains improving generalization with model width through reduced output fluctuations, while attributing the jamming cusp to a divergent unregularized network norm. It recommends ensembling networks slightly beyond the transition.
- ||fN − f̄N||µ ∼ N−1/4 leads to the predicted test-error scaling ϵN − ϵ∞ = C0N−1/2 + C1N−3/4 + O(N−1).The scaling is reported as consistent with MNIST observations.
- At N = N*, the unregularized hinge-loss network norm is argued to diverge as ||fN|| ∼ (N − N*)−α with α = 1.This predicted blow-up is used to explain the test-error spike at the transition.
- Near-optimal generalization is suggested by averaging networks of fixed size a few times larger than N*, rather than making one network much wider.The usefulness of averaging breaks down near N* because output variance is too large.
- The proposed ensemble strategy is identified as practically important but requiring tests across broader architectures and datasets.
A Materials and methods
The experiments use distinct initialization and optimization protocols for fully connected and convolutional networks, with several regularization-related choices omitted.
- Fully connected networks use random orthogonal weight initialization, zero biases, full-batch ADAM, and a learning rate min(10^-1h^-1.5, 10^-4).
- Convolutional networks use Xavier initialization, stochastic gradient descent, learning rate 1/f, batch size 250, and linear-hinge loss.
- Momentum, weight decay, and data augmentation are not used in the convolutional-network experiments.
B Robustness of the boundaries distance δ(x) estimate
The paper tests decision-boundary distance estimates and develops NTK-based finite-width analyses linking output fluctuations to training dynamics and generalization behavior.
- The estimate δ(x) = δf(x)/||∇f(x)|| for the distance between decision boundaries holds for ReLU networks and improves as N increases.
- For test points lying between fN = 0 and ¯fN = 0, the linear prediction of the boundary intersection is tested using 200 output-function curves.
- The typical boundary distance is estimated as δ ∼||δf||µ/||∇f||µ, with µ denoting the uniform measure over test points.
- The finite-size analysis is presented as a heuristic Central Limit Theorem counterpart to the NTK's large-width law-of-large-numbers behavior.
- At infinite width, training uses a deterministic NTK; positive definiteness yields a global minimum and exact training labels at convergence.
- Outside the training set, initialization-dependent terms partly cancel, while the remaining NTK kernel-regression term is deterministic.
- For finite width, initialization noise is O(N^-1/4), whereas NTK evolution is O(N^-1/2), making the initialization fluctuation dominant.
- Under hinge loss, training outputs remain random because initialization and NTK fluctuations affect the limiting margin-constrained values.