PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining Neural Scaling Laws

Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, Utkarsh Sharma

arXiv:2102.06701v2cs.LGcond-mat.dis-nnstat.ML

TL;DR

The paper asks how neural-network scaling laws arise and relate across dataset and model size. It develops a framework connecting variance- and resolution-limited regimes to data-manifold and kernel spectra, then tests its predictions empirically.

  • Problem

    The paper investigates how kernel eigenvalue spectra relate to data-manifold dimension and explain neural-network scaling exponents.

  • Method

    The paper categorizes four scaling regimes and derives their behavior using asymptotic expansions, data-manifold assumptions, and kernel spectra.

  • Results

    The predictions agree with deep networks on real datasets; variance-limited regimes give αD = 1 and αW = 1, while resolution-limited exponents depend on data and model.

  • Takeaways & Limitations

    The framework distinguishes mechanisms behind loss improvements and relates scaling exponents to dataset, architecture, noise, and task changes.

  • Takeaways & Limitations

    The theory is asymptotic, requires dataset–model size hierarchies, and relies on an imprecisely defined data manifold for deep networks.

Abstract

from arXiv · show

The population loss of trained deep neural networks often follows precise power-law scaling relations with either the size of the training dataset or the number of parameters in the network. We propose a theory that explains the origins of and connects these scaling laws. We identify variance-limited and resolution-limited scaling behavior for both dataset and model size, for a total of four scaling regimes. The variance-limited scaling follows simply from the existence of a well-behaved infinite data or infinite width limit, while the resolution-limited regime can be explained by positing that models are effectively resolving a smooth data manifold. In the large width limit, this can be equivalently obtained from the spectrum of certain kernels, and we present evidence that large width and large dataset resolution-limited scaling exponents are related by a duality. We exhibit all four scaling regimes in the controlled setting of large random feature and pretrained models and test the predictions empirically on a range of standard architectures and datasets. We also observe several empirical relationships between datasets and scaling exponents under modifications of task and architecture aspect ratio. Our work provides a taxonomy for classifying different scaling regimes, underscores that there can be different mechanisms driving improvements in loss, and lends insight into the microscopic origins of and relationships between scaling exponents.

1 Scaling laws for neural networks

The paper develops a taxonomy of four neural-network scaling regimes, distinguishing variance-limited behavior from resolution-limited behavior and connecting the latter to data-manifold geometry and kernel spectra. It derives predictions in random-feature models and finds empirical support in deep, pretrained, and fine-tuned models while identifying dataset- and architecture-dependent exponent changes.

  • Variance-limited regime: In the variance-limited regime, smooth test losses have universal exponents αD = 1 for D ≫P and αW = 1 for w ≫D.These predictions arise when one parameter approaches an infinite-data or infinite-width limit.
  • Contributions: The framework distinguishes four scaling regimes formed by variance-limited and resolution-limited behavior in dataset-size and model-size scaling.The two mechanisms are presented as distinct origins of neural-network scaling laws.
  • Resolution-limited regime: Resolution-limited scaling models additional data or parameters as refining a smooth data manifold, yielding exponents proportional to 1/d.The paper tests this dependence on intrinsic dimension in deep networks on standard datasets.
  • Kernel and random-feature theory: Random-feature teacher-student models yield exact variance- and resolution-limited exponents, with the latter related to kernel spectral decay and a duality between model- and dataset-size scaling.For continuous kernels on d-dimensional manifolds, the spectrum connects kernel behavior to manifold dimension.
  • Empirical evaluation: The predictions show excellent agreement in pretrained and fine-tuned models and support all four regimes in deep networks on standard datasets.The study also reports strong exponent effects from changing input distributions or adding noise, but not from superclassing the target task.

2 Four scaling regimes

The paper distinguishes variance-limited scaling, arising from well-behaved infinite-data or infinite-width limits, from resolution-limited scaling, arising when models resolve smooth data manifolds. Together these mechanisms produce four dataset- and model-size scaling regimes, with random-feature models exhibiting explicit power laws and a duality between dataset and parameter scaling.

  • Resolution-limited scaling: In the resolution-limited regime, nearest-neighbor distances on a d-dimensional manifold scale as O(D^-1/d), yielding loss scaling L ∝ D^-n/d when the leading nonzero loss term has order n ≥ 2.For accurate piecewise-linear approximations, the leading order is generically n ≥ 4.
  • Four regimes: Random-feature models explicitly exhibit all four regimes, including variance-limited behavior in one asymptotic direction and resolution-limited behavior in the other.The construction uses linear teacher-student models trained to convergence and demonstrates nonzero loss when teacher and student feature spans mismatch.
  • Variance-limited scaling: Variance-limited loss corrections scale as O(D^-1) with dataset size or O(P^-1) with parameter count under concentration and smooth-loss conditions.Theorem 1 applies when trained outputs concentrate; the stated conditions include finite-degree polynomial losses, bounded second derivatives, and 2-Hölder losses.
  • Spectral duality: When 1 ≪ P ≪ D or 1 ≪ D ≪ P, power-law kernel spectra control scaling, with αP = αD = αK empirically supported for pooled MNIST.For sufficiently smooth kernels on a d-dimensional manifold, the spectral exponent satisfies αK ∝ d^-1, connecting kernel spectra to manifold resolution.
  • Spectral duality: The underparameterized and overparameterized losses are directly related by exchanging projections onto random features and random training points.This establishes a stronger model–dataset duality in the linear setup than merely sharing a common asymptotic exponent.

3 Experiments

Experiments test the proposed scaling regimes across teacher-student models, standard datasets, and architecture variations. They find universal variance-limited exponents, dataset- and model-dependent resolution-limited behavior, and systematic effects from task distribution and aspect ratio.

  • Teacher-student models: Teacher-student experiments scan model size, dataset size, and input-manifold dimension to test the predicted interpolation exponents.The input dimension is controllable through the teacher network, allowing direct variation of the data-manifold dimension.
  • Variance-limited scaling: αD, αW = 1 across datasets, architectures, batch sizes, and loss types in the variance-limited regime.The agreement is reported for deep fully connected and convolutional networks with ReLU or Erf nonlinearities.
  • Resolution-limited scaling: Resolution-limited scaling varies across standard classification datasets and depends on the data and model setting.The experiments include Wide ResNet models and a four-hidden-layer convolutional network.
  • Task and data distribution: αD is insensitive to the number of target categories but changes more substantially when Gaussian input noise is added.The result supports a picture in which the network models the input data manifold largely independently of the classification task.
  • Architecture aspect ratio: The dataset-scaling exponent magnitude increases with width up to a critical width, while its dependence on depth is milder.This is an empirical aspect-ratio effect on dataset scaling.

4 Discussion

The discussion presents the framework as a taxonomy of four scaling regimes with distinct mechanisms. It reports agreement with deep networks and real datasets while identifying asymptotic and finite-width limitations.

  • Framework: The paper categorizes four neural-network scaling regimes and identifies different origins for variance-limited and resolution-limited behavior.It provides empirical support for all four regimes in deep networks on standard datasets.
  • Variance-limited regime: αD = 1 for D ≫P and αW = 1 for w ≫D in the variance-scaling regime for smooth test losses.These are universal predictions under the stated asymptotic hierarchies.
  • Resolution-limited regime: Resolution-limited exponents have numerical values that depend on the data and model.Unlike variance-limited exponents, they are not presented as universal constants.
  • Future theory: Finite-depth and finite-width network kernels evolve during training, so deriving feature-learning effects remains an open direction beyond the random-feature correspondence.The random-feature models are in exact correspondence with deep networks only in the large-width limit.
  • Limitations: The theory is asymptotic, whereas experiments use finite models and datasets, causing predicted scaling to break down when the required hierarchy is lost.The resolution-limited analysis also relies on positing a data manifold whose precise definition is lacking.

5 Outlook

The outlook frames theory grounded in realistic experiments as a way to understand learning in modern deep networks. It highlights questions about emergent behavior and representation learning as directions for further study.

  • Scientific motivation: A theory connecting mathematically rigorous analysis with realistic experiments could help guide understanding of deep learning.The paper motivates this approach in the context of large datasets, models, and computational power.
  • Emergent behavior: Whether qualitatively new behavior emerges in large neural models trained on rich datasets remains an open question.The discussion specifically points to emergent abilities in neural language models.
  • Future methodology: Physics-inspired theoretical methods may help investigate learning in deep networks when paired with realistic models of data and neural representations.The proposed direction is motivated by the need to study large-model behavior without separating theory from experimental realism.

Supplemental Material

The supplemental experiments specify the model, optimization, dataset, and evaluation settings used to test scaling behavior. They also compare alternative metrics and stopping conditions.

  • Experimental setup: Most experiments use small three-layer, width-8 fully connected networks with P ∼O(1000), full-batch gradient descent, and small learning rates.The setup is based on the Neural Tangents library, with convolutional models used where indicated.
  • Wide ResNet protocol: Wide ResNet 28-10 models are trained for 78125 steps with cosine decay and random flips and crops.Reported metrics are evaluated after this training protocol.
  • Width variation: A depth-10 residual network is used to examine width variation while keeping the setup otherwise identical to Figure 1 top-right.This isolates width effects within the corresponding scaling experiment.
  • Width-scaling experiments: Width-scaling experiments use two-hidden-layer fully connected networks ranging from w = 64 to w = 11,585 with ReLU nonlinearities unless specified otherwise.They use 100 training samples, full-batch gradient descent, and cross-entropy unless MSE is specified.
  • Teacher-student setup: The teacher-student experiments use teacher width 96, student width 192, and input dimensions d = 2, 3, · · ·, 9.Eight random seeds and a fixed test dataset are used to reduce noise while scanning dataset sizes.
  • Metric robustness: Loss and error show similar scaling behavior, as do final metrics and best early-stopped metrics.This comparison is summarized in Figure S1.

A.6 CNN architecture for resolution-limited scaling

CNN experiments on standard image datasets used specified training schedules and nearest-neighbor estimates of input-space dimensionality to study dataset scaling.

  • Experimental setup: CNNs were trained on CIFAR-10, MNIST, Fashion MNIST, CIFAR-100, and SVHN using dataset-specific epoch counts.Training lasted 50 epochs for CIFAR-10, MNIST, and Fashion MNIST; 100 for CIFAR-100; and 10 for SVHN.
  • Scaling measurements: Figure S2 reports MSE-loss scaling with dataset size for teacher/student models, while extracted exponents and input-space dimensionalities appear in Figure 1b.
  • Experimental setup: The CNN architecture configurations for these datasets are summarized in Table 1.
  • Dimensionality estimation: Input-space dimensionality was estimated with nearest-neighbor methods using 2, 3, and 4 neighbors, then averaged.

A.7 Teacher-student experiment for scaling of loss with model size

The paper examines model-size scaling in teacher-student networks and aspect-ratio effects in Wide ResNets, alongside dataset-scaling measurements on image datasets.

  • Teacher-student model-size scaling: The teacher-student setup was replicated to demonstrate scaling of loss with model size.
  • Teacher-student model-size scaling: The variation of −4/αP with input-space dimensionality was measured using a fixed randomly generated teacher and 15 log-space iterations.
  • Aspect-ratio effects: For Wide ResNets trained on CIFAR-10, depth had a mild effect on dataset scaling, whereas width affected scaling until a saturating width.
  • Image-dataset scaling: Figure S3 shows cross-entropy-loss scaling with dataset size for various image datasets, with exponents and input-space dimensionalities reported in Figure 1b.
  • Teacher-student model-size scaling: Figure S4 shows αP, the model-size loss-scaling exponent, as a function of input-space dimension for the teacher-student setup.

B Proof of Theorem 1

Theorem 1 bounds the difference between expected loss at a random trained output and loss at its mean under concentration and regularity conditions on the loss.

  • Theorem statement: Theorem 1 assumes concentrated network outputs, with all moments of deviations from their mean scaling as O(ϵ).
  • Theorem statement: Under finite-degree polynomial, bounded-second-derivative, or 2-Hölder losses, E[ℓ(fT)] − ℓ(E[fT]) = O(ϵ).
  • Proof strategy: The proof treats the three loss classes separately, using polynomial expansions, the quadratic mean value theorem, and the 2-Hölder condition.
  • Proof strategy: The initial inequality bounds the loss difference by Kℓ(fT − E[fT])², after which expectation completes the relevant argument.
  • Typical-instance implication: For typical loss instances, the assumptions alone generally yield O(√ϵ), but an average local-minimum condition restores O(ϵ) scaling.

C Variance-limited dataset scaling

Variance-limited dataset scaling arises when finite-data training and evaluation remain sufficiently smooth, so concentration around the infinite-data limit propagates through optimization.

  • Intuition and assumptions: For smooth losses and training procedures, finite-data test loss approaches its infinite-data limit with variance-limited power-law behavior.
  • Violations: The scaling can fail for non-smooth losses, unbounded losses, or unbounded first derivatives, including examples with behavior changing at times larger than dataset size.
  • Intuition and assumptions: Training-loss concentration around population loss propagates through a sufficiently smooth optimization map to the final weights and population loss.
  • Finite-time training: For a fixed number of gradient-descent steps, repeated concentration arguments imply the same finite-data scaling for the resulting test loss.
  • Late-time training: Near a local population minimum, the corresponding training minimum lies at distance O(·), and smoothness yields an O(D^-1) expected loss difference.
  • Mini-batch training: Mini-batch averaging can be reformulated as a smooth function of empirical training-set means, allowing the concentration argument to be reused.
  • Mini-batch training: Without shuffling, if D > BT for T steps and batch size B, the loss is constant for all larger D, making the asymptotic difference zero.
  • Polynomial analysis: Polynomial networks and losses reduce the analysis to concentration of polynomials and monomials in empirical sample means.

C.3 Non-smooth examples

The section presents two examples where nonsmooth or unbounded losses violate variance-limited scaling, through late-time divergence or singular gradients.

  • Scope: Two worked examples show that non-bounded or non-smooth losses can violate variance-dominated scaling.The first changes behavior after training times exceed dataset size; the second violates the scaling after two gradient-descent steps.
  • Example 1: For α > 1/2, the training-loss minimum may occur at zero or ±∞, so late training need not concentrate around the population-loss minimum.The outcome depends on whether the sample contains more or fewer than D/2 points of the first variety.
  • Example 1: A mismatch larger than O requires training times T larger than a constant multiple of D, separating late-time behavior from the usual finite-data limit.In this example, the minimum of the infinite-data test-loss limit differs from the population-loss minimum, and one diverges while the other remains finite.
  • Example 2: For 0 < α < 1, a singular gradient at the origin produces a scaling exponent different from the variance-limited prediction.The example trains a model initialized at θ0 = 1 for two gradient-descent steps with learning rate 1.
  • Example 2: The second example therefore achieves different scaling through a diverging gradient.Its test loss is the population loss evaluated at θ2 and averaged over test-set draws.

D Proof of Theorems 2 and 3

The proofs connect resolution-limited scaling to nearest-neighbor distances on smooth data manifolds and derive model-size scaling through a dataset–parameter duality. The random-feature analysis further expresses test loss through feature covariance and kernel spectra.

  • Proof strategy: Nearest-neighbor distances for D samples on a d-dimensional manifold provide the key geometric input to the proofs.The argument averages nearest-neighbor distances over datapoints and independent dataset draws.
  • Theorems 2 and 3: Under Lipschitz assumptions and exact interpolation on training data, Theorem 2 bounds dataset-dependent test loss using manifold resolution.Theorem 3 applies the same reasoning when interpolation holds at P sampled points, yielding model-size scaling.
  • Random-feature setup: The random-feature calculation averages over i.i.d. training datasets and input points, with feature and data second moments determining the loss expressions.The formulas apply to both parameter regimes, using Moore–Penrose pseudoinverses when necessary.
  • Kernel analysis: In the large-width setting, test-loss scaling follows from the asymptotic decay of the feature-covariance or kernel spectrum.For generic continuous kernels on a d-dimensional manifold, the spectral description can be related to manifold dimension.
  • Duality: Finite-feature and finite-dataset expressions are related by exchanging projections, yielding a general dataset–parameter duality in linear feature models.The resulting parameter-scaling behavior follows from the dataset-scaling analysis through this duality.

F Learned features

The learned-feature experiments use pretrained EfficientNet-B5 embeddings with linear classifiers to vary feature dimension and dataset size. They observe all four scaling regimes and a strong dataset–feature-size duality, while target and feature learning affect exponents.

  • Experimental setup: Pretrained EfficientNet-B5 penultimate-layer embeddings are resized from ImageNet inputs for CIFAR-10, then used to train a linear classifier.Feature dimensions and training-set sizes are randomly subsampled, averaged over five seeds, and evaluated with linear-kernel ridge regression.
  • Scaling regimes: All four scaling regimes appear for both fixed low regularization and optimal regularization in pretrained embedding features.The experiments use real CIFAR-10 labels.
  • Duality: Dataset-size and feature-size losses track each other for both small and tuned regularization, supporting the dataset–feature-number duality.The low-regularization case corresponds to long training time and exhibits double descent, whereas tuned regularization corresponds to optimal early stopping.
  • Targets and features: Teacher–student targets and fully trained real-data models have significantly larger exponents than fixed-feature models with realistic targets.The comparison includes random features at infinite width and learned features from trained WRN 28-10 models.
  • Target coefficients: For real targets, approximately constant feature-label coefficients occur only for the well-performing Myrtle-10 and trained WRN features.Teacher–student settings show approximately constant coefficients across feature index in all cases.
Loading 2102.06701v2…