PaperlayerView exact revision on arXiv

Source-linked AI summary

A Dynamical Model of Neural Scaling Laws

Blake Bordelon, Alexander Atanasov, Cengiz Pehlevan

arXiv:2402.01092v4stat.MLcond-mat.dis-nncs.LG

TL;DR

Neural scaling laws describe regular performance changes with training time, dataset size, and model size, but their exponents and compute-optimal trade-offs remain important questions. The paper analyzes a solvable random-feature model and finds distinct time and model-size exponents, with compute-optimal training time increasing faster than model size. It also identifies finite-width and data-reuse effects, while noting that feature learning yields scaling improvements beyond the model’s linearized trends.

  • Problem

    Neural scaling laws are observed across tasks, but understanding their exponents and the optimal trade-off between model size and training time remains an open question.

    Source: §1 · p. 1

  • Method

    The paper analyzes a solvable linearized random-feature model that varies training time, model size, and dataset size.

    Source: §1 · p. 2

  • Results

    The model exhibits different scaling exponents for training time and model size, producing an asymmetric compute-optimal strategy in which training time scales faster than model size.

    Source: §1 · p. 3, §6 · p. 9

  • Takeaways & Limitations

    The results interpret compute-optimal scaling as competition between infinite-width, infinite-data training dynamics and a finite-model-size bottleneck.

    Source: §6 · p. 9

    The random-feature model captures linearized scaling trends but not the acceleration of scaling induced by feature learning.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude. This phenomenon is known as a neural scaling law. Of fundamental importance is the compute-optimal scaling law, which reports the performance as a function of units of compute when choosing model sizes optimally. We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. This reproduces many observations about neural scaling laws. First, our model makes a prediction about why the scaling of performance with training time and with model size have different power law exponents. Consequently, the theory predicts an asymmetric compute-optimal scaling rule where the number of training steps are increased faster than model parameters, consistent with recent empirical observations. Second, it has been observed that early in training, networks converge to their infinite-width dynamics at a rate $1/\textit{width}$ but at late time exhibit a rate $\textit{width}^{-c}$, where $c$ depends on the structure of the architecture and task. We show that our model exhibits this behavior. Lastly, our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.

1. Introduction

The introduction frames neural scaling laws as regular performance improvements with model size, training time, and compute, while emphasizing open questions about their exponents and optimal resource allocation. The paper develops a solvable model to explain these phenomena, including data-reuse effects and differences between training-time and model-size scaling.

1. Introduction

  • Data reuse can accumulate over training and worsen test performance, while finite-width performance gaps also grow over time and can reverse the “wider is better” pattern.The cited examples include overfitting on a limited dataset.Source: §1 · p. 2
  • Prior work finds early-time finite-width loss corrections scaling as 1/width, whereas long-trained finite-width models can have nontrivial, task-dependent width exponents.The introduction cites examples on CIFAR-5M and Wikitext.Source: §1 · p. 2
  • The paper develops a mathematically tractable model that varies training time, model size, and dataset size to analyze neural scaling laws.It focuses on linearized models for analytical tractability, while noting that this regime does not capture feature learning.Source: §1 · p. 2
  • The model predicts different power-law exponents for training time and model size, yielding an asymmetric compute-optimal strategy that scales training time faster than model size.The introduction also reports that ensembling offers less performance benefit than increasing model size.Source: §1 · p. 3

1.1. Related Works

  • Unlike prior analyses focused on final losses, this work studies training-time dependence and finds a different time exponent from the model-size and dataset-size exponents.Its infinite-time DMFT limit recovers final losses computed in those prior works.Source: §1.1 · p. 3
  • Other scaling-law hypotheses model a discrete collection of learned subtasks or skills, while this theory focuses on spectral components of a data distribution.Source: §1.1 · p. 3

2. Setup of the Model

The setup is a teacher-student problem in which a student random-feature model learns from data generated by a teacher. A mismatch between teacher and student is central to the phenomena studied.

2. Setup of the Model

  • A teacher generates the training data and a student random-feature model learns from it; their particular mismatch is identified as key to the paper’s phenomena.Source: §2 · p. 3
  • Power-law structure in the feature eigenvalues and target weights is used to produce power-law scaling in test loss and related quantities.Source: §2 · p. 4
  • The student projects base features through a random matrix A, modeling finite-width features as a lower-dimensional subspace related to infinite-width kernel features.The model interprets N as model size, with N → ∞ recovering the original kernel.Source: §2 · p. 4
  • The analysis assumes iid entries of A with zero mean and unit variance, a tractable choice that makes the student kernel approach the infinite-width kernel as N grows.The paper notes that the projection matrix is generally unknown in more realistic finite-width NTK settings.Source: §2 · p. 4

3. DMFT for Scaling Laws

The paper characterizes learning curves with a dynamical mean field theory (DMFT) for large systems. Correlation and response functions provide a closed description, and Fourier-domain analysis supports both exact response solutions and recovery of temporal behavior.

3. DMFT for Scaling Laws

  • DMFT is derived for large M, N, and P in proportional and non-proportional limits, with structurally identical theories in the two regimes.In the proportional regime, N/M = ν and P/M = α; the non-proportional regime takes M → ∞ first.Source: §3 · p. 4
  • Generalization can be limited when A⊤A or Ψ⊤Ψ is low rank, because dynamics are frozen in the corresponding nullspace.Source: §3 · p. 4
  • The concentrated correlation and response functions satisfy a closed set of integro-differential equations governing the learning dynamics.The response functions measure sensitivity to perturbations, while the correlation functions encode cross-time training and test errors.Source: §3 · p. 5
  • The response functions are time-translation invariant and can be solved exactly in the Fourier domain, yielding a transfer function for each kernel mode.The Fourier transform of the transfer function is expressed through the response functions R1 and R3.Source: §3 · p. 5
  • High frequencies describe early dynamics, low frequencies describe late-time behavior, and an inverse Fourier transform recovers the full temporal trajectory.Source: §3 · p. 5

4. Results

The results derive time-, data-, and model-limited loss scalings for power-law features, then show how finite data, online SGD, and ensembling affect those bottlenecks. The model predicts compute-optimal tradeoffs and explains how train–test gaps build during training.

Test loss power laws

  • Test loss scales as t^−(a−1)/b when time-limited, P^−min{a−1,2b} when data-limited, and N^−min{a−1,2b} when model-limited.For power-law features, eigenvalues and target coefficients decay as power laws; very easy tasks with a > 2b + 1 instead have N^−2b and P^−2b scaling.Source: §4 · p. 5

Connection to Online Learning with SGD

  • In online SGD, finite batch size adds variance to test loss but does not create an asymptotic plateau; the finite-N bottleneck remains.The same time- and model-size scalings hold, while batch-size dependence is weaker; the SGD noise scales as η/B.Source: §4 · p. 6, Figure 3 · p. 7

Asymmetric Compute Optimal Scaling Strategy

  • In the stated regime, rapid eigenspectrum decay favors allocating greater resources to training time rather than model size as compute increases; as b approaches 1, time and parameter count scale linearly together.Figure 4 verifies this scaling rule and its b-dependence.Source: §4 · p. 7

Wider is Better Requires Sufficient Data

  • With limited data, increasing N has diminishing test-loss returns; for N > P, training loss can keep decaying even after test loss saturates.For N < P, the model is underparameterized and cannot reach zero training loss; train and test losses gradually separate at a rate depending on P.Source: §4 · pp. 7–8, Figure 5 · p. 8

Gradual Buildup of Overfitting Effects

  • The train–test gap is O(1/P) early in training, develops task-dependent P scaling at late times, and takes longer to emerge for larger datasets.The model expresses the gap through DMFT order parameters and illustrates its gradual accumulation under finite-data reuse.Source: §4 · p. 8

Ensembling is Not Always Compute Optimal

  • Ensembling reduces initialization variance, but cannot recover the infinite-N, infinite-P learning curve because finite N and P constrain the dynamics' bias.The theory predicts ensembling's effect, while larger models can have lower bias than ensembles that only reduce variance.Source: §4 · p. 8

5. Tests on Realistic Networks

For linearized Wide ResNets on CIFAR-5M, the predicted compute-optimal scaling agrees with the observed scaling of the linearized models. Feature-learning networks show substantially different compute-optimal exponents, which the random-feature model does not capture.

5. Tests on Realistic Networks

  • The task-power and spectral-decay exponents are extracted from initial finite-width NTK spectra and the fraction of task captured by the top k eigenmodes.The unexplained task fraction is 1 − C(k); the width parameter is the number of hidden-layer channels.Source: §5 · p. 8
  • At fixed compute C = NEt, increasing model size is preferable to ensembling because it also reduces bias.Ensembling reduces initialization variance by a factor of E, but finite N and P leave a bias that prevents bagging from recovering the infinite-size, infinite-data learning curve.Source: Figure 6 · p. 9
  • The predicted compute-optimal scaling for linearized ResNets on CIFAR-5M agrees excellently with the linearized-model curves, which scale as L⋆(C) ∼ C^-0.05.The reported large-k power laws are λ_k ∼ k^-2.0 and k^-0.15.Source: §5 · p. 9, Figure 7 · p. 9

5.1. The Role of Feature Learning

  • Feature-learning networks have substantially different compute-optimal exponents, so the random-feature model captures linearized scaling trends but not feature-learning acceleration.The after-kernels continue to evolve substantially during training.Source: §5.1 · p. 9
  • A full explanation of compute-optimal exponents requires something resembling a mechanistic theory of kernel evolution.Source: §5.1 · p. 9

6. Conclusion

The paper presents a model that recovers many phenomena observed in deep learning and treats training dynamics alongside model and dataset size. It interprets compute-optimal scaling as competition between infinite-width, infinite-data training dynamics and finite-model bottlenecks.

6. Conclusion

  • The theory interprets compute-optimal scaling as competition between infinite-width, infinite-data training dynamics and finite model-size bottlenecks.It finds different performance-scaling exponents for model size and training steps.Source: §6 · p. 9

A. Derivation of Dynamical Model of Scaling Laws

The appendix derives a projected linear model's learning dynamics and their statistical description in a proportional asymptotic limit. It closes the response and correlation equations, solves the time-translation-invariant responses in Fourier space, and represents correlations using a double Fourier transform.

A. Derivation of Dynamical Model of Scaling Laws

  • The model uses a linear model with projected features, gradient-descent training on a random dataset with potentially noise-corrupted targets, and train/test losses computed from evolving fields.The dynamics are then described using dynamical mean field theory.Source: A. Derivation of Dynamical Model of… · p. 13

A.1. DMFT Equations for the Asymptotic Limit

  • In the joint large-M, N, P proportional limit, dynamical mean field theory gives correlation and response equations that are exact for any α and ν.Source: §A.1 · pp. 13–14

A.2. Closing the Equations for the Order Parameters

  • Closed integro-differential response equations determine the correlation functions, whose time-time diagonals yield the train and test losses.The response functions are causal, vanishing for t < s.Source: §A.2 · p. 14

A.3. Time-translation Invariant (TTI) Solution to Response Functions

  • Time-translation-invariant response functions are solved in Fourier space, and their transfer-function poles generate ν- and α-dependent convergence timescales.The solution is transformed back to obtain v0(t) and compute the test error.Source: §A.3 · p. 15

A.4. Fourier Representations for Correlation Functions

  • Because correlation functions are not time-translation invariant, they are represented with a double Fourier transform and solved as linear equations over frequency pairs.An inverse transform returns the time-domain correlations.Source: §A.4 · pp. 15–16

B. Field Theoretic Derivation of DMFT Equations

This section develops the DMFT and field-theoretic description using path-integral and cavity methods, examining long- and early-time limits and the evolving train–test loss gap. It also interprets finite-size dynamics through timescale densities, recovers a classical random-matrix result, and clarifies dimension-free limits, concentration, and bias and variance under the stated statistical assumptions.

B.1. Statistical Assumptions for DMFT

  • The MSR path-integral derivation assumes zero-mean Gaussian matrix entries, while the cavity derivation permits independent entries from any well-behaved distribution satisfying a central-limit condition.The stated Gaussian assumptions are sufficient rather than necessary; prior work also establishes universality for data matrices with bounded covariance spectral norm (Gerbelot et al., 2022).Source: §B.1 · p. 16

B.2. Path Integral Derivation

  • In the path-integral derivation, the large-M saddle point yields the dynamical equations, with response functions interpreted as the effect of a kick to another field’s dynamics.The derivation sets the auxiliary correlation fields to zero, enforcing vanishing response-field correlations.Source: §B.2 · p. 18, §B.2.1 · p. 19

B.3. Cavity Derivation

  • The cavity derivation adds a feature, sample, or projected feature in turn, using linear response to track perturbations and obtain the marginal statistics of added variables.Combining these dynamics recovers the DMFT equations of Appendix A.2.Source: §B.3 · pp. 19, 21

C. Final Losses (the t →∞Limit of DMFT)

  • The large-time and large-system limits commute for the loss, and the limiting test loss is given by the long-time limit of C0(t, s).The final-value theorem and response functions are used to extract limiting order parameters.Source: C. Final Losses (the t →∞Limit of DMFT) · p. 21

C.2. Learning Curves for Kernel Regression ν, t →∞

  • As ν and time tend to infinity, the theory recovers the kernel-regression learning curve with eigenvalues λk.For α > 1, κ = 0; the term 1/(1−γ) is attributed to variance from the randomly sampled dataset.Source: §C.2 · p. 23

D. Early Time Dynamics (High-Frequency Range)

  • Early-time corrections to the dynamics and correlations scale as 1/α and 1/ν, and build up over time.The analysis uses high-frequency expansions; it also states that finite data and model size harm performance in this account.Source: D. Early Time Dynamics (High-Frequency… · pp. 23–24

E. Buildup of Overfitting Effects

  • At early times, the test–train loss gap is exactly C0(t, t) − C1(t, t); its late-time behavior depends on parameterization, with both losses approaching limits when under-parameterized.The early-time gap equation follows from a high-frequency expansion of the correlation functions.Source: §E.1 · p. 24, §E.2 · p. 25

E.2. Low Frequency Range/Late Time

  • For over-parameterized models, asymptotic train loss is zero while test loss remains nonzero; under-parameterized models have limiting values for both losses.The under-parameterized limits are computed from the expressions in Appendix C.Source: §E.2 · p. 25

F. Timescale/Eigenvalue Density Interpretation

F.1. Recovering the Marchenko-Pastur Law from DMFT Response Functions

  • With λk = 1 and ν →∞, the DMFT response-function method recovers the bulk of the Marchenko-Pastur law.The derivation combines coupled equations and identifies the bulk through the imaginary part of the radical in the ϵ →0 limit.Source: §F.1 · pp. 25–26

G. Non-Proportional (Dimension-Free) Limit

  • For infinite-dimensional base features with a trace-class kernel, the dimension-free field equations match the proportional-limit equations after substituting ν →N and α →P.The derivation assumes the base kernel diagonalizes over the data distribution and uses the cavity argument to obtain the field description.Source: G. Non-Proportional (Dimension-Free)… · pp. 26–27

H.1. What Does/Doesn’t Concentrate in the DMFT Limit?

  • Response and correlation functions and train/test losses self-average, while state vectors and Gaussian sources remain realization-dependent, so infinite ensembling eliminates projection-matrix noise.Model outputs can therefore vary across datasets or initializations.Source: §H.1 · p. 27

H.2. Definition of Bias and Variance

  • Dataset and initialization variances are defined by averaging predictions over the other source of randomness, while their biases are the corresponding averaged predictors’ test errors.The irreducible bias is the test error of the predictor averaged over both datasets and initializations.Source: §H.2 · p. 27

H.3. Derivation

The derivation tracks correlations across ensembles and datasets to characterize bagging and ensembling. It finds that ensembling can be compute-inefficient, while the white bandlimited model can show late-time non-monotonic scaling that optimal early stopping avoids.

H.3. Derivation

  • The ensemble-and-bagging loss contains irreducible bias, a term scaling as 1/(νE), a term scaling as 1/(αB), and terms coupling B and E.The 1/(νE) and 1/(αB) terms persist when the other resource grows without bound.Source: §H.3 · p. 29

H.4. Ensembling is Not Always Compute Optimal

  • Ensembling is strictly dominated by increasing model size ν at fixed compute in the analyzed setting.Ensembling reduces variance, but both bias and variance decrease with ν.Source: §H.4 · p. 30

I. White Bandlimited Model

  • For the white bandlimited model with α < 1, loss can be non-monotonic in ν late in training, but optimal early stopping restores monotonic scaling.The figure describes the late-time non-monotonicity as double descent.Source: §I · p. 30

I.1. Derivation

The white bandlimited analysis identifies finite-width timescale corrections, while the power-law-feature analysis separates time and model bottlenecks. In the model-bottleneck regime, difficult tasks have loss scaling as N^-(a−1).

I.1. Derivation

  • With λk = 1, the response equations simplify because all M features are statistically identical.This is the simpler white bandlimited case.Source: §I.1 · p. 30

I.2. Timescale Corrections in The Small ν Regime

  • At finite ν in the small-ν regime, the transfer function exceeds its ν → ∞ value e^-τ in the perturbative approximation, implying higher bias.The result follows from the approximate response function.Source: §I.2 · p. 31

I.3. Timescale corrections in fully expressive regime ν > 1

  • For large but finite ν, the perturbative approximation captures a slowdown in the white bandlimited dynamics.The slowdown is shown in Figure 11.Source: §I.3 · p. 32, Figure 11 · p. 32

J.1. Time Bottleneck

  • In the time-bottleneck limit α, ν → ∞, the loss scaling with time follows from the limiting loss expression.The scaling can be obtained by change of variables or steepest descent.Source: §J.1 · p. 32

J.2. Model Bottleneck

  • For difficult tasks where a − 1 < 2b, the expected loss follows a power-law scaling of the form L ∼ N^−(a−1) in this regime.Source: §J.2 · p. 33

J.3. Data Bottleneck

The data-bottleneck regime yields a power-law loss in dataset size, while the appendices extend the DMFT treatment to discrete-time optimization and one-pass SGD. In one-pass SGD, finite batch size adds SGD noise rather than a data-bottleneck asymptote.

J.3. Data Bottleneck

  • For difficult tasks with a < 2b + 1, the loss scales as P^−(a−1) in this data-bottleneck regime.Source: §J.3 · p. 33

K.1. Discrete Time

  • Discrete-time DMFT equations can be closed and solved over T × T matrices, or in frequency space with a Z-transform.Final losses are extracted in the z → 1 limit.Source: §K.1 · p. 34

K.2. Momentum

  • Momentum modifies the response functions, and discrete-time momentum updates can be rewritten as a second-order difference equation.The update uses a filtered loss gradient and momentum coefficient.Source: §K.2 · pp. 34–35

K.3. One Pass SGD

  • One-pass SGD has no train–test loss gap; finite batch size adds variance, while test loss remains limited by training time or model size.Unlike data reuse, one-pass SGD has no data-bottleneck regime; its batch-size dependence is weaker and its noise scales as η/B.Source: §K.3 · p. 35

L. Kernel Analysis of Feature Learning Networks

For feature-learning networks on animate-inanimate CIFAR-5m, the observed test-loss scaling is better than the after-kernel predicts, while kernel-target alignment increases during training. The appendix also describes iterative DMFT computation and a Fourier-transform procedure for stabilizing a singular integral.

L. Kernel Analysis of Feature Learning Networks

M.1. Iteration of DMFT Equations on Time × Time matrices

  • The DMFT response-function equations are solved as time-by-time matrices by iteration from an initial condition.The converged response functions are then used to iterate the correlation-function equations and obtain observables.Source: §M.1 · p. 37, §2 · p. 37

M.2. Fourier Transform Method

  • In the model/data bottleneck regime, the Fourier method separates a singular contribution from a shifted, non-divergent integrand for numerical integration.The separated Dirac-mass term is integrated directly, and the remaining integral is computed numerically for greater stability.Source: §M.2 · pp. 37–38

N. Compute Optimal Scaling from Sum of Power-Laws

The section minimizes a loss expression under fixed compute, reducing the problem to one dimension and evaluating loss at the optimum.

N. Compute Optimal Scaling from Sum of Power-Laws

Loading 2402.01092v4…