A Solvable Model of Neural Scaling Laws

1 Introduction

• The motivating gap is an explanation of why performance scales predictably with parameters, data, or compute until another resource bottlenecks, and when that behavior breaks down.The questions include which datasets and model classes support scaling laws and what mechanism produces them.Source: §1 · p. 3
• The paper jointly models generative data and random features, then solves for test loss as a function of dataset size and parameter count.Its random-matrix-theory approach uses diagrammatic methods, with planar diagrams capturing dominant contributions in the jointly large regime.Source: §1 · pp. 4–6
• Power-law spectral statistics yield power-law test-loss scaling without resource bottlenecks, while nonlinear feature maps extend the representation spectrum’s power law.The authors also identify exact equiparameterization as optimal for regularized generalized linear models; this may not apply in the same way to models that learn representations.Source: §1 · p. 4
• A latent space much larger than both training-set size and parameter count is required for the familiar scaling regime; exceeding that scale produces different test-loss behavior.The authors note that this new regime has not yet been seen in LLM experiments, and that the corresponding scale in natural data remains an open question.Source: §1 · pp. 4–5
• The analysis uses exact optimization and therefore effectively assumes infinite compute, so it addresses tradeoffs between data and parameter resources only.The paper contrasts this jointly large-resource regime with classical underparameterized and modern overparameterized settings.Source: §1 · pp. 3–4
• Earlier theory often imposed an infinite-extent data power law or omitted random feature maps; this work keeps latent-space, training-set, and feature counts finite.It also extends prior random-feature test-loss analysis to student-teacher models and averages over a generative data model.Source: §1 · p. 5

2 Prerequisites for Neural Scaling

• The [18] phenomenological law predicts power-law loss improvements when jointly scaling parameters and training data, while a limited resource causes a plateau.The joint scaling balances the two resources; gains from increasing one resource alone stop when the other becomes the bottleneck.Source: Figure 1 · p. 7, §2 · pp. 8–9

2.1 Data Properties

• Natural image and embedded-language datasets show approximate spectral power laws across orders of magnitude, ending in a rapid tail; more samples or input features extend the power-law range.The rapid decline occurs near T when T < Nin and near Nin when T > Nin.Source: §2.1 · pp. 11–12

2.2 Feature Map Properties

• A linear map only adds zero eigenvalues, whereas nonlinear ReLU features extend the spectral power-law range when T > Nin, with useful-feature extent limited by min(N,T).This extension effect is specific to natural datasets with the properties described in §2.1.Source: §2.2 · pp. 14–15

Aside: Random Feature Maps vs. DNNs

• DNNs differ from random-feature models through components such as self-attention and learned representations; the authors identify the NTK as the proper way to define network features.For the single-layer ReLU random-feature map, the model has N × Nin parameters but N features.Source: Aside: Random Feature Maps vs. DNNs · pp. 15–16
• For infinite-width architectures, gradient-trained DNNs correspond to generalized linear models with NTK features defined by output derivatives, and their spectra show similar phenomenology to the paper’s nonlinear feature map.The NTK provides a feature for every parameter, so increasing parameter count increases the effective number of accessible features.Source: Aside: Random Feature Maps vs. DNNs · p. 16
• Away from infinite width, finite-width networks learn nontrivial input representations, yet the model’s power-law and plateau behavior suggests feature learning may not be essential to scaling laws.Perturbatively, the output still depends on the NTK with a parameter-number of features [16].Source: Aside: Random Feature Maps vs. DNNs · p. 16
• The authors conjecture that better DNN architectures preserve power-law structure when transforming dataset spectra, and leave relating model performance to feature spectra for future work.They characterize this as “power-law in, power-law out.”Source: Aside: Random Feature Maps vs. DNNs · pp. 16–17

• The model combines generative data and random features whose spectrum has an approximate power-law region controlled by the smaller of feature count N and training-set size T.Source: §3 · p. 17
• The model uses those features in generalized linear regression so that test loss can exhibit the targeted scaling-law behavior.Source: §3 · p. 17
• The authors aim to compute averaged test loss analytically with random matrix theory and diagrammatic techniques for sufficiently large N and T.The section also verifies model properties with simulations and outlines how the methods could model spectral extension in nonlinear feature maps.Source: §3 · p. 17

Generative Data Model

• In the latent-data generative model, for T < M the power-law extent increases with dataset size, whereas for T ≥ M it remains fixed as larger datasets sharpen the rapid decline toward zero.Source: Generative Data Model · p. 19

Random Feature Model

• The random feature map produces an approximate power-law spectrum whose extent is controlled by min(N, T), as intended by the model.The model uses a linear map from a larger latent space to a smaller feature space, simplifying analysis relative to directly modeling a nonlinear map.Source: Random Feature Model · pp. 20–21, Figure 6 · p. 21

(Generalized) Linear Regression

• A generalized linear model learns a transformation of the random features to predict labels generated from latent inputs, fitting parameters by ridge-regularized mean squared error.Training labels can include Gaussian noise, and the fitted parameters have a standard closed-form solution.Source: (Generalized) Linear Regression · pp. 21–22

Computing Performance

• Test performance is measured by mean squared error on a separately sampled test set, using the converged regression solution and optimizing the ridge parameter.The resulting simulations compare test loss across training-set and feature counts, selecting γ = γ⋆ for best performance.Source: Computing Performance · pp. 23–24
• Optimally regularized simulations reproduce power-law test-loss scaling with training-set size until a parameter-limited plateau, matching the reported LLM phenomenology.When parameters are not the bottleneck, L(N, T) ∼ T^-α; when they are, loss plateaus at a value scaling as N^-α. The analysis assumes converged training and therefore does not capture compute-limited scaling laws.Source: Computing Performance · pp. 23–24, Figure 7 · p. 25

Average Goals

• The analytical goal is the expected test loss averaged over latent inputs, labels, noise, and feature weights, a random-matrix calculation.Averaging separates label and noise contributions; the feature functions are centered but have nontrivial covariances, including cross-correlations between latent inputs and features.Source: Average Goals · pp. 24, 26
• Independent train and test samples have no cross-correlations between latent or random features, while mixed latent–random-feature covariances vanish when averaged over feature weights.These vanishing covariances simplify the data averages used in the analysis.Source: Average Goals · p. 27

Final Definitions

• Expanding the feature resolvent q(γ) around γ →∞ and using data-averaged covariances yields an implicit equation that determines averaged test loss.The equation can be solved in certain limits and averaged over random features.Source: Final Definitions · p. 27
• For N > T, the analysis rewrites the regression solution using a data-data resolvent Q, whose identities simplify averaging by eliminating resolvent and feature-matrix factors.The resolvent relations also enable rewriting the linear regression solution in terms of Q.Source: Final Definitions · pp. 27–28
• The authors distinguish their averaging approach from Ref. [38] and note that models without random feature maps behave qualitatively differently from the LLMs observed in Ref. [18].The comparison concerns models with the right generative data model but no random feature maps.Source: Final Definitions · p. 27

3.2 Data and Feature Averaging

• The averaging calculation is given in the limit M, T, N ≫ 1 with their ratios unrestricted, where the authors say its solutions are accurate for large datasets and models.Subleading corrections are suppressed by inverse powers of M, T, and N and matter mainly when T or N is ≲10.Source: §3.2 · p. 28

3.2.1 The Noise Term

• At large T and N with fixed ratio, planar diagrams dominate, while diagrams of more complex topology are suppressed by powers of N.The argument assumes feature-covariance entries remain order one as N grows, as in the paper’s data models.Source: §3.2.1 · p. 32
• Reorganizing the averaged resolvent through irreducible diagrams makes its computation tractable in the planar limit, though the resulting coupled equations generally lack a simple closed-form solution.The coupled equations determine the averaged resolvent and the quantity needed for the noise term; they can be solved numerically or analyzed in the ridgeless limit.Source: §3.2.1 · pp. 34–35
• In the ridgeless limit, the noise term has a simple expression determined by solving a self-consistent equation.The solution depends on whether T>N or T<N and on the data- and feature-size regime; the paper relegates its case-by-case calculation to Appendix B.2.Source: §3.2.1 · p. 35
• The noise-term expansion is motivated by a possible 1/γ divergence as γ→0 when ∆F becomes large.The subsection then introduces a final expression for the noise term, but its formula is not legible in the supplied passage.Source: §3.2.1 · pp. 35–36

3.2.2 The Label Term

• The label-term result is α + 1/(N/T − 1) when T < N and N < M, α + 1/(M/T − 1) when T < M and N > M, and 1/(T/M − 1) when T > M and N > M.Source: §3.2.2 · p. 36

The Underparameterized Regime

• In the underparameterized regime, the label-term average uses planar diagrams for Gaussian latent and random features, then reorganizes contributions into connected and factorized terms.The method assumes M, N, and T grow large at fixed ratios; the random-feature average is then recast using machinery from the noise-term calculation.Source: The Underparameterized Regime · pp. 36, 40, 44
• In the under-parameterized ridgeless limit, the leading averaged-resolvent term is independent of ζ, allowing ζ = 1 and removing σ²_u from the label-term solution.This simplification applies for N < T and N < M.Source: The Underparameterized Regime · p. 47

The Over-Parameterized Regime

• In the over-parameterized regime, a training-data–random-feature duality maps the calculation to the under-parameterized one, exchanging N and T in the final answer.The exact duality is broken in the presence of label noise.Source: The Over-Parameterized Regime · p. 48, §3.2.3 · p. 52

3.2.3 The Result

• For N,T < M, the label term is (1 − N/T)^−1 Δ^−1(N, M) when N < T and (1 − T/N)^−1 Δ^−1(T, M) when N > T.The noiseless test loss is symmetric under exchanging N and T, and the result holds for general latent covariance Λ.Source: §3.2.3 · p. 50
• Δ^−1(N, M) measures latent covariance missed by too few features, while Δ^−1(T, M) measures covariance missed by too few samples.Which limitation dominates depends on whether N < T or N > T.Source: §3.2.3 · p. 51
• At N ≈ T, the ridgeless label term has a spectrum-independent noise-like divergence from overfitting random-feature noise, which disappears under optimal regularization.This divergence is a feature of the ridgeless limit.Source: §3.2.3 · p. 51
• The power-law spectrum yields test-loss power laws as T or N and M grow, followed by a plateau when the scaled resource exceeds the fixed parameter.Figure 8 simulations closely fit the ridgeless RMT result and show the T ↔ N symmetry.Source: §3.2.3 · pp. 51–53, Figure 8 · p. 53

3.3 (Towards) Modeling Spectral Extension

• Nonlinear maps from smaller input spaces offer a more realistic alternative to the model’s linear transformation from a large latent space.The authors frame their analysis as a step toward understanding how these maps extend the input spectrum’s power law.Source: §3.3 · p. 54
• The useful-feature count grows with the smaller of training-set size T and feature count N, matching the model’s power-law extent and linking nonlinear spectral extension to scaling behavior.The authors describe useful features as decoded from a nonlinear transformation of input features.Source: §3.3 · p. 54
• With sufficiently large input dimension, random features are approximately Gaussian, allowing the random-matrix techniques to analyze the nonlinear model.Adding a layer or using preactivations can remove the large-input-dimension requirement.Source: §3.3 · p. 54
• For Gaussian features, the spectrum is determined by the data-data covariance Σ, while linearizing the kernel can miss the small-eigenvalue spectral density relevant to power-law extension.The kernel is explicitly calculable for many single-hidden-layer activations, including ReLU.Source: §3.3 · pp. 54–55
• A rigorous derivation of how a nonlinear map extends the power law for a given data model or natural dataset remains an open calculation.The proposed route is to combine the infinite-feature kernel spectrum with the resolvent for the finite-feature kernel.Source: §3.3 · p. 55

3.4 Comparing to Other Methods

• The diagrammatic derivation shares an intermediate result with Ref. [34] and can be used with either random-teacher averaging or substituted covariance matrices.The latter substitutions recover the result in Ref. [34], up to trivial constants.Source: §3.4 · p. 56
• The paper’s novel result is the test loss averaged jointly over random features and training data, alongside a dual expression for averaging over training samples.The authors present this as a generalization of Ref. [34].Source: §3.4 · p. 56
• Feynman diagrams are straightforward and can accommodate subleading 1/N corrections, but are not readily suited to expectations of non-analytic functions such as logarithms.Replica methods can be used for such quantities, though analytic continuation is unique only for certain constrained distributions and quantities.Source: §3.4 · p. 57
• Loop equations and orthogonal polynomials can handle more complicated non-Gaussian distributions, but are typically unnecessary for Gaussian matrix expectations.The authors note that, to their knowledge, these techniques had not yet been used in a machine-learning context.Source: §3.4 · p. 57

4 Discussion of Results

• The discussion analyzes the model for jointly large-but-finite model size N, training-set size T, and latent-space size M.The section interprets what the model implies about scaling laws.Source: §4 · p. 57

4.1 Breakdown of Scaling Laws

• Neural scaling-law behavior breaks down as model size N or training-set size T approaches latent-space size M.The discussion studies the limit N, T ∼ M.Source: §4 · p. 57

4.2 Equiparameterization

• Large training sets and models push resource-efficient, properly regularized models toward equiparameterization; the authors say double descent is not really relevant in this regime.Source: §4 · p. 57

4.3 Dimensionality and Latent Space

• The discussion seeks to reconcile the large latent space needed for scaling-law datasets with low-dimensional intrinsic representations, noting that dimensionality notions matter differently for different questions.Source: §4 · pp. 57–58

4.4 Limitations

• The authors identify limitations of their generative data model as a target for future analyses, without specifying them in this passage.Source: §4 · p. 58

4.1 The Breakdown of Neural Scaling Laws

• The breakdown analysis asks what happens when the latent space is no longer the largest scale, since the usual phenomenological loss formula depends on N and T but not M.The familiar fit was important in the regime N, T ≪ M.Source: §4.1 · p. 58

No noise

• Without noise, test loss drops rapidly to zero when the training-set size reaches the latent-space size, T = M.The plotted model has N > T, M; the figure also contrasts the usual regime with the large-latent-space regimes.Source: No noise · p. 58, Figure 9 · p. 59
• For T ≪ M, the no-noise test loss retains the earlier power-law scaling, but near T = M it is controlled by the coincident regime and vanishes as T approaches M from below.Adding optimal ridge regularization does not materially change this behavior in the N → ∞ limit.Source: No noise · pp. 59–60

Noise

• For noisy models with N > M, the fit transitions from L ∼ T^-α to noise-dominated L ∼ T^-1, before reaching a resource-bottlenecked plateau in the usual regime.The discussion says this transition limits resource-driven gains when α > 1, but enhances them when α < 1.Source: Noise · pp. 61–62
• With label noise and N, T > M, both unregularized and regularized test loss show a universal ∼1/T falloff at large T.The stated universality assumes label noise is not correlated from sample to sample.Source: Noise · p. 61

The Battle of the Parameterizations

• The section frames the contrast between overparameterized interpolation and underparameterized bias–variance tradeoffs as the context for double descent.For a fixed training set, the underparameterized test error has an extremum where bias and variance balance.Source: §4.2 · p. 62

Optimality of Equiparameterization

• In the ridgeless fixed-training-set case, test loss has a local minimum at N = N⋆ before rising to a peak near equiparameterization, then improving slowly and asymptotically for larger models.The local minimum trades off bias from models that are too small against variance from models that are too large.Source: §4.2 · pp. 62–63
• Proper regularization makes test loss decrease monotonically with model size, removing special behavior around equiparameterization and the ridgeless double-descent peak.The authors describe the peak as an artifact of not regularizing in a small region around N ∼ T.Source: Figure 11 · p. 63, §4.2 · p. 64
• Jointly scaling model and training-set sizes gives the best test loss because growing model size alone eventually plateaus, while more data extends the power-law gains.In the model, each added sample informs an additional eigen-feature, which requires an additional model feature to represent it.Source: Figure 11 · p. 63, §4.2 · p. 64
• For this generalized linear model, setting N(T) = T avoids the plateau; other models such as LLMs may instead follow a relation N(T) ∼ T^p.The section concludes that joint near-equiparameterization with proper regularization is the practical regime of interest.Source: §4.2 · p. 64

4.3 The Battle of the Dimensions

• The power-law scaling picture requires latent-space size M to be the largest scale, whereas the usual manifold picture treats intrinsic dimension as smaller than input dimension.In this model’s scaling picture, the latent-to-input transformation acts like a projection, so many samples are needed to reconstruct the underlying description.Source: §4.3 · p. 65

Defining and Measuring Intrinsic Dimension

• Intrinsic dimension is operationally defined by the power relating average nearest-neighbor distance to point density, and can be estimated from the slope of that relation on a log-log plot.The paper uses the scaling of average nearest-neighbor distance with dataset size as a measurement procedure.Source: Defining and Measuring Intrinsic… · p. 66

Natural (Power-Law) Data

• Simulations support an inverse relationship between intrinsic dimension and the power-law exponent α, and suggest that intrinsic dimension does not vary with latent-space size M.The figure varies α while holding M fixed, then varies M while holding α fixed.Source: Natural (Power-Law) Data · p. 66, Figure 12 · p. 67
• For continuous power-law spectra, α provides an estimate of intrinsic dimension because the spectrum has no natural cutoff separating informative from uninformative features.The paper contrasts this with spectra having a gap, where principal components above the gap can estimate intrinsic dimension.Source: Natural (Power-Law) Data · p. 67

Latent Features vs. Trained Features

• Intrinsic dimension may differ between raw or latent inputs and trained features because task labels can make a trained network ignore irrelevant input features.In this statistical model the distinction is unimportant because labels depend on all latent features and the model uses random, not label-trained, features.Source: Latent Features vs. Trained Features · p. 68

4.4 The Breakdown of our Data Model

• The model relies on many decades of power-law covariance eigenvalues; simpler isotropic Gaussian data would not produce power-law test-loss scaling.As more samples or features are added, a larger fraction of latent eigenvalues becomes visible in the empirical covariance matrix.Source: §4.4 · pp. 68–69
• The ReLU feature map extends the power-law spectrum as dataset size grows, with its extent determined by the limiting resource T when feature dimension is effectively infinite.The exponent α also decreases after the ReLU map, which the paper says leads to worse performance for a given set of resources.Source: §4.4 · p. 69, Figure 13 · p. 70
• Unlike the model, the MNIST comparison continues to show power-law test-loss falloff beyond Nin, indicating that spectral extension translates into loss scaling there.The MNIST loss is rescaled and the model exponent is matched to facilitate the comparison.Source: §4.4 · pp. 70–71
• The random-feature data model’s test loss plateaus once training-set size T reaches input dimension Nin, despite a much larger ReLU feature width.The authors conclude that spectral extension alone is insufficient to extend the test-loss power law in this model.Source: §4.4 · p. 70, Figure 14 · p. 71
• The authors’ heuristic explanation is that random down-sampling discards latent eigen-directions needed to predict labels, and random ReLU features need not align with those directions.They identify understanding how nonlinear activations decode relevant eigenvectors in natural datasets as an open extension of the model.Source: §4.4 · p. 72

5 Conclusion and Featured Directions

• The joint data and random-feature model captures neural scaling behavior and helps explain its mechanism using random matrix theory.The paper argues that simpler data or feature models do not capture the full behavior.Source: §5 · p. 72
• The model probes M < N, T and suggests measuring latent-space size M to estimate when power-law gains give way to a noise-dominated regime.The authors say including M improves the test-loss fit and that knowing its size could clarify how much further language-model performance can improve.Source: §5 · p. 72
• The authors hope large-N diagrammatic RMT and differential-operator methods will help simplify machine-learning calculations.In the jointly large dataset and parameter regime, planar diagrams capture dominant RMT contributions.Source: §5 · p. 72

Representation Learning?

• The quadratic model’s parameter coupling makes it nonlinear in its parameters, allowing it to learn input representations; with power-law covariance data, it may yield unequal scaling exponents.The authors propose extending the analysis to this model as a route beyond random-feature linear regression.Source: Representation Learning? · pp. 73–74
• In the quadratic model, nonlinear contributions improve performance for a given data model when their test predictions are anti-aligned, at sufficiently small ε.The sign can vary across data models.Source: Representation Learning? · p. 74

A Linear Models

• The linear model uses input features directly, with no separate latent feature space or random feature map, and increasing feature count also increases label complexity.Thus, model capacity cannot be increased without also making the task harder.Source: A Linear Models · pp. 75–76
• Averaging the test loss over training inputs, test inputs, label weights, and noise reduces the calculation to the training-set average of tr{Λq}.The resulting expression is written as a differential operator acting on the quantity that determines the self-consistent equation for ∆.Source: A Linear Models · pp. 76–77
• The test loss has a simple form in terms of ∆−1 and ∆0, which can be determined from the underlying data spectrum.The section evaluates the solution for Marchenko-Pastur and power-law data models.Source: A Linear Models · p. 78
• The linear model lacks the random-feature model’s N ↔ T symmetry because it has no separation between the model-feature and latent-feature scales.Its phase transition occurs at T = N, rather than exchanging regimes when T or N crosses M.Source: A Linear Models · p. 78

A.1 Marchenko-Pastur Data

• Near T = N, the power-law-data test-loss formula behaves as if the data were MP, and the MP solution fits simulations across varied signal strengths.The MP expression is exact, while the coincident-regime power-law expression is only leading order in ε.Source: §A.1 · pp. 79–80
• For T < N, test loss decreases linearly as (1 − T/N) with increasing T; at T = N, it formally diverges when noise is nonzero.In the noiseless limit, the loss at equiparameterization is identically zero.Source: §A.1 · p. 80
• For T > N, only noise contributes to test loss, and that contribution eventually falls as increasing T averages noise across samples.This is the underparameterized regime.Source: §A.1 · p. 81
• MP data produces no power-law test-loss regime because its spectrum lacks a power law.The analysis describes MP data as structureless noise, unlike the richer behavior of LLMs.Source: §A.1 · p. 81
• The MP-data linear model is not symmetric under exchanging T and N because increasing N also increases the complexity of the labels.Consequently, training few features on many samples is fundamentally easier than training many features on few samples.Source: §A.1 · p. 81

A.2 Power-Law Data

• Near T = N, the test loss behaves as if the data were structureless MP data.The power-law-data formula fits numerical simulations well.Source: §A.2 · p. 82
• For power-law data, test loss scales as LPL(N, T) ∼ T^-α when T ≪ N, but no dual power law in feature count appears.For T > N, the loss has only the universal noise contribution.Source: §A.2 · p. 82
• A simple linear model does not reproduce the full LLM scaling phenomenology; the authors distinguish latent data features from model features using a random-feature model.This distinction is presented as necessary for a sufficient minimal model of neural scaling behavior.Source: §A.2 · p. 82

B.1 Explicit Solution for ∆−1

• For the linear model on power-law data, test loss is asymmetric under exchanging training-set size T and feature count N: it scales as T^-α when T ≪N, but grows as N when N ≪T.The figure description reports no plateau for T > N.Source: §B.1 · p. 83
• The solution is constructed from two asymptotic regimes: µ approaches M in the coincident regime, while µ becomes large after taking M →∞ in the scaling regime.The scaling solution is specific to a power-law spectrum, whereas the coincident-limit expression is determined by the underlying spectrum.Source: §B.1 · pp. 84–85
• The interpolating solution for ∆−1 matches its coincident and scaling limits and agrees closely with numerical solutions except for µ ≲10.The authors expect corrections at small µ, where their random matrix theory analysis may also deviate.Source: §B.1 · p. 86
• For µ > M, the empirical covariance is full rank, so the resolvent has no pole as γ → 0, and consequently ∆ has no γ → 0 pole either.Source: §B.1 · p. 86

B.2 Explicit Solution for ∆0

• In the coincident regime, the leading-order ∆0 answer is independent of the spectrum; in the scaling regime, ∆0 is determined by α.The two regime solutions are combined into an interpolating solution.Source: §B.2 · p. 88
• The ∆0 approximation agrees closely with numerical evaluation, with deviations at µ ≲10 and small crossover error for µ < M.The crossover fit could be improved by retaining terms through order ε^2 in the coincident-region equations.Source: §B.2 · p. 88, Figure 18 · p. 89
• For projected spectra with N < M, ∆F0 scales as (α + 1)N/T − 1 when T < N and as T/N − 1 when T > N.The analytical approximation fits numerical evaluation reasonably well, with small-T and crossover deviations.Source: ∆0 for the Truncated Power-Law Spectrum · p. 90
• For N > M, the projected-spectrum analysis truncates the sum at M, yielding ∆F0 scaling as (α + 1)M/T − 1 when T < M and as T/M − 1 when T > M.The spectrum contains the full power law plus N − M zero eigenvalues in this setting.Source: ∆0 for the Truncated Power-Law Spectrum · pp. 90–91
• Although the self-consistent equation for ∆F depends on the projection u, ∆F0 depends only on its size N, not on the particular u.Source: ∆0 for the Truncated Power-Law Spectrum · p. 91