PaperlayerView exact revision on arXiv

Source-linked AI summary

Deriving Neural Scaling Laws from the statistics of natural language

Francesco Cagnetta, Allan Raventós, Surya Ganguli, Matthieu Wyart

arXiv:2602.07488v3cs.LGcs.AIstat.ML

TL;DR

The paper addresses the missing quantitative theory for data-limited neural scaling exponents in modern language models. It derives the exponent from two measurable language statistics and tests the prediction on GPT-2- and LLaMA-style models across TinyStories and WikiText, finding strong agreement in the tested regimes.

  • Problem

    Existing theory does not quantitatively predict data-limited neural scaling exponents for modern LLMs trained on natural-language datasets, despite the practical importance of these laws.

  • Method

    The paper decomposes loss into prediction-horizon and within-horizon errors, then uses conditional-entropy and token-correlation decay statistics to derive the exponent without fitted parameters or synthetic data models.

  • Results

    Across architectures and TinyStories and WikiText, predicted and observed scaling agree in loss exponents and scaling collapse over the tested dataset sizes and context lengths.

  • Takeaways & Limitations

    The shape of data-limited neural scaling laws is directly linked to measurable statistical structure in natural language within the tested regimes.

  • Takeaways & Limitations

    The experiments probe a finite-range, horizon-limited regime, with accessible horizons reaching only a few tens of tokens and the theory’s broader large-scale applicability remaining an open question.

Abstract

from arXiv · show

Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM trained on any natural language dataset. We provide the first such theory in the case of data-limited scaling laws. We isolate two key statistical properties of language that alone can predict neural scaling exponents: (i) the decay of pairwise token correlations with time separation between token pairs, and (ii) the decay of the next-token conditional entropy with the length of the conditioning context. We further derive a simple formula in terms of these statistics that predicts data-limited neural scaling exponents from first principles without any free parameters or synthetic data models. Our theory exhibits a remarkable match with experimentally measured neural scaling laws obtained from training GPT-2 and LLaMA style models from scratch on two qualitatively different benchmarks, TinyStories and WikiText.

1. Introduction

Neural scaling laws guide training decisions, but the mechanism controlling the data-scaling exponent remains unknown. This work proposes a parameter-free theory linking that exponent to measurable statistical properties of natural language and tests it across models and datasets.

  • Motivation: Neural scaling laws describe power-law performance improvements with dataset size, model capacity, and compute, guiding large-scale training decisions.The data-limited relation is especially important because it governs expected returns from collecting additional data.
  • Related work: Existing theories mainly derive learning-curve exponents from fixed-kernel spectra, whereas LLMs learn representations during training.Other approaches study latent hierarchical structure using synthetic or toy language models, with limited empirical scope.
  • Contribution: The paper develops a general framework that predicts loss-learning exponents from measurable natural-language statistics without synthetic data models or fitted free parameters.The framework targets data-limited scaling laws for LLMs trained on natural language.
  • Contribution: Two language statistics determine the predicted exponents: conditional-entropy decay with context length and token-correlation decay with temporal separation.These statistics are measured in TinyStories and WikiText, then compared with GPT-2- and LLaMA-style training experiments.
  • Results: Across architectures and datasets, theoretical and experimental scaling behavior agrees in loss exponents and scaling collapse over the tested dataset sizes and context lengths.The reported agreement links neural scaling-law shape directly to language’s statistical structure.

2. Notation and setup

The paper formalizes language as a distribution over token sequences and defines next-token prediction at each context horizon. Its n-gram loss is minimized by the corresponding dataset conditional entropy, which decreases as context grows.

  • Corpus and sequence model: A corpus is modeled as a distribution over token sequences, with contiguous subsequences sampled from concatenated documents for training.The sequence distribution defines length-n marginals over the finite token vocabulary.
  • Corpus and sequence model: At horizon n, the language model predicts the next token conditioned on the preceding n tokens within a maximum context size T.The horizon n ranges from 1 through T.
  • Loss and entropy: The n-gram loss L_n is minimized at the next-token conditional entropy H_n of the dataset.H_n depends only on the dataset rather than the model parameters.
  • Loss and entropy: The conditional entropy H_n decreases as the time horizon n increases.Longer contexts therefore provide a lower dataset-level prediction-loss floor.

3. Theory: data-limited scaling exponents from natural language statistics

The theory decomposes data-limited loss into learning a data-dependent prediction horizon and using information within that horizon. Two power-law language statistics then yield the scaling exponent and predict collapse of rescaled n-gram learning curves.

  • Loss decomposition: The loss is decomposed into horizon growth and suboptimal information use within the accessible prediction horizon.Increasing the horizon lowers the conditional-entropy floor, while better within-horizon prediction lowers residual errors.
  • Data-dependent prediction time horizon: The data-dependent horizon n*(P) is the largest context window that training data P allows the model to use beneficially for next-token prediction.It grows with P because more data reveals weaker correlations at longer temporal separations.
  • Data-dependent prediction time horizon: The theory estimates the usable horizon by measuring token-correlation strength and accounting for sampling noise from the finite training set.The correlation signal is quantified through the operator norm of the covariance matrix, with sampling noise scaling as O(P^-1/2).
  • Language statistics: The theory assumes power-law decay for conditional entropy with context length and token-correlation strength with temporal separation.These hypotheses connect measurable language statistics to the data-limited learning curve.
  • Data-limited scaling exponent: n*(P) ≍ P^(1/(2β)) follows from the correlation scaling, and the resulting data-limited exponent is α_D = γ/(2β) when within-horizon errors decay faster than the entropy floor.This is the horizon-limited regime in which accessible statistical dependencies dominate excess loss.
  • Scope of the theory: The fast within-horizon learning assumption is architecture-dependent and may fail for shallow networks, kernel methods, or n-gram models.The paper expects it to apply instead to a class of deep networks that includes the tested LLMs.
  • Collapse of the n-gram losses: The framework predicts that rescaling each n-gram curve by its entropy scale H_n and data threshold should produce a common master curve.The predicted collapse is observed experimentally, supporting the scaling form.

4. Empirical verification on text corpora

The paper validates its theory on TinyStories and WikiText by estimating language exponents from model losses and token correlations, then comparing predicted and empirical data-limited scaling. Across architectures and datasets, rescaled n-gram curves collapse and αD = γ/(2β) matches observed autoregressive loss scaling without fitted parameters.

  • Language statistics: The validation measures γ from next-token conditional-entropy decay and β from temporal token-correlation decay on TinyStories and WikiText.These exponents are combined to predict the data-limited learning-curve exponent.
  • TinyStories: Increasing data primarily extends the effective prediction horizon n∗(P), producing a collapse of n-gram learning curves under the theory’s rescaling.The collapse supports an n-dependent data threshold and entropy normalization for Ln(P).
  • TinyStories: αD = γ/(2β) predicts TinyStories scaling using γ = 0.325 and β = 0.88, with empirical curves matching across context lengths.The agreement is interpreted under the condition n∗(P) ≪ T.
  • WikiText: WikiText shows the same rescaled n-gram collapse and autoregressive power law, with αD = γ/(2β) compatible across different context lengths.The measured WikiText exponents are γ = 0.27 and β = 0.94.
  • Robustness: The collapse is sensitive to β and γ uncertainty, while the Llama TinyStories exception remains compatible with the predicted αD.For WikiText, collapse deterioration occurs outside β’s standard-error interval or γ ∈ [0.23, 0.30].

5. Improvement within the prediction time horizon

The analysis tests the horizon-limited regime through n-gram learning curves, showing that increasing data extends the usable prediction horizon while within-horizon learning rapidly saturates.

  • Testing the horizon-limited regime: The n-gram curves directly test whether excess losses decay faster than P^-γ/(2β), the horizon-limited prediction.This provides an empirical route to testing the regime because the excess losses themselves are not easily accessible.
  • Estimating n-gram exponents: For n ≤12, the analysis fits large-P n-gram losses to A × P^-δn + Hn by grid-searching Hn and maximizing the logarithmic fit’s R2.The fitted slope δn estimates the decay exponent after subtracting the asymptote.
  • Interpretation: As P increases, the maximal prediction horizon grows as n*(P) ≍ P^1/(2β), while the model quickly learns to use tokens within that horizon.Thus the loss is controlled by the data-dependent horizon and the conditional entropy at that horizon.

6. Conclusions

The paper develops a data-limited scaling theory whose parameters come from measurable language statistics and discusses its possible universality and performance boundaries. Its evidence includes faster-than-predicted n-gram convergence and raises open questions about architectures beyond the tested setting.

  • Conclusions: The theory explains data-limited LLM learning-curve exponents through the joint dependence of autoregressive loss on context length and dataset size.All parameters are inferred from empirical language statistics, enabling direct tests of the predictions.
  • Conclusions: The framework’s horizon-limited mechanism is characterized by loss dominated by a finite, data-dependent prediction horizon rather than suboptimal predictions within that horizon.The paper proposes that architectures sharing this mechanism could share dataset-dependent exponents while differing in prefactors.
  • Conclusions: The paper suggests that dataset statistics may impose a performance limit, while alternative architectures could potentially achieve larger data-limited exponents.Toy hierarchical-data results involving CNNs motivate the possibility of another universality class, but the authors qualify this for natural language.
  • Conclusions: The n-gram losses decay to their asymptote faster than P^-γ/(2β), supporting the proposed horizon-limited behavior.The accompanying analysis estimates the asymptotic decay from GPT-2-style transformer curves.

Limitations

The empirical tests cover only a finite-range scaling regime determined by the largest dataset size and maximal context length. The tested effective horizons reach only a few tens of tokens, leaving extension to industrial-scale settings unresolved.

  • Finite-range regime: The tests are constrained by the finite Pmax and maximal context length T available in the experiments.At fixed Pmax, the largest effective horizon n*(Pmax) that can be unlocked is limited.
  • Scope of evidence: The experiments probe horizons of only a few tens of tokens, corresponding to a few sentences despite encompassing syntax.Whether the horizon-limited abstraction extends to trillion-scale datasets and context windows ≳10^5 remains an open question.

A. Theory of asymptotic scaling of the autoregressive loss

The theory decomposes autoregressive loss into limited prediction horizon and suboptimal within-horizon use, linking the horizon to temporal token correlations and entropy to context length. Under fast within-horizon learning, the data-limited scaling exponent is αD = γ/(2β), and rescaled n-gram curves should collapse.

  • Language statistics: Conditional entropy Hn decreases with context length and approaches the entropy rate H∞, providing the lower bound for n-gram losses.The theory assumes power-law decay of conditional entropy with context length.
  • Data-dependent prediction horizon: Token correlations determine the data threshold for using tokens n steps in the past: stronger temporal separation requires more data to detect useful correlations.The covariance operator norm is used to quantify correlation strength and empirically follows a power law with exponent β.
  • Data-dependent prediction horizon: The maximal prediction horizon grows with dataset size as n∗(P) ≍ P^1/(2β), allowing models to use progressively longer contexts as training data increases.This horizon is the largest context length that can be beneficially leveraged for next-token prediction at data amount P.
  • Loss decomposition: The autoregressive loss is decomposed into a boundary contribution from the data-dependent prediction horizon and excess loss from suboptimal within-horizon use.The boundary term is tied to conditional entropy at the maximal usable horizon, while excess losses capture inefficient use of tokens within that horizon.
  • Asymptotic scaling: When within-horizon learning is fast, the autoregressive loss decays with exponent αD = γ/(2β), determined only by entropy-decay exponent γ and correlation-decay exponent β.In the slow-learning regime, the within-horizon exponent δ instead dominates and the loss decays as P^-δ.
  • Scaling collapse: Rescaling each n-gram learning curve by its threshold P/P∗(n) predicts a common scaling collapse across disparate context lengths.The collapse follows when variations in the within-horizon decay exponent are negligible at large n.

C. Errors

The appendix quantifies uncertainty in the language exponents γ and β, propagates it to αD = γ/(2β), and tests the robustness of scaling collapses and empirical fits.

  • Uncertainties in γ and β are propagated to the predicted data-limited scaling exponent αD = γ/(2β).The same uncertainty analysis also evaluates robustness to variations in these exponents.

C.1. Errors on β and γ

The paper estimates β from correlation-decay fits and γ from power-law fits to limiting n-gram learning curves, using standard errors and bootstrap checks.

  • Errors on β and γ: β is estimated as the slope of a linear fit between log n and log ∥C(n)∥op.Standard-error and bootstrap estimates quantify uncertainty in the fitted slope.
  • Errors on β and γ: For TinyStories, β uncertainty from standard error and bootstrapping is 0.06.The estimate uses 15 logarithmically spaced separations from n = 1 to n = 200.
  • Errors on β and γ: γ is estimated by fitting a power law to the portion of the n-gram learning curve that has approached its infinite-data limit.TinyStories uses GPT-2-style models with T = 128 and horizons n = 1,...,16; other model classes yield compatible estimates.

C.2. Quality of collapses within the error range

The paper tests whether independently measured γ and β organize n-gram learning-curve collapses, finding stability within uncertainty ranges and visible deterioration outside them, with a LLaMA exception.

  • C.2. Quality of collapses within the error range: For TinyStories, the collapse remains stable across the displayed γ range but deteriorates when β leaves its bootstrap uncertainty range.This supports the claim that independently measured exponents organize the n-gram learning curves.
  • C.2. Quality of collapses within the error range: For WikiText, varying γ within its bootstrap uncertainty range does not qualitatively change the collapse.The figure captions also examine deterioration when γ is swept past its uncertainty range.
  • C.2. Quality of collapses within the error range: For LLaMA on TinyStories, the sharpest visual collapse occurs for β ∈[0.6, 0.7], outside the standard-error interval.The corresponding scaling law remains compatible with the predicted exponent αD, so the discrepancy does not affect the main conclusion.
  • C.2. Quality of collapses within the error range: For WikiText, varying β within its uncertainty range produces qualitative deterioration of the collapse.This sensitivity is illustrated while γ is held fixed.

C.3. Error on the data-limited exponent and fits of the empirical learning curves

The paper propagates uncertainty from γ and β to αD, fits empirical exponents from autoregressive-loss envelopes, and compares predicted and measured scaling across model classes, context lengths, and datasets.

  • C.3. Error on the data-limited exponent and fits of the empirical learning curves: The predicted exponent αD is obtained by standard propagation of error from the uncertainties in β and γ.The resulting prediction intervals are compared with empirical learning-curve estimates.
  • C.3. Error on the data-limited exponent and fits of the empirical learning curves: Empirical αD is estimated by fitting the first m ≥5 points of the lower envelope of autoregressive losses across context lengths on a log-log scale.The fitted negative slope is converted into a positive exponent, with 95% bootstrap confidence intervals.
  • Supporting figures: The supporting figures test n-gram collapse and exponent predictions across WikiText and TinyStories, including T = 512, RoPE, and LLaMA-style models.The LLaMA TinyStories figure reports compatibility with αD = γ/(2β) using γ = 0.325 and β = 0.88.

E. Experiment details

The experiments evaluate GPT-2, LLaMA, and Mamba models on TinyStories and WikiText using standardized tokenization, architecture configurations, and validation-loss-driven hyperparameter tuning. Hyperparameter search is locally optimized rather than exhaustive because of computational constraints.

  • Datasets and tokenization: Experiments use TinyStories and WikiText, tokenized with whitespace pretokenization followed by BPE with an 8192-token vocabulary and EOS separators.The resulting token sequences are concatenated into a single stream.
  • Architectures: Most experiments use a GPT-2-style architecture with 768-dimensional embeddings, 12 layers, 12 attention heads, and approximately 98M trainable parameters.The base configuration uses a feedforward dimension four times the hidden size.
  • Architectures: LLaMA experiments use three reduced LLaMA-3.2-1B configurations, while Mamba experiments use 12 standard Mamba blocks with 768-dimensional embeddings.The LLaMA configurations vary embedding dimension, depth, attention heads, key-value heads, and feedforward dimension.
  • Training hyperparameters: For each architecture and dataset size, optimization hyperparameters are tuned by grid search to minimize validation loss using AdamW.The search covers learning rate, weight decay, training epochs, and batch size, with GPT-2 learning-rate and weight-decay grids explicitly specified.
  • Model-specific tuning: LLaMA TinyStories experiments fix batch size at 64 and tune learning rate, weight decay, and training epochs, while Mamba experiments use the same grid as the other TinyStories runs.The LLaMA learning-rate search includes 3e-4, 1e-3, and 3e-3.
  • Compute and search limitations: Computational constraints prevent exhaustive hyperparameter search, so the experiments target local optimality and use optimal settings at one dataset size to initialize tuning at the next.All experiments run on NVIDIA H100 GPUs; GPT-2 uses one GPU and LLaMA uses eight GPUs with data parallelism.
Loading 2602.07488v3…