PaperlayerView exact revision on arXiv

Source-linked AI summary

Skaling: Chinchilla's Exponents Meet Kaplan's Coupling

Mathurin Videau, Badr Youbi-Idrissi, David Lopez-Paz, Kartik Ahuja

arXiv:2608.07222v1cs.CL

TL;DR

Standard additive scaling laws assume model size and data affect loss independently, limiting accuracy at imbalanced grid extremes. Skaling adds one coupling exponent and reduces extrapolation MAPE by 1.5–3× while enabling accurate profiling with approximately 10× less compute.

  • Problem

    The Chinchilla scaling law assumes model size and training data affect loss independently, producing large errors at imbalanced training-grid extremes.

  • Method

    Skaling generalizes Chinchilla with a single coupling exponent between model size and data, supported by non-zero mixed derivatives of the loss surface.

  • Results

    1.5–3× lower extrapolation MAPE than Chinchilla is achieved across multiple cross-validation strategies, including boundary and far-extrapolation settings.

  • Takeaways & Limitations

    A sparse L-shaped grid preserves predictive accuracy while reducing profiling compute by approximately 10×, supporting more efficient scaling prediction and compute allocation.

  • Takeaways & Limitations

    The direction of Skaling’s allocation trend is dataset-specific and depends on the fitted data and architecture.

Abstract

from arXiv · show

Neural scaling laws are foundational for language model development, yet standard formulations systematically under- and overestimate loss at data-scarce and overtraining extremes. This failure originates in the underlying assumption that model size and training data impact the loss independently. To address this, we introduce the Skaling law, a generalized functional form that couples model capacity and data through a single interaction exponent. This simple extension reduces the Mean Absolute Percentage Error (MAPE) by 1.5-3x across both interpolation and extrapolation regimes. When paired with a sparse grid strategy restricted to low-compute regimes, the Skaling law achieves accurate full-grid extrapolation using approximately 10x less compute than uniform sweeps. By enabling reliable performance prediction from small-scale experiments, the Skaling law provides a more robust and resource-efficient framework for allocating compute budgets in next-generation model training.

1 Introduction

Standard Chinchilla scaling treats model size and training data as independent, causing boundary errors, while Skaling restores their interaction with one coupling exponent. This improves extrapolation accuracy and supports sparse L-shaped profiling grids that reduce compute while maintaining accuracy.

  • Limitations of standard scaling laws: Standard Chinchilla models reducible loss as independent additive terms in model size and training tokens, enforcing a cross-derivative of exactly zero.This independence produces large, oppositely signed corner errors reaching several percent, despite accuracy in the grid interior.
  • Skaling law: Skaling introduces a single coupling exponent between model size and data volume, restoring the interaction discarded by the additive Chinchilla law.The proposed form is a minimal generalization designed to correct boundary biases.
  • Skaling law: 1.5–3×: Skaling reduces extrapolation MAPE relative to Chinchilla across multiple cross-validation strategies.The improvement comes from adding only one parameter to the Chinchilla form.
  • Sparse profiling grids: ∼10×: L-shaped sparse profiling grids reduce compute while maintaining Skaling’s predictive accuracy.The sparse sampling scheme preserves predictive accuracy and enables efficient profiling in low-compute regimes.

2 The loss surface couples model size and data

Derivative-based analysis shows that model size and training data interact on the loss surface: although first-order projections appear nearly separable, the mixed derivative is non-zero throughout the grid. This coupling motivates replacing Chinchilla’s additive form with the Skaling law’s single coupling exponent.

  • Derivative-based analysis: Moving least squares estimates convert logarithmic-grid log-slopes into real-space derivatives, removing irreducible error E to isolate reducible-loss structure.The estimator is described as the Lancaster and Salkauskas (1981) method.
  • First-order structure: αN ≈ αD ≈ −1.3, while γN ≈ 0.13 and γD ≈ 0.07, indicating power-law marginal decay and small, asymmetric cross-axis dependence.The log-linear diagnostic is used only to characterize first-order structure, not as the scaling law itself.
  • Mixed derivative: ∂2L/∂N∂D is non-zero across the entire grid, contradicting every additive law L = f(N) + g(D) + E.The estimated mixed derivative follows its own power-law decay; Figure 3 reports non-zero estimates throughout the grid.
  • Mixed derivative: a ≈ b ≈ −1.1 and a predominantly negative sign show that jointly scaling N and D lowers loss more than scaling either alone.This synergy cannot be represented by an additive law and motivates the single coupling exponent in the Skaling form.

3 Methods

The methods introduce Skaling as an outer-exponent coupling of Chinchilla’s base terms, then use low-compute L-shape sampling and cross-validation to estimate parameters and test interpolation and extrapolation.

  • Skaling law: Skaling raises Chinchilla’s additive base terms to a single free outer exponent k, coupling model size and data while preserving independent inner exponents.This contrasts with Kaplan’s coupling of inner terms through their exponent ratio.
  • Skaling law: At k = 1, Skaling recovers Chinchilla; for k ≠ 1, it produces Kaplan-style coupling and a non-zero cross-derivative without tying the inner exponents.The coupling is introduced through k rather than Kaplan’s ratio constraint.
  • Compute allocation: The Skaling law retains Chinchilla’s closed-form compute-optimal allocation, while its predicted optimal token-to-parameter ratio can differ substantially in practice.The Chinchilla-predicted ratio accumulates to a 100-fold discrepancy from Skaling at frontier compute, according to Section A.3.
  • Validation framework: Cross-validation evaluates interpolation, N extrapolation, D extrapolation, and far extrapolation beyond both training boundaries.MAPE is reported across evaluation sets, while R2 is restricted to interpolation because it is unstable on sparse extrapolation slices.
  • Experimental grid: The L-shape strategy fits data parameters by sweeping D for the smallest models and size parameters by sweeping N at the shortest training horizons.It avoids filling the full N–D space and concentrates training at the lowest compute scales.

4 Experiments

Experiments on Farseer and SK-Grid show that Skaling improves boundary extrapolation by coupling model size and data, while preserving accuracy under sparse profiling. Its gains arise from the functional form and depend on the observed coupling structure, not simply parameter count or interpolation fit.

  • Boundary errors: Skaling reduces full-grid single-axis MAPE from 1.48 to 0.47 and from 1.98 to 0.88 on Farseer, and from 0.83 to 0.39 and from 1.44 to 0.58 on SK-Grid.The largest gains occur at single-axis and far-extrapolation boundaries, where additive Chinchilla errors grow.
  • Sparse profiling: 10× less fitting compute preserves Skaling accuracy on the L-shape grid, while Chinchilla interpolation MAPE rises from 0.77 to 2.51 on Farseer.On SK-Grid, Chinchilla far-extrapolation MAPE increases from 5.17 to 14.63 under the same sparse restriction.
  • Interpolation is not sufficient: 0.995 and 0.992 interpolation R2 for Chinchilla on Farseer and SK-Grid do not prevent substantially larger extrapolation errors than Skaling at grid boundaries.The experiments identify systematic bending of the loss surface away from the observed interior as the relevant failure mode.
  • Stable coupling: k ≈0.31–0.45 across the grids shows that fitted Skaling coupling remains sub-unit rather than collapsing to Chinchilla’s additive k = 1.The fitted irreducible loss changes from 0.45 to 0.03 on Farseer full-grid data and from 0.59 to 0.05 on its L-shape, while SK-Grid changes from 1.75 to 1.14.
  • Parameter count: 4.13 and 4.45 data-extrapolation MAPE on the Farseer and SK-Grid full grids show that the nine-parameter Farseer baseline does not remove boundary failure.Its larger parameter count does not explain Skaling’s advantage; the functional form’s inductive bias better matches the observed N–D interaction.
  • Allocation frontier: −0.14 and −0.15 fitted allocation exponents agree with Skaling’s −0.11 analytic exponent, whereas Chinchilla predicts a near-flat +0.03 trend on Farseer data.On SK-Grid, α > β reverses the direction, so coupling changes allocation behavior without imposing a universal trend.

5 Related Work

Prior scaling-law work largely adopted additive independent power laws, while attempts to reconcile competing results focused on fitting procedures rather than functional form. Related alternatives include Busbridge et al.’s untied outer-exponent form and Farseer’s more parameterized N–D interaction, whereas this work makes the interaction central.

  • Early scaling laws modeled loss as a power law in scale, while Kaplan et al. modeled loss jointly in model size N and data D.
  • Hoffmann et al.’s additive form, E + A/N^α + B/D^β, became the field’s default and supported compute-optimal allocation and compute extrapolation.The form treats model size and data as acting independently on loss.
  • Competing Kaplan- and Chinchilla-style scaling results were reconciled through parameter counting, FLOP accounting, warmup, and optimizer tuning, leaving the additive form intact.
  • Busbridge et al. use the untied form E + (A/N^α + B/D^β)^γ, but do not report a controlled comparison of supervised forms or study the resulting N–D interaction.Their main contribution is a distillation law predicting student loss from teacher losses and corresponding supervised student losses.
  • Nine fitted parameters give Farseer an N–D interaction, but its added flexibility is harder to fit and less accurate than Skaling at grid boundaries in the authors’ experiments.

6 Conclusion … A.1 Numerical gradients

The Skaling law couples model size and data with one exponent, enabling accurate full-grid reconstruction from sparse boundary runs. Numerical gradients are estimated using local MLS and global GP procedures, with GP derivatives obtained analytically.

  • 6 Conclusion: The Skaling law replaces independent model-size and data effects with a single coupling exponent that captures their interaction in the empirical loss surface.This extension addresses prediction failures at imbalanced training-grid extremes.
  • 6 Conclusion: A sparse, L-shaped grid of inexpensive boundary runs is sufficient to recover the full-grid law without measuring the entire parameter surface.The approach reduces the computational burden of reliable scaling prediction for compute allocation.
  • A.1 Numerical gradients: At each grid point, the analysis estimates loss-surface derivatives directly from measured runs using moving least squares and Gaussian-process procedures.MLS is local, whereas GP is global.
  • A.1 Numerical gradients: MLS estimates derivatives by fitting a truncated multivariate Taylor polynomial to neighboring points around each target location.The polynomial coefficients correspond to derivatives of different orders, including the gradient and Hessian terms.
  • A.1 Numerical gradients: The MLS coefficient vector is fitted by distance-weighted ridge regression over the k nearest neighbors, using a design matrix, Gaussian weights, and ridge regularization.Closer points receive more influence, while regularization stabilizes inversion for poorly distributed local data.
  • A.1 Numerical gradients: MLS derivatives are read from designated coefficient slots, including the mixed cross-derivative, but accuracy is sensitive to local noise and neighborhood choices.The mixed term corresponds to the ∆xi,1∆xi,2 basis coefficient.
  • A.1 Numerical gradients: The GP fits all points with a smooth noisy-loss model and differentiates its posterior mean in closed form, while analytically differentiable kernels make the computation exact.The GP also provides a closed-form predictive variance for gradient uncertainty and automatically tunes length-scales and noise levels.

A.2 Form of the coupling · A.3 Empirical optimal token to parameter ratio

The Skaling coupling preserves marginal power-law behavior while introducing an interaction through a shared multiplicative factor, making monotonicity and negative mixed curvature compatible. Compute-optimal allocation is identified by balancing log-loss gradients, with empirical token-to-parameter ratios decreasing with compute in agreement with Skaling rather than Chinchilla.

  • A.2 Form of the coupling: The Skaling form preserves marginal power-law behavior while coupling model size and data through the shared factor u^(k−1).Its first derivatives remain dominated by same-variable power-law factors, while cross-variable dependence enters through the shared factor.
  • A.2 Form of the coupling: k = 1 recovers the additive Chinchilla law, whereas 0 < k < 1 introduces a negative mixed derivative matching the observed sign.The interaction appears when k ≠ 1, while the empirically relevant regime 0 < k < 1 yields negative mixed curvature.
  • A.2 Form of the coupling: An additive interaction term creates a sign conflict: matching negative mixed curvature requires G < 0, but then its size contribution opposes decreasing loss with model size.Because µ ≈0.1 is smaller than α, the positive contribution can decay more slowly in N than the leading negative term at large N and small D.
  • A.2 Form of the coupling: The multiplicative Skaling coupling keeps ∂L/∂N negative for every k > 0 while producing ∂²L/∂N∂D < 0 when 0 < k < 1.Increasing D strengthens the magnitude of the already-negative size gradient, so monotonicity and synergy are compatible by construction.
  • A.2 Form of the coupling: γN ≈0.13 exceeds γD ≈0.07, explained by the coupling relation γN/γD = (β/α)(wD/wN) when β > α and the inner sum is data-leaning.The single coupling exponent reproduces the measured asymmetric first-order slopes without requiring a skewed interaction term.
  • A.3 Empirical optimal token to parameter ratio: For fixed compute C = 6ND, the empirical optimum is the lowest-loss point along the iso-compute curve, where equal percentage increases in model size and data have the same signed loss effect.This is constrained stationarity along the iso-compute tangent; the full gradient need not vanish.
  • A.3 Empirical optimal token to parameter ratio: The optimality condition becomes ∂z/∂x = ∂z/∂y in log coordinates, and GP and MLS independently estimate the frontier by tracking this gradient equality.Both methods use continuous or local surrogate estimates to locate the zero-crossing boundary from discrete runs.
  • A.3 Empirical optimal token to parameter ratio: m ≈−0.14 for the GP and m ≈−0.15 for MLS, close to Skaling’s exponent of −0.11; both ratios decrease with compute, unlike Chinchilla’s nearly flat prediction.The empirical frontiers track Skaling’s Ropt qualitatively across compute levels.

B Compute-optimal training … C.2 Hyperparameters

The Skaling law preserves Chinchilla’s compute-optimal allocation while separating loss-landscape shape from allocation location. Fitting remains fragile, but configured L-BFGS-B with basin-hopping and BIPOP-CMA-ES achieve equally good fits, with reported results using L-BFGS-B.

  • B Compute-optimal training: Skaling inherits Chinchilla’s compute-optimal allocation unchanged because minimizing its loss requires the additive term’s stationarity condition Z′(N) = 0.The compute budget is defined as C = 6ND, and optimization minimizes Skaling loss subject to this constraint.
  • B Compute-optimal training: When α ≈ β, the optimal token-to-parameter ratio D/N remains constant across scales.The separation arises because k controls loss-landscape shape, while Z′(N) = 0 determines allocation location.
  • C Fitting Details: The Skaling law gains expressivity without sacrificing tractability because its monotone outer map rescales loss values but leaves the minimizer unchanged.This follows from separating the loss-landscape shape, controlled by k, from the optimal allocation location.
  • C.1 Challenges: L-BFGS and CMA-ES reach equally good fits, but L-BFGS requires careful initialization, restarts, tuning, and a log-space objective, whereas CMA-ES works out of the box.The fitting objective is minimized in log space to absorb much of the parameter scale disparity.
  • C.2 Hyperparameters: L-BFGS-B with basin-hopping combines analytical autograd gradients, multi-start global search, and Sobol-sequence initialization over bounded parameter space.The alternative is BIPOP-CMA-ES, using doubled population size, active CMA, and 9 BIPOP restarts.
  • C.2 Hyperparameters: All reported results use L-BFGS-B with basin-hopping, despite both optimization strategies reaching equally good fits.The common optimizer ensures comparisons reflect functional form rather than fitting recipe.
  • C.2 Hyperparameters: All laws use 2000 basin-hopping restarts, a log-space Huber objective, and analytic autograd gradients; Farseer uses the ExpExpExp parameterization.Farseer is fit with the common direct optimizer because its original pipeline requires evenly spaced D values unavailable in sparse L-shape and scattered datasets.

D Additional datasets

The study extends cross-validation to Farseer-code and Chinchilla data, covering grid-structured and irregular datasets with distinct split protocols. Across these datasets, Skaling fits k < 1 and achieves smaller E than Chinchilla, with weaker coupling of k ≈0.77–0.90.

  • D Additional datasets: Farseer-code comprises 117 runs across 9 model sizes (201M–3.18B) and 20 token budgets (2B–128B).Its compute range is C=6ND from 2.4 × 10^18 to 2.4 × 10^21 FLOPs.
  • D Additional datasets: Because Farseer-code is grid-structured, it supports both full-grid and L-shape cross-validation splits.The other dataset is irregular, so only the full-grid random split applies, with largest-N and largest-D points forming the extrapolation sets.
  • D Additional datasets: Skaling fits k < 1 and achieves smaller E than Chinchilla, while coupling is weaker at k ≈0.77–0.90.The weaker coupling aligns with the more mixed accuracy gains reported in Table 5.

E Pre-training Details · E.1 Model

The pre-training study uses SK-Grid, an internal set of 125 runs spanning 14 model sizes, with architectures and token budgets varied under compute constraints. Models range from 134M to 4.9B parameters and train on token budgets from 316M to 316B, forming a staircase grid.

  • E.1 Model: 125 SK-Grid runs span 14 model sizes.SK-Grid is the internal pretraining-run set used in this section.
  • E.1 Model: For the Farseer grid, the study uses released runs and refers to Li et al. (2025a) for details.The configurations described here specifically define SK-Grid.
  • E.1 Model: The models range from 134M to 4.9B parameters.Model dimension increases from 672 to 3264, while depth grows from 7 to 34 layers.
  • E.1 Model: Model dimension grows from 672 to 3264 and depth from 7 to 34 layers.Width and depth are scaled together across model sizes.
  • E.1 Model: All models use the Llama 3 tokenizer with a vocabulary of 128,256.The tokenizer configuration is shared across the model sizes.
  • E.1 Model: Token budgets span 316M to 316B tokens on a geometric ladder with five budgets per decade.Each model size is trained across a size-dependent subset of this budget ladder.
  • E.1 Model: Per-run compute is capped, so larger models use fewer token budgets.The smallest models cover all 16 horizons, whereas the largest uses a single budget.
  • E.1 Model: The size-dependent budget coverage produces a staircase grid whose row counts appear in Table 7.Budget coverage decreases as model size increases.

E.2 Hyperparameters

Run-specific batch size and peak learning rate are fitted as compute- and data-dependent power laws, while other training settings remain fixed across the grid. These hyperparameter choices can alter the apparent N–D interaction and inferred compute-optimal ratio.

  • Run-specific hyperparameters: Batch size B and peak learning rate η follow StepLaw power laws in per-token compute F and token budget D.The coefficients are refit to the authors’ setup.
  • Sensitivity: Hyperparameter parameterizations can change the apparent N–D interaction and shift the inferred compute-optimal ratio.A broadly mistuned grid may artificially dampen the measured interaction or skew the optimal ratio.
  • Shared configuration: All remaining settings are fixed across the grid, and every run uses a mixture of 60% DCLM-Edu web text, 30% code, and 10% math.The fitted loss is validation loss on a held-out split of this mixture.

F Dominated-pair fitting

Dominated-pair fitting removes the irreducible loss floor E from the fitting objective, estimates the reducible shape from pairwise differences, and then recovers E separately. This primarily corrects Chinchilla’s extrapolation errors, especially in boundary regimes where fitting the loss floor matters most.

  • Method: Dominated-pair fitting cancels E for ordered configurations where i dominates j, allowing shape parameters (A, B, α, β, k) to be fit on pairwise loss differences.The floor is then recovered as E = mediank, decoupling the reducible shape from the constant offset.
  • Results: 2.46% to 0.79%: on Farseer full grids, dominated-pair fitting reduces Chinchilla’s far-extrapolation error.This illustrates the method’s correction of Chinchilla’s additive law in a boundary regime.
  • Results: 5.17% to 3.67%: on SK-Grid full grids, dominated-pair fitting reduces Chinchilla’s far-extrapolation error.The passage reports similar improvements on L-shape grids across all reported cases.
  • Interpretation: The method’s gains suggest that weak identification of E drives much of Chinchilla’s extrapolation error, while its objective reweights data toward extreme boundary points.Removing E prevents the constant offset from distorting the reducible-component fit, and the reweighting compensates for Chinchilla’s imperfect global shape.
Loading 2608.07222v1…