Prescriptive Scaling Laws for Data Constrained Training

1 Introduction

• Training compute is outpacing high-quality data, making data—not compute—the bottleneck and shifting the problem toward extracting value from a fixed data pool.The constraint is especially strong in specialized domains, where datasets can be orders of magnitude smaller than available compute can absorb (Lewkowycz et al., 2022).Source: §1 · p. 1
• Existing repetition extensions can model diminishing returns but not overfitting-driven loss increases or the interaction between overfitting and model size (Muennighoff et al., 2023).The Chinchilla law assumes every token is unique, while larger models overfit faster on repeated data.Source: §1 · p. 1
• The proposed additive penalty models repeated tokens as useful while separately accounting for overfitting that grows with repetition, limited data, and model size.The paper reports that even its one-parameter form substantially outperforms prior data-constrained scaling laws; its 1-, 2-, and 4-parameter forms trace a fit-quality-versus-complexity Pareto frontier.Source: §1 · pp. 1–2
• The law-recommended configurations achieve the strongest performance in both perplexity and downstream evaluation, and it predicts that further repetition eventually becomes counterproductive.Beyond a data-dependent threshold, the paper recommends spending compute on model capacity instead of repetition.Source: §1 · p. 2
• Strong weight decay (λ = 1.0) reduces the fitted overfitting coefficient by approximately 70%.The law also predicts the compute budget at which strong regularization overtakes the standard setting.Source: §1 · p. 2
• The study trains over 300 models spanning 15M–1B parameters, 50M–6B unique tokens, two weight-decay strengths, and up to 16 epochs.Source: §1 · p. 2

Background and Related Work

• Chinchilla (Hoffmann et al., 2022) decomposes loss into irreducible uncertainty, model-size cost, and data-shortage cost; under C≈6ND, it selects a model–dataset pair for a fixed compute budget.The optimal N:D ratio depends on fitted exponents and varies across datasets and training setups.Source: §2 · p. 2
• Because Chinchilla assumes each training token is seen once, it misses the diminishing returns—and possible performance harm—when models train for multiple epochs on scarce data.Additional passes can eventually contribute almost nothing or hurt performance through overfitting.Source: §2 · pp. 2–3
• Muennighoff et al. (2023) model repetition with saturating effective data and effective parameters, capturing diminishing token value and overparameterized models’ reduced capacity under repetition.Their formulation measures excess parameterization relative to the Chinchilla compute-optimal size for the unique-token dataset.Source: §2 · p. 3
• The prior formulation represents overparameterization indirectly through capacity saturation, motivating an explicit repetition-related overfitting penalty whose strength can reflect regularization.The paper argues that the exponential saturation form for excess parameters is not clearly justified.Source: §2 · p. 3
• At high repetition, Chinchilla and Muennighoff et al. (2023)’s b_D/b_N formulation systematically underpredict validation loss across four model sizes at UD=200M.The figure compares predicted and observed loss for both formulations.Source: Figure 1 · p. 4

3 Experimental setup

• Llama 2 models trained on FineWeb vary in size, unique-data budget, repetition, and weight decay, with other hyperparameters held constant.The study sweeps 15M–1B parameters, 50M–6B unique tokens, repetition counts of 0, 1, 3, 7, 11, and 15, and λ = 0.1 or 1.0; it reports final validation loss, perplexity, and average BPB across 19 OLMES tasks.Source: §3 · p. 4

4 Scaling laws for repeated data

• The effective-data formulations increasingly underpredict validation loss at high repetition, with the gap larger when model capacity is high relative to unique data.The exponential saturation in those formulations does not capture this interaction.Source: §4 · p. 4
• δ > 1.0 indicates superlinear repetition damage, so each additional repeated epoch costs more than the last.The residual is measured against Chinchilla predictions that treat repeated tokens as fresh data.Source: §4 · p. 4, Figure 2 · p. 5
• A linear-in-RD penalty using the N/UD ratio substantially outperforms the Muennighoff et al. (2023) formulations.The paper presents penalty forms as a Pareto frontier trading off free parameters against fit quality.Source: §4 · p. 5
• Across configurations, κ > 1 means the overfitting penalty grows superlinearly with model capacity relative to unique data.The full four-parameter form adds a superlinear repetition exponent δ and separates the data-budget exponent γ from the model-size exponent κ.Source: §4 · p. 5
• At RD = 0, the penalty forms reduce exactly to Chinchilla; with repetition, tokens both improve data sufficiency and incur growing overfitting cost.The simplest penalty form uses one free parameter and the dimensionless ratio N/UD.Source: §4 · p. 5

Scaling law validation

• The additive law tracks observed loss under heavy repetition better than effective-data formulations, while its four-parameter form achieves near-perfect fit on the authors’ data.Validation uses two independent sweeps, including Muennighoff et al. (2023) runs through 64 epochs; even the one-parameter form substantially outperforms the compared effective-data forms.Source: §5 · pp. 5–6, Figure 3 · p. 6, Figure 4 · p. 6

Compute-optimal allocation under repetition

• At high compute, our law predicts a point beyond which repetition is counterproductive, favoring larger models trained for fewer epochs.Chinchilla always recommends more repetition, while the Muennighoff et al. (2023) formulation prescribes diminishing returns but never stopping; the apparent disagreement with their smaller-model recommendation is traced to a methodological choice.Source: §5 · p. 6, Figure 5 · p. 7

Prescriptive validation

• Across tested budget settings, our law recommends larger models with fewer epochs and achieves the best perplexity and downstream performance.Prescriptive validation trains the configuration recommended for each token-budget and compute-budget pair; Table 1 describes the comparison among Chinchilla, effective-parameter, and proposed-law recommendations.Source: §5 · pp. 6–7, Table 1 · p. 7

Generalization to external data

• On completely held-out Muennighoff et al. (2023) data, the additive penalty forms significantly outperform those formulations across differing pretraining implementations.The external data differs in choices including backbone architecture and tokenization.Source: §5 · p. 7

Single-epoch scaling

• In the single-epoch regime, strong weight decay incurs higher loss at every compute budget and favors larger models relative to data.Separate Chinchilla parameters are fit for each weight-decay setting.Source: §6 · p. 8, Figure 6 · p. 8

Scaling under data repetition

• Strong weight decay reduces the fitted overfitting coefficient P by approximately 70%, with a lower-magnitude superlinear repetition penalty.The comparison uses the same model-size and data-budget grid at λ = 0.1 and λ = 1.0, holding other hyperparameters constant.Source: §6 · pp. 7–8, Figure 7 · p. 8

Crossover under repetition

• At UD = 250M, strong weight decay overtakes standard weight decay at C ≈ 3.2 × 10^18 FLOPs as repetition erodes its single-epoch advantage.The law predicts this crossover as the compute budget where the lower repetition penalty compensates for the single-epoch tax.Source: §6 · p. 8, Figure 8 · p. 9

Prescriptive validation

• At C = 3 × 10^19, strong weight decay achieves 16.65 vs. 18.16 perplexity with recommended configurations.At C = 1 × 10^19, strong weight decay reduces perplexity by 2.8 points; near the predicted crossover, the settings perform comparably. This supports Kim et al. (2026)'s finding that optimal weight decay in data-constrained regimes can be an order of magnitude larger than standard practice.Source: §6 · pp. 8–9, Table 2 · p. 9

Practical implications

• Beyond a compute threshold, further repetition is counterproductive, so practitioners should spend additional compute on model capacity and tune model-size–epoch tradeoffs.Increasing regularization strength and choosing the right model-size–epoch tradeoff are complementary levers in data-constrained settings.Source: §6 · p. 9

Limitations

• The study tests models up to 1B parameters, repetition up to 16 epochs, and only λ ∈ {0.1, 1.0}, limiting confidence in frontier-scale extrapolation.The law may fail at extreme repetition counts and does not capture double descent (Nakkiran et al., 2020); directly incorporating regularization strength is left for future work.Source: §6 · p. 9

• Beyond a threshold compute budget, further data repetition is counterproductive, so compute should instead scale model capacity.Source: §7 · p. 10
• Strong weight decay (λ = 1.0) reduces the overfitting penalty by approximately 70%, explaining and potentially replacing expensive per-configuration hyperparameter sweeps (Kim et al., 2026).Source: §7 · p. 10
• The one-, two-, and four-parameter complexity ladder provides a Pareto frontier of fit quality versus complexity, and even its simplest form substantially outperforms prior effective-data formulations.Source: §7 · p. 10
• The additive overfitting penalty quantifies robustness to data repetition as a new comparison axis, increasingly important as compute grows faster than data and high-quality data becomes the binding constraint.Source: §7 · p. 10

A Reanalysis of Muennighoff et al. compute-optimal allocation

• The disagreement reflects a baseline fit to Hoffmann et al. (2022)’s scraped C4 runs rather than Muennighoff et al. (2023)’s own runs, whose tokenization, preprocessing, and code differ.All results use Muennighoff et al. (2023)’s filtered 182-run split and outlier-removal criteria.Source: A Reanalysis of Muennighoff et al.… · p. 14
• R2 rises from 0.711 with published Chinchilla parameters to 0.989 after refitting on Muennighoff et al.’s 29 single-epoch runs.A tied refit also achieves R2 = 0.989, so the improvement does not depend on relaxing the α = β constraint.Source: A Reanalysis of Muennighoff et al.… · p. 14
• The fitted R*N rate constant changes from 5.31 under the published baseline to 3,294 under the refit, effectively disabling bN and removing its pull toward smaller models.The refit bD, bN law then recommends larger models with fewer epochs.Source: A Reanalysis of Muennighoff et al.… · p. 14
• With the published baseline, the bD, bN mechanism improves R2 by +0.346, versus +0.070 with the refit baseline, showing its apparent value shrinks when baseline error is corrected.The refit improves absolute fit quality from Huber 0.0158 to 0.00720.Source: A Reanalysis of Muennighoff et al.… · p. 14, Table 4 · p. 15
• At a unique data budget of 25B tokens, refitting reverses the allocation recommendation toward larger models, aligning with the paper’s finding that compute is better spent on capacity than repetition.The published law instead allocates toward smaller models and more epochs than Chinchilla.Source: A Reanalysis of Muennighoff et al.… · p. 14, Figure 9 · p. 15

B.1 Architecture

• Models use Llama 2 architecture with MHA, RoPE, SwiGLU, RMSNorm pre-normalization, Flash Attention 2, torch.compile, and a 32,000-token Llama 2 vocabulary.Source: §B.1 · p. 16

B.2 Model configurations

• The scaling study uses nine model sizes, with four additional configurations reserved exclusively for held-out validation of the fitted scaling law.Source: §B.2 · p. 16, Table 5 · p. 16, Table 6 · p. 16

B.3 Training hyperparameters

• The shared learning rate decays cosine-wise from its peak to 10% of peak over training, with linear warmup during the first 1% of steps.Table 7 summarizes hyperparameters shared across all runs.Source: §B.3 · p. 16, Table 7 · p. 17

B.4 Dataset

• Training uses FineWeb sample-10BT, a 10-billion-token sample of FineWeb.The cited dataset is Penedo et al. (2024).Source: §B.4 · p. 16
• Documents are packed into fixed-length chunks of 512 tokens, separated by end-of-sequence tokens.Source: §B.4 · p. 16

B.5 Evaluation

• Evaluation reports final validation loss over 6,400 validation documents, without early stopping.Source: §B.5 · p. 16

B.6 Infrastructure

• Experiments run one per GPU on NVIDIA B200 or NVIDIA A6000 GPUs.Source: §B.6 · p. 16
• The single-epoch grid uses standard weight decay (λ = 0.1) across 9 model sizes and 8 data budgets; a checkmark denotes a completed run.Source: Table 8 · p. 17

C Experimental grid

• The grid uses Llama 2 on FineWeb with a causal language modeling objective, spanning nine model sizes (15M–1B), eight data budgets (50M–6B tokens), and repetition up to 16 epochs.Multi-epoch runs use RD ∈ {0, 1, 3, 7, 11, 15}.Source: C Experimental grid · p. 17

C.1 Standard study (λ = 0.1)

• The standard-decay study uses a banded single-epoch grid because larger models are compute-constrained, then adds multi-epoch runs at RD = 0, 1, 3, 7, 11, and 15.The single-epoch grid is shown in Table 8 and the multi-epoch cells in Table 9.Source: §C.1 · p. 17

C.2 Strong weight decay study (λ = 1.0)

• In the strong-decay study, larger models and larger budgets for mid-sized models have only single-epoch runs, anchoring the Chinchilla baseline; a separate grid provides multi-epoch cells.The single-epoch grid is shown in Table 10 and the multi-epoch cells in Table 11.Source: §C.2 · p. 17

D Scaling law fit details

• The appendix reports fit estimates, goodness-of-fit metrics, and Muennighoff et al. (2023) C4 external validation; Phase 2 compares five repetition-penalty forms.The forms include two prior-work baselines and three additive penalties.Source: D Scaling law fit details · p. 18, §D.3 · p. 19

D.1 Fitting methodology

• Fits minimize Huber loss in log-space (δ_Huber = 10^-3) with L-BFGS and strong Wolfe line search, using 324 base-law and 7–160 repetition-form initializations.R2 is reported over all points, multi-epoch runs, and single-epoch runs. Phase 1 parameters are locked during Phase 2; RD = epochs − 1, and uncertainties are median absolute deviations across bootstrap resamples.Source: §D.1 · pp. 18–19

D.2 Phase 1: Chinchilla parameters

• Strong weight decay (λ = 1.0) raises the entropy floor E by ~0.2 nats and increases B by 6×, with a steeper β, consistent with reduced effective capacity at fixed parameter count.The Chinchilla base law is fit to single-epoch runs only.Source: §D.2 · p. 19

D.3 Phase 2: Repetition penalty parameters

• For standard weight decay, the one-parameter additive penalty reaches R2_multi = 0.9503 versus 0.5825 for exponential decay; the four-parameter form reaches R2 = 0.9971.Fits use FineWeb data (n = 157 total; n_multi = 81 multi-epoch runs).Source: §D.3 · p. 20
• With strong weight decay, exponential-decay R2_multi is 0.9476 versus 0.5825 under standard decay, while R* is 12.73 versus 7.76, indicating approximately 60% greater repetition tolerance.The four-parameter additive fit recovers γ ≈ 1.02 under strong decay versus 0.64 under standard decay.Source: §D.3 · p. 20

D.4 External validation on Muennighoff data

• On Muennighoff et al. (2023) deduplicated C4 (n = 158, up to 64 epochs), all three additive penalty forms outperform both baselines on multi-epoch runs, and the four-parameter form achieves the best overall fit.The bD, bN form reaches R2_single = 0.9832 because its bN saturation changes predictions for over-parameterized single-epoch runs.Source: §D.4 · pp. 20–21

D.5 FLOPs approximation

• The compute sweep estimates training FLOPs as Ctrain ≈ 6 · N · UD · (1 + RD), then selects the epoch count and implied model size that minimize predicted loss under fixed compute and unique-data budgets.The implied model size is N = Ctrain/(6 · UD · epochs).Source: §D.5 · p. 21

E Downstream evaluation details

• The appendix provides evaluation methodology and task-level results supporting the aggregate OLMES Avg. BPB scores reported in Tables 1 and 2.Source: E Downstream evaluation details · p. 21

E.1 Evaluation setup

• Final checkpoints are evaluated with OLMES using 5-shot reading comprehension, BPB, and a 512-token maximum sequence length.BPB is used instead of accuracy because it remains informative when models perform near chance on multiple-choice tasks (Heineman et al., 2025).Source: §E.1 · p. 21

E.2 Task inventory

• The OLMES Avg. BPB is computed over 19 tasks listed in Table 16.Source: §E.2 · p. 21

E.3 Task-level BPB breakdown at UD = 250M

• Table 17 reports per-task BPB for three matched-compute configurations at UD = 250M and C = 5 × 10^18.Each column uses the configuration recommended by its corresponding scaling law, and bold marks the lowest per-task BPB.Source: §E.3 · p. 21, Table 17 · p. 22

E.4 Per-task BPB at UD = 500M

• Tables 18 and 19 report per-task BPB for the UD = 500M configurations from Table 1.Source: §E.4 · p. 21, Table 18 · p. 23, Table 19 · p. 23

E.5 Weight decay validation: per-task BPB

• Tables 20 and 21 report per-task BPB for all weight-decay validation configurations from Table 2.Source: §E.5 · p. 22