PaperlayerView exact revision on arXiv

Source-linked AI summary

Prescriptive Scaling Laws for Data Constrained Training

Justin Lovelace, Christian Belardi, Srivatsa Kundurthy, Shriya Sudhakar, Kilian Q. Weinberger

arXiv:2605.01640v1cs.LGcs.CL

TL;DR

As training compute outpaces high-quality data, existing scaling laws inadequately model overfitting from repeated tokens. This paper adds an additive overfitting penalty and finds that strong weight decay (λ = 1.0) reduces it by approximately 70%.

  • Problem

    Existing scaling laws assume unique tokens or model diminishing returns without capturing loss increases from overfitting under repetition and model-size interactions.

  • Method

    The paper augments the Chinchilla law with an additive penalty for repeated tokens that grows with repetition and model capacity relative to unique data.

  • Results

    Strong weight decay (λ = 1.0) reduces the overfitting penalty by approximately 70%, while the law recommends scaling model size and reducing epochs beyond a compute threshold.

  • Takeaways & Limitations

    The scaling law quantifies robustness to data repetition and provides prescriptive allocation guidance for data-constrained training configurations.

  • Takeaways & Limitations

    The study covers models up to 1B parameters and 16 epochs, so the fitted exponents and law may not extrapolate to frontier scales or extreme repetition.

Abstract

from arXiv · show

Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law assumes every training token is unique. This limits its ability to guide pretraining decisions in data-constrained regimes. We model the excess loss under repetition with a simple additive overfitting penalty and find that it accurately describes model behavior. Our scaling law yields qualitatively new compute-optimal allocation advice. Beyond a point, further repetition is counterproductive and compute is better spent on model capacity. We show that following our law's recommended configuration improves performance in data-constrained regimes. Finally, because our one-parameter form isolates overfitting in a single coefficient, it enables direct comparison across training configurations. As a case study, we show that strong weight decay ($λ=1.0$) reduces this coefficient by approximately 70%, providing a scaling-law explanation for recent findings that optimal weight decay in data-constrained regimes is an order of magnitude larger than standard practice.

1 Introduction

As high-quality data becomes the bottleneck for scaling, existing repetition-aware laws cannot capture overfitting or its interaction with model size. The paper introduces an additive overfitting penalty for repeated tokens and evaluates it across a broad model and data range.

  • Motivation: Data, not compute, is increasingly the bottleneck as training compute outpaces high-quality data supply.The constraint is stronger in specialized domains such as mathematics, code, and low-resource languages.
  • Problem: Existing repetition extensions can model diminishing returns but not loss increases from overfitting or its interaction with model size.Larger models overfit faster on repeated data.
  • Method: The proposed law treats repeated tokens as useful while adding a separate overfitting penalty that grows with repetitions, limited data, and model size.This additive formulation encodes the intuition that overfitting worsens with limited data and larger models.
  • Evaluation: 300 models spanning 15M–1B parameters, 50M–6B unique tokens, two weight decay strengths, and up to 16 epochs support the study.The experiments cover multiple model, data, regularization, and repetition scales.
  • Contribution: The central contribution is an additive overfitting penalty that augments the Chinchilla law with a simple repetition term.A 1-, 2-, and 4-parameter complexity ladder traces a fit-quality-versus-complexity Pareto frontier.

2 Background and Related Work

Neural scaling laws relate model size and training data to loss and compute, with Chinchilla selecting a compute-optimal allocation under unique-token training. Repetition violates that assumption: prior effective-data and effective-model formulations capture diminishing returns but miss repetition-induced overfitting, motivating an explicit penalty.

  • Neural scaling laws: Neural scaling laws predictably relate held-out loss to model size and dataset size, guiding how training choices affect compute and performance.Kaplan et al. established these trends, while Hoffmann et al. refined them into the Chinchilla scaling law.
  • Neural scaling laws: Chinchilla expresses loss as irreducible uncertainty plus penalties for insufficient model size and insufficient training data.Its parameters include N, the model parameter count, and D, the number of training tokens.
  • Neural scaling laws: For fixed compute, Chinchilla selects a model-size and dataset-size pair that minimizes predicted loss, using C ≈6ND.The optimal N:D ratio depends on fitted exponents, datasets, and training setups.
  • Scaling under data repetition: Chinchilla assumes every training token is unique, but practical training often repeats scarce high-quality data across multiple epochs.Repeated passes yield progressively diminishing returns and can eventually hurt performance through overfitting.
  • Scaling under data repetition: Prior repetition-aware laws replace raw data with saturating effective data and model capacity, yet existing formulations underpredict loss when repetition is high.The Muennighoff et al. formulation also makes repetition depend on model size, but treats overfitting indirectly and assumes analogous exponential saturation.

3 Experimental setup

The experiments pretrain Llama 2 decoder-only language models on FineWeb across model sizes, unique data budgets, and repetition counts. The study spans 15M–1B parameters, 50M–6B unique-data tokens, and six repetition settings.

  • Model and data: The study pretrains decoder-only language models using the Llama 2 architecture and tokenizer on the FineWeb filtered web corpus.FineWeb is described as a large-scale filtered web corpus.
  • Experimental grid: 15M–1B parameters define the swept model-size range.Model sizes are denoted N.
  • Experimental grid: 50M–6B tokens define the swept unique-data budget range.Unique data budgets are denoted UD.
  • Experimental grid: {0, 1, 3, 7, 11, 15} defines the six repetition counts evaluated.Repetition counts are denoted RD.

4 Scaling laws for repeated data

The effective-data approach increasingly underpredicts validation loss as repetition rises, motivating an additive overfitting penalty. Repetition remains useful for data sufficiency but incurs a growing, configuration-dependent cost that can be superlinear.

  • Limitations of the effective-data approach: Validation-loss prediction gaps grow with repetition, so effective-data formulations systematically underpredict loss at high epoch counts.The comparison spans four model sizes under the Chinchilla baseline and effective-data forms.
  • An additive overfitting penalty: Superlinear repetition damage means each additional repetition epoch inflicts more loss than the previous one.The fitted shared exponent satisfies δ > 1.0 across configurations.
  • A complexity ladder of penalty forms: Larger models and smaller unique data budgets incur steeper overfitting penalties.This configuration dependence motivates increasingly complex additive penalty forms.
  • A complexity ladder of penalty forms: The linear-in-RD penalty substantially outperforms the Muennighoff et al. (2023) formulations.A second parameter, κ, allows the penalty to scale nonlinearly with model capacity relative to unique data.
  • A complexity ladder of penalty forms: κ > 1 indicates that overfitting grows superlinearly with model capacity relative to unique data.The full form additionally introduces δ for repetition count and separates γ from κ.
  • A complexity ladder of penalty forms: At RD = 0, every penalty form exactly reduces to Chinchilla, while repeated tokens both improve data sufficiency and add overfitting cost.This dual role distinguishes the approach from treating repeated tokens solely as fresh effective data.

5 Scaling law validation

The additive overfitting scaling laws accurately fit observed loss across independent sweeps and external data, including degradation under heavy repetition that effective-data formulations miss. Their prescriptive recommendations also improve performance by favoring fewer epochs, while producing qualitatively different compute-optimal allocations from prior laws.

  • Scaling law fit: The additive penalty laws are validated against Chinchilla and Muennighoff et al. formulations on two independent scaling sweeps.The second sweep uses public Muennighoff et al. (2023) data, including runs up to 64 epochs.
  • Scaling law fit: The proposed law tracks observed loss across all configurations and captures degradation under heavy repetition that effective-data formulations miss.Figure 3 compares predicted and observed loss across model sizes.
  • Scaling law fit: The one-parameter additive penalty substantially outperforms competing forms, while the four-parameter form achieves near-perfect fit on the authors’ data.The improvement extends to held-out Muennighoff et al. (2023) data spanning wider model-size and repetition ranges.
  • Compute-optimal allocation: The superlinear repetition penalty yields qualitatively different compute-optimal allocation advice from prior scaling laws.Chinchilla always recommends more repetition, while the Muennighoff et al. form prescribes diminishing returns without stopping repetition.
  • Prescriptive validation: The proposed law consistently recommends models with fewer epochs and achieves the best perplexity and downstream performance across all tested settings.Prescriptive validation trains the configurations recommended for each token-budget and compute-budget pair.
  • Generalization to external data: The additive penalty forms significantly outperform the Muennighoff et al. (2023) formulation on completely held-out published data.The external data uses different backbone architectures, tokenization, and other pretraining choices.

6 Case study: weight decay improves robustness to data repetition

Strong weight decay (λ = 1.0) reduces overfitting from data repetition by approximately 70%, overcoming its single-epoch loss premium at sufficiently high compute. Held-out validation confirms that the scaling law recommends stronger weight decay and more training epochs when data constraints bind.

  • Setup: The study compares standard λ = 0.1 and strong λ = 1.0 weight decay across the same model-size and data-budget grid.All other hyperparameters are held constant, and the overfitting coefficient quantifies robustness to repetition.
  • Single-epoch scaling: Strong weight decay incurs a loss premium at every compute budget in the single-epoch regime and favors larger models relative to data.Separate Chinchilla parameters are fitted for each weight-decay setting.
  • Scaling under data repetition: Approximately 70%: strong weight decay reduces the fitted overfitting coefficient P, meaning substantially less overfitting per repetition.The overfitting penalty grows superlinearly under both settings but has significantly lower magnitude with strong weight decay.
  • Crossover under repetition: At UD = 250M, strong weight decay overtakes standard weight decay at C ≈3.2 × 10^18 FLOPs as the standard setting’s steeper repetition penalty erodes its initial advantage.Standard weight decay performs better at modest compute despite strong weight decay’s lower overfitting cost.
  • Prescriptive validation: 2.8 points: at C = 1 × 10^19, strong weight decay reduces perplexity relative to standard weight decay near three times past the crossover.Held-out configurations show comparable performance near the predicted crossover and a widening advantage beyond it.
  • Prescriptive validation: 16.65 vs. 18.16 perplexity: at 3 × 10^19, strong weight decay achieves lower perplexity, while the law recommends training for more epochs under binding data constraints.This supports using stronger weight decay in data-constrained regimes.
  • Limitations: The study is limited to models up to 1B parameters, repetition up to 16 epochs, two weight-decay values, and scaling-law behavior that may not extrapolate to extreme regimes.The law also does not capture phenomena such as double descent.

7 Conclusion

The paper presents a simple data-constrained scaling law that models data repetition through an additive overfitting penalty. Its complexity ladder improves fit over prior effective-data formulations while quantifying robustness to repetition.

  • The proposed scaling law models the cost of data repetition with a simple, additive overfitting penalty.
  • A complexity ladder of one-, two-, and four-parameter forms defines a Pareto frontier balancing fit quality and complexity.
  • Even the simplest form substantially outperforms prior effective-data formulations.
  • The overfitting penalty provides a new axis for evaluating training configurations by directly quantifying robustness to data repetition.

A Reanalysis of Muennighoff et al. compute-optimal allocation … B.2 Model configurations

Reanalysis shows that correcting the Chinchilla baseline removes the apparent disagreement over compute-optimal allocation: the fitted repetition mechanism disables itself and favors larger models trained for fewer epochs. The study uses Llama 2–style models and nine scaling-study configurations, plus four validation-only configurations.

  • A Reanalysis of Muennighoff et al. compute-optimal allocation: 71.1% of variance is explained by the published Chinchilla parameters on Muennighoff et al.’s single-epoch data, versus 98.9% after refitting.The comparison uses 29 single-epoch runs, where repetition is absent and a Chinchilla law should apply.
  • A Reanalysis of Muennighoff et al. compute-optimal allocation: +0.346 is the bD, bN mechanism’s R2 improvement with the published base, compared with +0.070 with the refit base.The corresponding R2 changes are 0.445 to 0.791 and 0.861 to 0.931, while Huber improves from 0.0158 to 0.00720 with the refit base.
  • A Reanalysis of Muennighoff et al. compute-optimal allocation: N = 5.31 under the published base makes effective model size saturate quickly, biasing allocation toward smaller models.Under the refit base, the corresponding rate constant is effectively infinite, with R∗N = 3,294.
  • A Reanalysis of Muennighoff et al. compute-optimal allocation: N = 3,294 under the refit base effectively disables the bN mechanism, yielding larger models trained for fewer epochs.Without bN pulling allocation toward smaller models, the refit law aligns with the independent finding that capacity is preferable to repetition.
  • A Reanalysis of Muennighoff et al. compute-optimal allocation: At a unique data budget of 25B tokens, the published law favors smaller models and more epochs, whereas the refit law favors larger models.The reversal explains the original disagreement as bN absorbing systematic error from a misspecified Chinchilla baseline.
  • B.1 Architecture: All models use the Llama 2 architecture with MHA, RoPE, SwiGLU, RMSNorm pre-normalization, Flash Attention 2, torch.compile, and a 32,000-token vocabulary.The tokenizer is the Llama 2 tokenizer.
  • B.2 Model configurations: Nine model sizes are used in the scaling study, with four additional configurations reserved exclusively for held-out validation.Tables 5 and 6 list the scaling-study and validation-only configurations, respectively.

B.3 Training hyperparameters … D.3 Phase 2: Repetition penalty parameters

The appendix specifies the training setup, experimental grids, fitting methodology, Chinchilla baselines, and repetition-penalty comparisons used to analyze data-constrained training. Additive penalties fit repetition effects especially well, while strong weight decay makes repetition more tolerable.

  • B.3 Training hyperparameters; B.4 Dataset; B.5 Evaluation: The training uses cosine learning-rate decay with linear warmup, FineWeb sample-10BT packed into 512-token chunks, and final validation loss over 6400 documents without early stopping.The learning rate decays to 10% of peak over training, with warmup over the first 1% of steps.
  • B.6 Infrastructure; C Experimental grid: All runs use one NVIDIA B200 or A6000 GPU, the Llama 2 architecture, FineWeb, and a causal language-modeling objective across nine model sizes and eight data budgets.The grid covers 15M–1B parameters, 50M–6B tokens, and up to 16 epochs.
  • C.1 Standard study (λ = 0.1); C.2 Strong weight decay study (λ = 1.0): The standard study includes single-epoch and multi-epoch runs at repetition levels RD = 0, 1, 3, 7, 11, 15, with larger models restricted to smaller budgets.The strong-weight-decay study similarly includes single-epoch anchoring runs and multi-epoch cells.
  • D Scaling law fit details; D.1 Fitting methodology: Fits minimize Huber loss in log-space with L-BFGS, use initialization grid search, report three R2 variants, and lock Phase 1 parameters during Phase 2.Repetition is parameterized as RD = epochs −1, and uncertainties use median absolute deviation across bootstrap resamples.
  • D.2 Phase 1: Chinchilla parameters: Strong weight decay (λ = 1.0) raises the entropy floor E by ∼0.2 nats and increases B by 6× with a steeper data exponent β.The baseline law is fit to single-epoch runs only.
  • D.3 Phase 2: Repetition penalty parameters: Phase 2 compares two prior-work baselines with three additive penalty forms, including a one-parameter penalty C · RD · (N/UD) and generalized multi-parameter variants.The exponential baseline models effective data as bD = UD · (1 + R∗· (1 −exp(−RD/R∗))).
  • D.3 Phase 2: Repetition penalty parameters: R2 multi = 0.9503 for the one-parameter additive penalty versus 0.5825 for exponential decay under standard weight decay, while the four-parameter additive penalty reaches R2 = 0.9971.These fits use 157 total FineWeb runs, including 81 multi-epoch runs.
  • D.3 Phase 2: Repetition penalty parameters: Under strong weight decay, exponential decay reaches R2 multi = 0.9476, the four-parameter additive fit gives γ ≈1.02 versus 0.64 under standard decay, and R∗ = 12.73 versus 7.76.The authors interpret these differences as more tolerable repetition and regularization-dependent data-scaling behavior.

D.4 External validation on Muennighoff data … E.4 Per-task BPB at UD = 500M

The paper externally validates its repetition-penalty scaling law on deduplicated C4 data, specifies compute-optimal allocation under fixed compute and unique-data budgets, and details downstream BPB evaluation across 19 tasks. These sections cover matched-compute comparisons at unique-data budgets of 250M and 500M.

  • D.4 External validation on Muennighoff data: The repetition-penalty forms generalize to Muennighoff et al.’s deduplicated C4 dataset, refit across 158 models trained for up to 64 epochs.The Phase 1 Chinchilla fit uses E = 1.9031, A = 432.63, α = 0.3362, B = 5360.24, and β = 0.3868.
  • D.4 External validation on Muennighoff data: On multi-epoch runs, all three additive penalty forms outperform both baselines, while the four-parameter form achieves the best overall fit.The bD, bN form attains R2_single = 0.9832 because bN saturation modifies predictions for over-parameterized single-epoch runs.
  • D.5 FLOPs approximation: Compute-optimal allocation sweeps candidate epoch counts, derives N = Ctrain/(6 · UD · epochs), and selects the configuration with the lowest predicted loss.The FLOPs approximation is Ctrain ≈ 6 · N · Dtotal = 6 · N · UD · (1 + RD).
  • E Downstream evaluation details: Downstream evaluation uses final checkpoints under the Open Language Model Evaluation System, supporting the aggregate OLMES Avg. BPB scores.The appendix provides evaluation methodology and task-level results for the aggregate scores reported in Tables 1 and 2.
  • E.1 Evaluation setup: The downstream protocol uses 5-shot reading comprehension, BPB, and a maximum sequence length of 512 tokens.BPB is used instead of accuracy because it remains informative when models perform near chance on multiple-choice tasks.
  • E.2 Task inventory: The OLMES Avg. BPB is computed from 19 downstream tasks.The task inventory is listed in Table 16.
  • E.3 Task-level BPB breakdown at UD = 250M: Per-task BPB is reported for three matched-compute configurations at UD = 250M and for configurations at UD = 500M.The UD = 250M comparison uses C = 5 × 1018, with each column training the configuration recommended by its corresponding scaling law.

E.5 Weight decay validation: per-task BPB

The section reports per-task BPB for weight-decay validation configurations, comparing an additive-penalty model with Chinchilla and effective-parameter baselines. At UD = 500M and C = 1 × 1019, the reported lambada and average BPB values differ across the three configurations.

  • Weight decay validation: Tables 20–21 report per-task BPB for all weight decay validation configurations from Table 2.
  • Weight decay validation: At UD = 500M and C = 1 × 1019, Table 18 compares Add. Penalty (2B), Chinchilla (700M), and Eff. Param. (950M).
  • Weight decay validation: 0.8324 is the lambada BPB for Eff. Param. (950M), versus 0.8422 for Chinchilla (700M) and 0.8431 for Add. Penalty (2B).
  • Weight decay validation: 1.3380 is the Average BPB for Add. Penalty (2B), versus 1.3676 for Chinchilla (700M) and 1.4009 for Eff. Param. (950M).
Loading 2605.01640v1…