Scaling Data-Constrained Language Models

Motivation

• Chinchilla scaling laws [42] imply that large models require massive datasets, while most languages have far less data and even high-quality English may be exhausted by 2024.A 530-billion-parameter model would require 11 trillion tokens, or more than 30 terabytes of text, under the extrapolated scaling laws.Source: §1 · p. 2

Contributions

• Over 400 models show that up to 4 repeated-data epochs have negligible loss differences and no downstream-performance differences, while further repetition eventually yields zero returns.Single-epoch models consistently achieve the best validation loss per compute, but additional epochs remain beneficial before returns diminish to zero.Source: §1 · p. 2
• The study fits a data-constrained scaling law and finds that optimal scaling requires increasing both parameters and epochs, with epochs growing slightly faster.The law generalizes the Chinchilla scaling law [42] to repeated-data regimes and predicts loss more accurately there.Source: §1 · p. 2

Complementary approaches

• Mixing Python with language data provides a 2× increase in effective tokens for natural-language tasks, while filtering mainly helps noisy datasets.The study evaluates code at 10 mixing rates and revisits perplexity and deduplication filtering on noisy and clean data.Source: §1 · p. 2

• Scaling analysis optimizes held-out cross-entropy loss by allocating a fixed compute budget between parameters N and training tokens D.The central questions are the optimal resource allocation and the expected value of additional resources.Source: §2 · pp. 2–3
• Established scaling laws find that loss follows a compute power law and that resources should be divided roughly equally between parameters and data [42].These conclusions were established empirically through training runs and extrapolation.Source: §2 · p. 3
• Chinchilla [42] predicts scaling through fixed-parameter, fixed-FLOPs, and parametric-fit experiments.The parametric fit models loss as a function of parameters N and training tokens D, using variables learned from the first two approaches.Source: §2 · p. 3
• Because the fitted exponents satisfy α ≈β, Chinchilla [42] concludes that compute-optimal training should scale N and D proportionally.The resulting allocation assigns compute between model parameters and training data according to the fitted exponents.Source: §2 · p. 3
• Chinchilla [42] also validates its allocation conclusions on downstream tasks because held-out loss can imperfectly proxy natural-language performance.The validation covers various downstream tasks.Source: §2 · p. 3

3 Method: Data-Constrained Scaling Laws

• The method modifies prior scaling laws [46] [42] to optimize compute allocation and returns when the available unique training data is limited.It introduces a data-budget constraint and fits the modified law using a large body of experiments.Source: §3 · p. 3
• Unique tokens are UD = min{DC, D}, repetitions are RD = (D/UD) −1, and the method minimizes the original objective subject to data budget DC.Repeated training allocates FLOPs across multiple epochs on the same data; single-epoch training has RD = 0.Source: §3 · p. 3
• The parameter decomposition sets UN to the compute-optimal size for UD unique tokens, then defines excess-parameter repetition as RN = (N/UN) −1.UN is capped by the available model size N, so it represents the optimal parameter allocation or a smaller model when N < Nopt.Source: §3 · pp. 3–4

3.1 Parametric Fit

• The parametric fit replaces D and N with effective data D′ and effective parameters N′, using exponential decay to model the declining value of repeated tokens.The fitted half-life parameter R∗D controls repetition decay; with no repetitions, D′ = UD = D, while extensive repetition drives effective data far below processed tokens.Source: §3.1 · p. 4
• The symmetric effective-parameter formulation captures diminishing returns from models whose sizes vastly exceed available data, alongside diminishing returns from repeated tokens.The framework is evaluated through fixed-unique-data, fixed-FLOPs, and parametric-fit protocols targeting allocation, return, and predictive capability.Source: §3.1 · p. 4, §3 · p. 4

Model and dataset setup

• Experiments train GPT-2 models on nested C4 subsets [90], scaling to 8.7 billion parameters and 900 billion total tokens.Runs repeat the entire available dataset, shuffle data after each epoch, and ensure lower-data runs use subsets of higher-data runs.Source: §4 · p. 5, Figure 2 · p. 5

Evaluation

• Held-out test loss is reported to capture extreme overfitting from repeated data rather than relying on training loss.The held-out data comes from the same underlying dataset, so this evaluation choice is not expected to alter the findings.Source: §4 · p. 5

Optimization and training protocol

• Models use cosine learning-rate schedules decaying 10× during training, with other hyperparameters based on prior work [89] [42].Early stopping is omitted to examine overfitting when data are repeated.Source: §4 · p. 5

Experimental Setup

• Experiments fix unique-data budgets at 100M, 400M, or 1.5B tokens and vary compute allocation between model parameters and epochs.Figure 3’s main results use the 100M-token constraint, with larger budgets reported in Appendix C.Source: §5 · p. 5

Fixed Data Constraints

• More than 50% lower loss is achieved by combining repeated-data training with 20–60× more parameters and epochs, at roughly 7000× more FLOPs.The one-epoch compute-optimal model substantially under-utilizes the available data.Source: §5 · p. 6

Data-Constrained Scaling Laws

• Fitting 182 runs shows that, beyond one epoch, excess parameters lose value faster than repeated data, so efficient allocation favors scaling epochs faster than parameters.This differs from Chinchilla’s equal scaling recommendation, whose fit assumes single-epoch training.Source: §5 · p. 6

Large-Scale Validation

• At 9.3 × 1021 FLOPs and 25 billion unique tokens, the data-constrained compute-optimal model uses 27% fewer parameters yet achieves better loss and downstream performance than the Chinchilla-suggested model.The result is reported in Figure 1 (right) and Table 8.Source: §5 · p. 6

Limitations

• Loss decreases with added parameters and epochs but can rise again when compute becomes excessive, a behavior not modeled by the proposed scaling-law contours.The authors suggest appropriate regularization could prevent harm from excess parameters or epochs.Source: §5 · p. 7

Return on repeated data

• 0.5% higher validation loss resulted from four epochs versus one for an 8.7B-parameter model, despite using 44B versus 178B unique tokens.Repeated data is worth less, but the loss difference across a few epochs remains negligible.Source: §6 · p. 7

Scaling-law fit

• The data-constrained scaling laws accurately measured the decay in repeated-data value, with empirical results closely tracking the parametric fit.The comparison used final test loss across experiments spanning three FLOP budgets and eight data budgets.Source: §6 · p. 7
• The fit significantly underestimated final test loss for failing models whose loss increased midway through training, including models trained for 44 epochs.These failing models were not depicted in the figure.Source: §6 · p. 7

Return on repeated data

• R*_D ≈15 marks the repeated-data half-life, corresponding to 15 repetitions or 16 epochs, after which returns diminish extremely fast.The half-life is defined as the point where repeated tokens have lost 1/e of their value, with diminishing returns flattening soon after 16 epochs.Source: §6 · p. 7

Methods

• The compared strategies augment missing natural-language data with shuffled Python code from The Stack [49] or expand availability by removing deduplication and perplexity filters.These approaches target improved downstream performance rather than directly minimizing loss.Source: §7 · p. 8

Experimental setup

• The experiments compare strategies under an 84-billion-token maximum budget, with filtering methods starting from approximately 178 billion noisy tokens.Perplexity filtering retains 44 billion tokens repeated nearly twice, while deduplication retains 21 billion tokens repeated four times.Source: §7 · p. 8

Evaluation

• Because strategies use different data distributions, models are evaluated on 19 natural-language tasks with zero to five in-context exemplars, producing 114 rescaled scores per model.Scores are rescaled before averaging to accommodate differing metrics and random baselines.Source: §7 · p. 8

Results

• Repeating data shows insignificant downstream differences through roughly 4 epochs, while code can replace up to 50% of training data without deterioration; beyond that, performance declines quickly.WebNLG [34] and bAbI [122] [57] improve when code is added, possibly because code supports long-range state tracking.Source: §7 · p. 8
• Perplexity filtering improves downstream performance, whereas deduplication does not help on the benchmark.Deduplication may still reduce memorization [45] [40] [16] [10], and filtering is more effective on a noisier dataset.Source: §7 · p. 8
• Combining code augmentation and repetition can yield 8× more training tokens expected to match 8× more unique data, by doubling data with code and repeating four epochs.The authors recommend reserving filtering for noisy datasets while using code augmentation and repetition to increase data tokens.Source: §7 · pp. 8–9

Large language models

• Large language models are commonly dense or sparse transformers trained for next-token prediction and evaluated through held-out loss or zero- to few-shot prompting.The paper focuses on dense transformers trained on large corpora and evaluated directly after pre-training.Source: §8 · p. 9

Scaling laws

• Scaling-law research has proposed different compute allocations, while prior up-sampling work found that repeating 0.1% of training data 100 times significantly degrades performance.This work instead studies repeating the entire pre-training corpus for multiple epochs.Source: §8 · p. 9

Alternative data strategies

• Prior data strategies filter web corpora, mix languages, and include limited code, but generally lack ablations of code’s drawbacks and natural-language benefits; this study benchmarks Python–natural-language mixing at 10 rates.Earlier studies reported that mixing programming or natural languages can underperform monolingual models, motivating a detailed mixing analysis.Source: §8 · p. 9

9 Conclusion

• The study focuses on optimally using compute when unique training data is limited.Source: §9 · p. 9
• The proposed Chinchilla-law extension models the declining value of repeated data using controlled experiments.Source: §9 · p. 9
• Multi-epoch training with repeated data is beneficial, and scaling laws continue to hold despite diminishing returns.Source: §9 · p. 9
• Code data enables language models to scale an additional 2×.Source: §9 · p. 9
• The findings may support further language-model scaling and unlock new capabilities with current data.Source: §9 · p. 9
• The results indicate limits on the scaling horizon, motivating both additional data collection and more effective use of existing data.Source: §9 · p. 9

A Derivation of Data-Constrained Scaling Laws

• Equation 14 generalizes Chinchilla scaling by exponentially decaying the value of repeated tokens and excess parameters, while reducing to the single-epoch law when repetitions vanish.The learned repetition scales mark where additional tokens or parameters yield sharply diminishing returns.Source: A Derivation of Data-Constrained… · p. 22
• The fitted model shows that both data and parameter decay are necessary, while Equation 14, Equation 10, and explicit exponential decay achieve R2 around 80.Equation 14 fits slightly worse than Equation 10 but is used because it has fewer terms and is easier to interpret.Source: A Derivation of Data-Constrained… · p. 23, Table 1 · p. 23

A.1 Analytical properties of compute-optimal point

• Compute-optimal scaling initially increases parameters and repeated tokens proportionally, then shifts toward processing more tokens when added parameters enter diminishing returns.At equal computational cost, each added parameter becomes less valuable than an added data point after the repetition thresholds are approached.Source: §A.1 · p. 24, Figure 7 · p. 24

B C4 Scaling Coefficients

• For a Gopher budget of 5.76 × 10^23 FLOPs, the C4 fit predicts Nopt = 70.0 billion parameters and Dopt = 1.37 trillion tokens, close to IsoFLOP’s 73 billion and 1.3 trillion.The authors therefore use the fitted C4 coefficients rather than MassiveWeb coefficients for Chinchilla-based computations.Source: B C4 Scaling Coefficients · pp. 24–25

C Additional Contour Plots

• Figure 8 shows that lower loss can result from exceeding single-epoch compute-optimal parameters and epochs, with extra compute favoring repeated data over added parameters.The contours cover 400 million and 1.5 billion unique-token datasets.Source: C Additional Contour Plots · p. 25, Figure 8 · p. 25

D Double Descent

• Models trained on 100 million tokens exhibit epoch-wise double descent: loss rises at 200 epochs before declining, adding noise because the fitted form assumes monotonic improvement.Most such examples are removed from the fitting.Source: D Double Descent · pp. 25–26, Figure 9 · p. 26

E Repeating on Heavily Deduplicated Data

• On heavily deduplicated C4, the optimal epoch count remains 59 and the overall trend closely matches regular C4 despite higher test loss after deduplication.Together with OSCAR results, this suggests the findings generalize across datasets with different inherent duplication levels.Source: E Repeating on Heavily Deduplicated Data · p. 26, Figure 10 · p. 26

F Do Excess Parameters Hurt, Plateau or Help?

• The authors retain an exponential-decay formulation that predicts plateauing rather than worsening excess parameters or epochs, because harmful training can be stopped or excess parameters removed.An alternative alpha-beta decay formulation allows eventual harm but incorrectly predicts slower diminishing returns and lacks rigorous mathematical justification.Source: F Do Excess Parameters Hurt, Plateau… · p. 27
• With 100 million tokens, models above 2 billion parameters have significantly higher validation loss than 200-million- to 1-billion-parameter models, contrasting with Chinchilla’s continued improvement as parameters increase.Figure 11 shows the empirical losses, default scaling prediction, and Chinchilla prediction; µP performs worse than the default hyperparameters.Source: F Do Excess Parameters Hurt, Plateau… · p. 27, Figure 11 · p. 28

G Case Study: Galactica

• For Galactica, data-constrained scaling predicts that a 40-billion-parameter model trained on 1.35 trillion tokens, or 12.75 epochs, would be optimal instead of the 120-billion-parameter allocation.The exact optimum may differ because the scaling laws were fitted on C4 rather than Galactica’s predominantly scientific pretraining data.Source: Figure 13 · p. 29, G Case Study: Galactica · p. 29

H Training Loss; I Scaling Curves on the OSCAR Corpus

• Training loss is unsuitable for repeated-data comparisons because models overfit limited data, so the study uses held-out test loss; on OSCAR, fewer repeats perform better but differences for a few repeats are insignificant.OSCAR is considered noisier than C4 because it applies less stringent deduplication.Source: H Training Loss · p. 30, Figure 14 · p. 30, I Scaling Curves on the OSCAR Corpus · p. 30

J Validation Loss by Epoch

• Validation-loss spikes at new epochs are temporary, so continuing beyond Galactica’s fifth-epoch spike could have improved performance.The study observes such spikes in the third and fourth epochs for a 7-epoch, 8.7-billion-parameter OSCAR model.Source: J Validation Loss by Epoch · pp. 31–32, Figure 17 · p. 31

K Evaluation Details; N Filtering Procedure

• Evaluation uses final test loss on fixed C4 validation tokens, held-out OSCAR validation data, and normalized averages across 19 datasets with 0–5 few-shots, while filtering includes perplexity selection, suffix-array deduplication, and ROOTS filters.The filtering procedure removes documents with insufficient length or grammatical words, excessive repetition or special characters, flagged words, low fastText language-identification scores, or high perplexity.Source: K Evaluation Details · p. 32, Table 2 · p. 32, Table 3 · p. 33

L Downstream Repetition Results

• Up to 4 epochs of repeated data produce no significant downstream differences, while C4 generally outperforms noisier OSCAR and fails on bAbI whereas OSCAR exceeds random.The comparison covers final checkpoints across the reported C4 and OSCAR configurations.Source: L Downstream Repetition Results · p. 33, Table 4 · p. 33, Table 5 · p. 34

M Detailed Code Augmentation Results

• Replacing up to 50% of C4 with Python maintains natural-language performance and yields huge bAbI gains, whereas OSCAR gains are less clear because it already contains code.These results are reported for 4.2B- and 2.8B-parameter models.Source: M Detailed Code Augmentation Results · p. 36, Table 10 · p. 36, Table 11 · p. 37

O Detailed Filtering Results

• Perplexity filtering improves C4 only at 4.2B parameters, but benefits OSCAR at both 2.8B and 4.2B parameters.The comparisons are reported in Table 13 for datasets repeated to fixed token budgets.Source: O Detailed Filtering Results · p. 38, Table 13 · p. 39
• Deduplication does not improve C4 downstream performance but is effective for noisier OSCAR.Table 14 reports the deduplication comparisons under fixed-token training budgets.Source: O Detailed Filtering Results · p. 38, Table 14 · p. 39
• ROOTS filtering is not better than unfiltered OSCAR on the benchmark, although it may reduce obscenity, templated messages, or repetition for particular use cases.The possible benefits are presented as use-case-dependent effects rather than benchmark gains.Source: O Detailed Filtering Results · p. 38, Table 14 · p. 39

P Loss Curves for Complementary Strategies

• Downstream natural-language performance is preferred over validation loss for comparing complementary data strategies because validation loss favors models trained on more same-distribution data.This avoids penalizing models that replace missing natural-language data with code.Source: P Loss Curves for Complementary… · p. 40
• The model trained on 90% Python and 10% C4 has the highest C4 validation loss but the lowest Python validation loss.This illustrates how validation loss changes with the fraction of data drawn from each distribution.Source: P Loss Curves for Complementary… · p. 40, Figure 19 · p. 40
• Deduplicated and perplexity-filtered models have higher validation loss on unfiltered held-out data because the validation distribution does not match their filtered training distributions.The unfiltered validation set more closely resembles the training distribution of unfiltered models.Source: P Loss Curves for Complementary… · p. 40, Figure 20 · p. 40
• Perplexity-filtered data yields the lowest training loss, whereas deduplicated data yields the highest validation and training loss.Filtering to the lowest-perplexity 25% removes high-loss examples, while deduplication removes repeated sequences that other models may memorize.Source: P Loss Curves for Complementary… · pp. 40–41, Figure 20 · p. 40

Q Limitations and Future Work

• Future work should adapt scaling laws for partially repeated data and test sensitivity to learning rate, dropout, and optimizer choices.Repeating data earlier may cause less damage, while higher learning rates may make diminishing returns from extra epochs appear earlier.Source: Q Limitations and Future Work · p. 41
• The findings may not generalize universally across datasets, modalities, architectures, fine-tuning, or untested mitigation strategies such as language augmentation and popularity or toxicity filtering.C4 and OSCAR showed overall agreement, but deduplication was more effective on noisier OSCAR; future work should test broader settings.Source: Q Limitations and Future Work · p. 41

R Contributions

• The authors assign distinct roles spanning experiments, analysis, writing, framing, data preparation, preprocessing, hardware enablement, and experimental guidance.Muennighoff led the project, while the other contributors supported analysis, writing, data choices, datasets, AMD training, and design.Source: R Contributions · p. 41

S Hyperparameters and Setup

• Training uses cosine-decayed learning rates, size-dependent batch sizes, bfloat16 Adam optimization, and GPT-style architectures with explicitly calculated parameter counts.Models were trained with up to 256 AMD GPUs and publicly released ROCm-enabled code; the broader effort used around 3 million GPU hours.Source: S Hyperparameters and Setup · p. 42

T Prompts and Samples

• Evaluation prompts stem from PromptSource [5] or GPT-3, use ground-truth datasets without generations, and include formatted examples for ANLI R1 (accuracy), E2E NLG (ROUGE), WikiLingua (ROUGE), and bAbI (exact match).Source: T Prompts and Samples · p. 44, Figure 21 · p. 44, Figure 35 · p. 47

U Other Experiments

• UL2 [106] [107] did not outperform regular causal language modeling, whereas models trained on The Pile [31] showed trends similar to OSCAR and C4.The Pile models are publicly available, while the UL2 result may reflect encoder-decoder suitability or implementation mistakes.Source: U Other Experiments · p. 50

W Version Control

• Version-control updates document corrected scores and plots, added fit comparisons and experiments, expanded methodological details, and recorded the hyperparameter-sensitivity limitation.The revision history also notes additions concerning complementary strategies, Galactica, UN calculations, score normalization, and model-head corrections.Source: W Version Control · p. 50

X Broader Impacts

• Releasing the models may amplify offensive language, social bias, and privacy harms [119], but the authors judge the research and release benefits to outweigh these downsides given larger models already available [13] [95] [11].The impact assessment explicitly balances potential misuse against the comparative availability of more capable open models.Source: X Broader Impacts · p. 50