PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Data-Constrained Language Models

Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, Colin Raffel

arXiv:2305.16264v5cs.CLcs.AIcs.LG

TL;DR

As unique text becomes limited, it is unclear how language models should allocate compute between model size and repeated data. The paper fits data-constrained scaling laws to over 400 experiments and finds that multiple epochs help, but repeated data has diminishing value.

  • Problem

    With limited unique data, the best allocation and return of computational resources for scaling language models remain unclear because prior laws assume unlimited data.

  • Method

    The authors train over 400 models under varied data and compute constraints, then fit a scaling law accounting for the declining value of repeated tokens.

  • Results

    Multiple epochs of repeated data improve scaling with diminishing returns, while adding code enables scaling an additional 2×.

  • Takeaways & Limitations

    Under data constraints, computational allocation should favor increasing epochs faster than parameters, contrasting with single-epoch compute-optimal scaling.

  • Takeaways & Limitations

    The proposed scaling law does not model the possibility that excess epochs or parameters can hurt performance.

Abstract

from arXiv · show

The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.

1 Introduction

This work examines how repeated data affects language-model scaling when training data is constrained. It uses over 400 empirical training runs to develop a data-constrained scaling law and evaluates code augmentation and relaxed filtering as complementary strategies.

  • Motivation: Compute-optimal scaling can favor smaller models trained on more data, as shown by Chinchilla outperforming Gopher at a similar compute budget.Chinchilla has 70 billion parameters, while Gopher has 280 billion parameters and uses four times less training data.
  • Research focus: The study quantifies how multiple epochs of repeated data affect large-language-model scaling in data-constrained regimes.Most prior large language models were trained for a single epoch, whereas Galactica models were trained for 4.25 epochs.
  • Experimental program: More than 400 models spanning 10 million to 9 billion parameters and up to 1500 epochs provide data for fitting a new data-constrained scaling law.The experiments record final test loss under varying data and compute constraints.
  • Mitigating data scarcity: The paper evaluates code-token incorporation and relaxed data filtering as ways to improve downstream accuracy without adding new natural-language data.These approaches complement data repetition under data constraints.

2 Background

This section frames LLM scaling as allocating compute between model parameters and training data while predicting both optimal allocation and the returns from additional resources. It reviews empirical power-law scaling and Chinchilla’s methods for deriving compute-optimal training, which suggest proportional growth of parameters and data.

  • Scaling objectives: Scaling analysis asks how to optimally allocate training resources and what additional resources are expected to return.For LLMs, the resource is compute, measured in FLOPs.
  • Scaling objectives: Compute can be allocated between training a larger model and training for more steps, with parameters N and data D as allocation targets.The objective is to minimize held-out-data loss subject to a compute constraint C.
  • Existing scaling laws: Established scaling practice finds that loss follows a compute power law and that resources are divided roughly equally between parameters and data.These laws were established empirically by training models and extrapolating their behavior.
  • Chinchilla methodology: Chinchilla estimates scaling behavior using fixed-parameter, fixed-FLOPs, and parametric-fit experiments.The parametric fit models loss as a function of parameters N and training tokens D.
  • Compute-optimal allocation: Because the fitted exponents satisfy α ≈β, Chinchilla concludes that parameters N and data D should scale proportionally for compute-optimal training.The conclusions are also validated on downstream tasks because held-out loss can imperfectly proxy natural-language performance.

3 Method: Data-Constrained Scaling Laws

The method extends scaling-law analysis to limited unique data by modeling repeated tokens and excess parameters as diminishing-value resources. It evaluates allocation and return through fixed-data, fixed-FLOPs, and parametric-fit experiments.

  • Method objective: The framework modifies Chinchilla scaling laws to account for constrained unique data and fit the resulting terms to extensive experiments.The goal is to determine the best allocation of computational resources and the return from computation when unique data is limited.
  • Data repetition: Given data budget DC and total tokens D, unique tokens are UD = min{DC, D}, while repetitions are RD = (D/UD) −1.RD = 0 corresponds to single-epoch training.
  • Parameter scaling: The parameter decomposition defines UN as the compute-optimal parameter count for UD tokens, capped at N, and excess-parameter repetition as RN = (N/UN) −1.UN is obtained by rearranging the compute-optimal scaling relation and applying its Nopt formula.
  • Experimental protocols: The experiments separately study allocation with fixed unique data, return with fixed FLOPs, and predictive parametric fitting across all training runs.Fixed-unique-data experiments vary epochs and parameters; fixed-FLOPs experiments vary DC, UD, and UN.
  • Effective resources: The proposed effective-data formulation makes repeated tokens less valuable than fresh tokens, with their value decaying exponentially as repetitions increase.For RD = 0, effective data equals the unique data; as RD grows, effective data becomes much smaller than total processed tokens.
  • Effective resources: Excess parameters are modeled symmetrically because models substantially larger than the available data provide diminishing returns per parameter.The parameter half-life is represented by a learned R∗N, analogous to the repeated-data constant R∗D.

4 Experimental Setup

The experiments use GPT-2 transformer language models with up to 8.7 billion parameters and 900 billion training tokens, while varying learning-rate schedules and allowing overfitting under repeated data. Dataset subsets are nested so lower-data runs always use subsets of higher-data runs.

  • Model and training configuration: Models use the GPT-2 architecture and tokenizer, with up to 8.7 billion parameters and 900 billion total training tokens.These settings define the model scale and maximum training budget across experiments.
  • Model and training configuration: Training uses cosine learning-rate schedules that decay tenfold over each model’s training course.The study notes that different schedules can produce different estimates.
  • Dataset setup: Runs with less data and more epochs always use a subset of the data used by runs with more data and fewer epochs.This nested dataset design supports comparisons across repetition levels.

5 Results: Resource Allocation for Data-Constrained Scaling

Under fixed data constraints, scaling epochs faster than parameters can substantially improve loss, with the best observed configurations using roughly 20–60× more parameters and epochs. The resulting data-constrained frontier favors additional epochs over parameters, although excessive compute can eventually worsen performance.

  • Resource Allocation: More than 50% lower loss is attainable by increasing both repeated-data epochs and model size beyond the one-epoch compute-optimal configuration.The best loss occurs at approximately 20–60× more parameters and epochs, requiring around 7000× more FLOPs.
  • Resource Allocation: The data-constrained efficient frontier allocates most additional compute to more epochs rather than more parameters, unlike Chinchilla’s equal scaling recommendation.This contrast arises because Chinchilla’s fit assumes models train for a single epoch without repeating the entire dataset.
  • Resource Allocation: Across 100M, 400M, and 1.5B-token budgets, allocation is optimized by scaling epochs faster than parameters.The experiments fix each unique-data budget and vary compute allocation between model parameters and training epochs.
  • Resource Allocation: 27% fewer parameters still yielded better loss and downstream performance for a data-constrained compute-optimal model than for the Chinchilla-suggested model.This comparison used 9.3 × 10^21 FLOPs and 25 billion unique tokens.
  • Resource Allocation: Adding parameters and epochs eventually increases loss, indicating that excessive compute can hurt performance.The proposed formula and isoLoss contours do not model this behavior, while appropriate regularization may prevent it.

6 Results: Resource Return for Data-Constrained Scaling

Repeated data retains meaningful value for scaling, with negligible loss changes for a few epochs but sharply diminishing returns beyond roughly 16 epochs. The proposed scaling laws accurately capture the declining value of repetition, except for failing models whose loss rises during training.

  • Three FLOP budgets were evaluated across eight data budgets to quantify the return on additional computation.
  • Repeated data is worth less than unique data, so models trained with fewer unique tokens and more epochs achieve consistently higher loss, although the difference is negligible for a few epochs.
  • The parametric scaling laws accurately measure the decay in repeated-data value, but significantly underestimate final test loss for failing models whose loss increases midway through training.
  • R∗_D ≈15 corresponds to 15 repetitions, or 16 epochs, marking the approximate half-life at which repeated tokens have lost 1/e of their value.
  • Meaningful gains from repeating data continue to around 16 epochs, after which returns diminish extremely fast.

7 Results: Complementary Strategies for Obtaining Additional Data

The section evaluates code augmentation, filtering choices, and data repetition for improving downstream performance in data-constrained regimes. Repetition up to around 4 epochs and code filling up to 50% preserve performance, while perplexity filtering helps and deduplication does not.

  • Strategies and evaluation: The study compares code augmentation, deduplication, perplexity filtering, and repetition under a maximum data budget of 84 billion tokens.Filtering experiments begin with approximately twice the budget, or 178 billion tokens.
  • Strategies and evaluation: Downstream performance is averaged across 114 scores from 19 natural language tasks with zero to five in-context few-shot exemplars.Scores are rescaled before averaging because tasks use different metrics and random baselines.
  • Downstream performance: Around 4 epochs of repeated data produce insignificant downstream differences, after which performance begins to decline.The 4-epoch setting corresponds to 25% of the data budget and aligns with test-loss results.
  • Downstream performance: Filling up to 50% of the data with code, or 42 billion tokens, causes no deterioration, but further code addition quickly reduces natural-language-task performance.More code may still benefit non-natural-language tasks.
  • Filtering and recommendations: Perplexity filtering is effective, whereas deduplication does not improve downstream performance, though deduplication may reduce memorization.The authors recommend reserving filtering for noisy datasets while combining code augmentation and repetition.
  • Filtering and recommendations: Doubling data with code and repeating the resulting dataset for four epochs yields 8× more training tokens expected to match 8× more unique data.This example motivates combining code augmentation with repetition in data-constrained training.

8 Related Work

Prior work relates language-model scaling to increasing model size, training data, and compute, while also exploring dataset filtering and data mixing as strategies for improving pre-training data. These studies motivate examining compute allocation and alternatives for handling constrained data.

  • Large language models: Scaling transformer language models across parameter count and training data has produced continuous performance gains, encompassing dense and sparse architectures.The passage traces this literature from the 1.4 billion parameter GPT-2 model onward.
  • Scaling laws: A prior scaling law proposed allocating a 10× compute increase to a 5.5× model-size increase and a 1.8× training-token increase.This allocation contributed to very large models trained on relatively little data, including the 530 billion parameter MT-NLG model trained on 270 billion tokens.
  • Alternative data strategies: Related data strategies include filtering pre-training datasets, perplexity-based noise reduction, and mixing text from different languages for multilingual models.The passage notes that perplexity-based filtering has benefited web-crawled datasets, while mixing is used in multilingual LLM pre-training.

9 Conclusion · Appendix

The work studies data-constrained scaling and optimal computational-resource use when unique data is limited. It extends Chinchilla scaling laws to account for repeated-data value decay and finds that multiple training epochs can be beneficial.

  • 9 Conclusion: The paper studies data-constrained scaling when unique training data is limited.
  • 9 Conclusion: Its focus is the optimal use of computational resources under limited unique data.
  • 9 Conclusion: The authors propose extending Chinchilla scaling laws to model the declining value of repeated data.
  • 9 Conclusion: They fit the proposed function using a large set of controlled experiments.
  • 9 Conclusion: Contrary to earlier recommendations, training large language models for multiple epochs by repeating data is beneficial.
  • 9 Conclusion: The conclusion states that scaling laws continue to hold in the reported setting.

A Derivation of Data-Constrained Scaling Laws … E Repeating on Heavily Deduplicated Data

The paper extends scaling laws to account for diminishing returns from repeated data and excess parameters, yielding a compute-optimal curve that eventually favors processing more tokens. Empirical evaluations support the formulation across C4 fits, isoLoss contours, double descent behavior, and heavily deduplicated data.

  • A Derivation of Data-Constrained Scaling Laws: Repeated tokens are modeled as geometrically declining in value, producing an effective dataset size that sums progressively discounted copies of the unique data.The discount factor is 1 −δ, with δ = 0 making repeats as valuable as new tokens and δ = 1 making them worthless.
  • A Derivation of Data-Constrained Scaling Laws: 1.8% is the difference between approximate and exact effective-data losses in the example with δ = 0.25, U = 1, and R_D = 4.The exact effective data is 3.05 units, while the approximation is 3.21; with β = 0.353, the resulting difference is 1.8%.
  • A Derivation of Data-Constrained Scaling Laws: The fitted scaling law decays both repeated-data and excess-parameter contributions, while Equation 14 remains a direct replacement for the single-epoch Chinchilla equation.Equation 14 gives the same optimal model and data estimates in the single-epoch case and generalizes to multiple epochs and excess parameters.
  • A.1 Analytical properties of compute-optimal point: As repetitions approach the learned decay threshold, the compute-optimal curve breaks from proportional parameter-token scaling because additional parameters become less valuable than additional data.The curve initially follows R_N = R_D, then favors processing more tokens once parameter returns decay faster.
  • B C4 Scaling Coefficients: 70.0 billion parameters and 1.37 trillion tokens are predicted as optimal for a compute budget of 5.76 × 10^23 FLOPs on C4.These predictions closely match the IsoFLOP C4 estimates of 73 billion parameters and 1.3 trillion tokens.
  • C Additional Contour Plots: Increasing parameters and epochs beyond single-epoch compute optimality can lower loss, with the lowest loss achieved by allocating more extra compute to repeated data than to additional parameters.This result is reported for empirical isoLoss contours at 400 million and 1.5 billion unique tokens.
  • D Double Descent: 200 epochs marks the loss increase before subsequent decrease in epoch-wise double descent for models trained on 100 million unique tokens.Because the fitted functional form assumes monotonically decreasing loss with epochs, most such examples are removed from fitting.
  • E Repeating on Heavily Deduplicated Data: 59 epochs is optimal on both regular and heavily deduplicated C4, although deduplication produces higher test loss.The overall performance trend remains very similar regardless of deduplication, using the same validation dataset from regular C4.

F Do Excess Parameters Hurt, Plateau or Help? … N Filtering Procedure

Across excess-parameter, repetition, evaluation, and data-filtering analyses, the paper finds that repeated data can remain useful, excess capacity should plateau rather than hurt, and code augmentation improves selected capabilities. Validation spikes can be temporary, while dataset choice and filtering materially affect downstream performance.

  • F Do Excess Parameters Hurt, Plateau or Help?: Excess parameters empirically hurt, but the authors attribute this potentially to suboptimal regularization and expect optimal regularization to produce a plateau instead.The default Chinchilla scaling law predicts loss will continue decreasing as parameters are added, contrasting with the empirical data.
  • F Do Excess Parameters Hurt, Plateau or Help?: The authors retain exponential decay because excess parameters or epochs can be handled by removing parameters or stopping training when loss is expected to plateau.An alternative alpha-beta decay formulation permits excess capacity to hurt but incorrectly predicts slower diminishing returns from repeated data.
  • G Case Study: Galactica: Galactica trained on 106 billion unique tokens for 4.25 epochs, and the authors find that even more epochs can remain beneficial despite a small validation-loss spike.Their repeated-data findings agree that multiple epochs are beneficial, while disputing early stopping solely because of a small spike.
  • H Training Loss: Training loss is unsuitable for repeated-data comparisons because models overfit limited data, so the authors use loss on a held-out test set as the key metric.Models trained on fewer unique tokens have better training loss despite greater repetition.
  • I Scaling Curves on the OSCAR Corpus: On OSCAR, the repetition trend matches C4: models with fewer repeats have better loss, but differences for a few repeats are insignificant.OSCAR is considered noisier than C4 because of less stringent duplication.
  • J Validation Loss by Epoch: Validation-loss increases at new epochs can be temporary; loss spikes for some models are followed by smooth declines, suggesting training should continue beyond such spikes.This raises the hypothesis that Galactica could have improved by continuing beyond the beginning-of-fifth-epoch spike.
  • K Evaluation Details: The evaluation protocol uses held-out validation loss and downstream scores averaged across 0-5 few-shots, with scores normalized from each task’s random baseline to 1.Final C4 test loss is computed on the same 210 million validation tokens for all models.
  • L Downstream Repetition Results; M Detailed Code Augmentation Results; N Filtering Procedure: C4 generally outperforms noisier OSCAR, but OSCAR performs better than random on bAbI, while up to 50% Python preserves natural-language performance and greatly improves C4 state-tracking.The difference is attributed partly to code being present in OSCAR but filtered from C4; filtering experiments use perplexity, deduplication, and ROOTS filters.

O Detailed Filtering Results · P Loss Curves for Complementary Strategies

Detailed filtering results show that perplexity filtering and deduplication help mainly on noisier or larger-model settings, while loss curves reveal distributional biases that complicate comparing complementary data strategies.

  • O Detailed Filtering Results: Perplexity filtering is effective for C4 only at 4.2B parameters, but for noisier OSCAR it is effective at both 2.8B and 4.2B parameters.
  • O Detailed Filtering Results: Deduplication does not improve downstream performance for C4, whereas it is effective for the substantially noisier OSCAR dataset.
  • O Detailed Filtering Results: Perplexity-filtering experiments repeat data to 55B tokens for 2.8B-parameter models and 84B tokens for 4.2B-parameter models.Table 13 reports normalized averages of 0-5 few-shots as percentages.
  • O Detailed Filtering Results: Table 14 compares deduplication and ROOTS filtering using repeated data matched to 55B or 84B training tokens, with unfiltered-model variability reported across five seeds.
  • P Loss Curves for Complementary Strategies: Validation loss is not used to compare complementary data strategies because it favors models trained on a larger fraction of the validation distribution.This issue arises when code data compensates for missing natural-language data.
  • P Loss Curves for Complementary Strategies: Deduplicated and perplexity-filtered models have higher validation loss when held-out validation data remains unfiltered, while perplexity-filtered data yields the lowest training loss.Figure 20 presents validation and smoothed training losses for different data strategies.
  • P Loss Curves for Complementary Strategies: Filtering repeated sequences raises loss on common unfiltered validation sequences because unfiltered models may have memorized them.Fewer repeated sequences also make unseen sequences harder to predict, increasing training loss.

Q Limitations and Future Work … X Broader Impacts

The paper identifies limitations concerning repeated-data strategies, hyperparameter sensitivity, datasets, modalities, and alternative mitigation strategies, while documenting contributions, training setup, evaluation prompts, additional experiments, revisions, and broader impacts.

  • Q Limitations and Future Work: Repeating only a fraction of the dataset may require scaling laws with an additional parameter for the repeated fraction.The authors give the example of repeating 10% of data for 10 epochs while repeating the remainder once.
  • Q Limitations and Future Work: Returns from additional epochs may depend heavily on learning rate, dropout, and optimizer choice, which the authors leave for future work.They hypothesize that increasing the learning rate could make diminishing returns from extra epochs begin earlier.
  • Q Limitations and Future Work: Optimal data strategies depend on the dataset, although findings on C4, OSCAR, and The Pile were broadly consistent.Deduplication was more effective on OSCAR because it was noisier than C4.
  • Q Limitations and Future Work: Future work could test these findings across other modalities, architectures, languages, and filtering strategies.The study used text datasets with GPT, examined Python augmentation, and considered alternatives such as popularity-based filtering.
  • R Contributions: Niklas Muennighoff led the experiments, analysis, writing, and overall project, while other authors contributed to framing, data, infrastructure, experimental guidance, and writing.Contributions also included perplexity and deduplication datasets, AMD hardware enablement, and early repetition experiments.
  • S Hyperparameters and Setup: Training used cosine-decayed learning rates, parameter-dependent batch sizes, bfloat16 precision, Adam, and up to 256 AMD Instinct MI250X GPUs across 64 nodes.The maximum learning rate was 2e-4, decayed to 2e-5, with 1% linear warm-up; models used data, tensor, and pipeline parallelism.
  • T Prompts and Samples: Evaluation prompts came from PromptSource or GPT-3, used ground-truth datasets without generations, and covered classification, generation, summarization, and exact-match tasks.The examples include ANLI, ARC, BoolQ, HellaSwag, PiQA, E2E NLG, XSUM, WebNLG, WikiLingua, and bAbI.
  • U Other Experiments / W Version Control: The authors report that UL2 did not outperform regular causal language modeling, and they publicly released models trained on The Pile after observing similar trends.They attribute UL2’s result possibly to encoder-decoder suitability or implementation mistakes; version updates also added analyses, corrections, and documentation.
Loading 2305.16264v5…