PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Domain Data Repetition in LLM Pretraining

Jingwei Li, Xinran Gu, Rui Dai, Xintong Hao, Chengyin Xu, Yan Wu, Shuran Zheng, Jingzhao Zhang

arXiv:2608.14071v1cs.AI

TL;DR

Scaling LLM training increases token budgets faster than high-quality domain data can be expanded, making repetition a trade-off between countering dilution and risking overfitting. This paper studies repetition at fixed tokens-per-parameter ratios and finds that optimal repetition rises mildly with model size, varies strongly with domain validation loss, and is nearly insensitive to unique-data fraction.

  • Problem

    The paper asks how much high-quality domain data should be repeated as LLM training budgets grow, balancing dilution against overfitting.

  • Method

    The study evaluates domain-data repetition across model scales while keeping the tokens-per-parameter ratio fixed and varying unique-data fractions.

  • Results

    At fixed TPP, optimal repetition increases mildly with model size, is strongly negatively correlated with final validation loss, and remains nearly insensitive to unique high-quality token fraction.

  • Takeaways & Limitations

    Repetition counts selected on smaller proxy models at the same TPP can provide conservative estimates for larger models.

  • Takeaways & Limitations

    The experiments repeat only one high-quality dataset at a time, leaving simultaneous repetition of multiple interacting domains for future work.

Abstract

from arXiv · show

As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixed \(\mathrm{TPP}\), the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the same \(\mathrm{TPP}\) can provide a practical estimate for larger models.

1 Introduction

This study examines repeating scarce high-quality domain data under practical fixed-TPP scaling, balancing domain-learning gains against overfitting. It finds that optimal repetition increases with model size, favors lower-loss domains, and is largely insensitive to unique-data amount.

  • Domain differences: Domains with lower final validation loss can generally tolerate and benefit from more repetitions, while higher-loss domains overfit earlier.The optimal repetition count is strongly negatively correlated with a domain’s final validation loss.
  • Scaling regime: Fixed-TPP scaling reverses the fixed-data-budget trend: optimal repetition increases with model size.The study focuses on fixed TPP because it better reflects practical LLM scaling.
  • Unique data: Across tested unique-token fractions, the repetition count minimizing validation loss remains nearly unchanged.The optimal repetition count is therefore largely insensitive to the amount of unique domain data.
  • Practical implication: A smaller proxy model at the same TPP can estimate relatively safe repetition counts for a target model.The proxy sweep can use any representative unique high-quality token fraction within the tested range.

2 Related Work

Prior work studies repetition in single-dataset training and pretraining mixtures, while data-mixture optimization generally assumes sufficient unique data and does not model repetition.

  • Data Repetition in Single-Dataset Training: Single-dataset studies extend scaling laws to repeated data and model validation loss under repetition, finding diminishing returns from heavier reuse.These studies report that a few repeated epochs can perform similarly to fresh data.
  • Data Repetition in Pretraining Mixtures: Mixture-pretraining studies reuse limited subsets alongside unique data and find that repetition can cause non-monotonic degradation.One setting draws 10% of training tokens from a repeatedly reused subset and 90% from unique data, with intermediate repetition levels particularly harmful.
  • Data Mixture Optimization for LLMs: Data-mixture optimization allocates fixed training budgets across domains using proxy training, predictive models, or scaling laws, but generally assumes sufficient unique data.Recent work relaxes this assumption, though the supplied passage does not provide further details.

3 Setup

The setup varies model size, unique high-quality data, and repetition count at a fixed tokens-per-parameter ratio, while evaluating both in-domain and out-of-domain validation losses. The total token budget is fixed for each model size, with larger models receiving proportionally larger budgets.

  • Model and data setup: Training uses model sizes with a constant TPP greater than 100, so larger models receive proportionally larger token budgets.Runs with the same model size use exactly the same number of training tokens.
  • Model and data setup: For each high-quality domain, experiments sweep unique-data fractions α and repetition counts e ∈ {1, 2, 3, 4, 5, 6, 7}.A fixed subset of unique domain tokens is selected and then revisited e times.
  • Model and data setup: The constraint eα ≤ 1 keeps the non-repeated web-data allocation nonnegative while preserving the total budget for each model size.Only the mixture composition changes between the target domain and web corpus.
  • Model and data setup: Each configuration targets one high-quality domain rather than mixing multiple high-quality domains, producing domain-specific experiment grids and a full experiment set.Models are indexed by domain, model size, unique-data fraction, and repetition count.
  • Evaluation: Models are evaluated with token-averaged negative log-likelihood on both in-distribution and out-of-distribution validation corpora.The out-of-distribution set measures performance outside the repeated target domain.

4 Analysis of the Optimal Repetition Count

The optimal repetition count is driven mainly by domain validation loss, with milder dependence on model size and little dependence on unique-data fraction. A knowledge-acquisition versus noise-fitting trade-off explains these patterns and motivates proxy-model tuning under matched TPP.

  • Empirical factors: The optimal repetition count varies substantially across domains, while its dependence on model size is mild and its dependence on unique high-quality data is limited.Math reaches its minimum at around 5 repetitions, whereas Wiki, Code, and Medical reach their minima earlier.
  • Empirical factors: −0.944: The optimal repetition count has a strong negative Pearson correlation with minimum validation loss.Higher-loss settings tend to prefer fewer repetitions, while lower-loss settings tend to prefer more.
  • Empirical factors: The optimal repetition count is mainly determined by high-quality domain loss, with smaller dependence on model size and little observable dependence on the unique high-quality data fraction.Minimum validation loss is used as a continuous proxy for domain-specific characteristics.
  • Practical implication: A smaller proxy model with the same tokens-per-parameter ratio can provide a conservative repetition-count estimate for large-scale training.Proxy-model validation loss can indicate which repetition-count range to consider for each domain.
  • Theoretical explanation: Repeated optimization decreases knowledge-acquisition error but increases noise-fitting error, so the optimum occurs when the latter marginal effect begins to dominate.This trade-off provides the theoretical explanation for overfitting after excessive repetition.
  • Theoretical explanation: Under fixed tokens-per-parameter scaling, increasing model size raises the optimal repetition count because the growing data budget lets knowledge-acquisition improvements dominate for longer.Theorem 4.4 quantifies this increase in optimal repetition count.

5 Additional Analyses

Additional analyses show that repetition’s impact depends strongly on the domain, while replacing unique tokens with repeated tokens has limited OOD effects. The optimal repetition count also varies with the learning-rate schedule, with later decay or a constant rate tolerating more repetitions.

  • 5.1 Repeated versus unique high-quality tokens: More unique data consistently lowers validation loss, but repetition is domain dependent: Math is relatively robust, whereas Wiki degrades sharply beyond 2 repetitions.At fixed total high-quality-data fractions, increasing repetition means using fewer unique tokens.
  • 5.1 Repeated versus unique high-quality tokens: From 1 to 4 repetitions, Math shows only a small validation-loss increase, so repeated Math data can largely substitute for additional unique Math data.Wiki instead shows modest loss from repeating twice and substantial degradation with heavier repetition.
  • 5.1 Repeated versus unique high-quality tokens: Repetition effectiveness is domain dependent rather than governed by a universal threshold; Code and Medical also worsen as unique data is replaced by repeated data.The magnitude of degradation varies across domains.
  • 5.2 OOD pretraining performance: At fixed high-quality and web-data fractions, OOD validation loss remains largely stable across 1, 2, and 4 repetitions, unlike in-domain loss.This pattern holds across model sizes and total high-quality data fractions, with evaluations on ArXiv and News.
  • 5.3 Learning-rate schedule: Earlier learning-rate decay causes degradation after fewer repetitions, whereas delayed decay shifts it later and a constant rate tolerates the most repetitions.Repeated samples encountered during low-learning-rate training may be fitted more closely, increasing memorization of sample-specific patterns.

6 Conclusion and Future Directions … A.1 Details of Datasets

At fixed tokens-per-parameter ratio, optimal repetition increases mildly with model size and is higher for domains with lower minimum validation loss. The paper also identifies multi-domain repetition and quantitative scaling rules as future directions, alongside implementation and dataset details.

  • 6 Conclusion and Future Directions: Optimal repetition count is strongly negatively correlated with a domain’s minimum validation loss: lower-loss domains generally support more repetitions.At fixed tokens-per-parameter ratio, repetition increases mildly with model size and is nearly insensitive to the fraction of unique high-quality data.
  • 6 Conclusion and Future Directions: Earlier learning-rate decay causes degradation after fewer repetitions, whereas delaying or removing decay allows more repetitions.The comparison uses WSD schedules with decay beginning at 0%, 25%, 50%, or 75% of training, followed by a constant learning rate.
  • 6 Conclusion and Future Directions: Future work should repeat multiple domains simultaneously to study their interactions, rather than repeating only one high-quality dataset per run.The current setup repeats one high-quality dataset in each training run.
  • 6 Conclusion and Future Directions: A quantitative scaling rule could predict large-model optimal repetition counts from small-model results, improving the accuracy and efficiency of data-recipe selection.This direction is motivated by the need to transfer repetition estimates across model scales.
  • A General Implementation Details: The appendix describes the datasets and training hyperparameters used in the experiments.These details are presented as general implementation information.
  • A.1 Details of Datasets: The pretraining corpus combines general web data with four high-quality domains: code, math, wiki, and medical data.General web data spans diverse sources, topics, and writing styles; the domain datasets cover programming, mathematics, encyclopedic knowledge, and medical or health content.

A.2 Details of Training Hyperparameters

Training-token budgets scaled proportionally with model size at a fixed tokens-per-parameter ratio, while optimization hyperparameters followed specified scaling and scheduling rules. Training used Muon with attention dropout and a 200-step learning-rate warmup.

  • Scaling rules: Training-token budgets scaled proportionally with model size under a fixed tokens-per-parameter ratio.Learning rate and global batch size were set using power-law functions of model size.
  • Optimization schedule: Learning rate was linearly warmed up for the first 200 optimization steps, then held constant.The schedule applied across training after the warmup period.
  • Optimization settings: Training used an attention dropout rate of 0.1 and the Muon optimizer, a variant of AdamW.Muon was used for optimization together with the specified dropout setting.

B Additional Results on Section 4.1

Across Math, Wiki, Code, and Medical, the optimal repetition count increases with model size but remains nearly unchanged across α at fixed model size. Math supports the most repetition, while Medical supports the least.

  • The optimal repetition count increases with model size across all four domains but remains nearly unchanged across α at fixed model size.The complete domain results cover Math, Wiki, Code, and Medical.
  • Math reaches its minimum validation loss after about 5–6 repetitions, the highest optimal count among the four domains.Increasing the unique data fraction lowers absolute validation loss but has little effect on the optimal repetition count.
  • Code generally reaches its minimum validation loss after 4–5 repetitions, with the optimum stable across unique data fractions and mildly increasing with model size.
  • Wiki and Medical generally reach their minimum validation loss after 3–4 repetitions, with Medical having the lowest optimal repetition count.Both domains show stability across unique data fractions and a mild increase in the optimum with model size.

C Additional Results on Section 5.1 · D Proofs · D.1 Proof of Theorem 4.2

Additional fixed-token-budget results show that repetition’s effectiveness depends strongly on domain, while the proof of Theorem 4.2 establishes the small-noise repetition-rate asymptotic.

  • C Additional Results on Section 5.1: Fixed total high-quality token fractions compare repeated data against an equivalent number of fresh unique tokens.Increasing repetition reduces the amount of unique high-quality data while holding the total high-quality fraction fixed.
  • C Additional Results on Section 5.1: Code and Medical show clearer validation-loss degradation as repetition replaces unique data, similarly to Wikipedia.The results indicate that matching fresh data under a fixed token budget depends strongly on the domain.
  • D Proofs: The proofs in this block establish the results stated in Section 4.2.
  • D.1 Proof of Theorem 4.2: For coordinate k, conditioning on Mk = m models the sample count as Binomial(D, pk) with probability πk,m in the exact averaged population risk.The variance term is defined as 0 when m = 0, with am = 1 −ηm/D.
  • D.1 Proof of Theorem 4.2: When σ2^2 > σ1^2, the earliest minimizer under the higher noise variance satisfies rG ≤ rF.The proof uses that the difference function ∆(r) is non-decreasing, and strictly increasing under the stated variance inequality.
  • D.1 Proof of Theorem 4.2: For D ≥2, setting ρ = a2/a < 1 yields arτ = Θ(τ), and the same bounds hold directly when D = 1.The constants in the bounds are independent of τ, and the remainder vanishes identically when D = 1.

D.2 Proof of Theorem 4.3

The proof shows that when N exceeds N0(D, σ2), the risk increment ∆N(r) is non-decreasing in r. This monotonicity implies that the earliest minimizer for N+1 is no larger than that for N.

  • The proof defines the risk increment as ∆N(r) := 2RD,N+1(r) − 2RD,N(r).
  • When N > N0(D, σ2), the sufficient condition holds and ∆N(r) is non-decreasing in r.
  • Monotonicity of ∆N(r) makes rG > rF impossible, because G(rG) ≥ G(rF).Here F(r) := 2RD,N(r), G(r) := 2RD,N+1(r), and rF and rG are their respective earliest minimizers.
  • Therefore, the earliest minimizers satisfy rG ≤ rF.A strict inequality contradicts optimality of rG, while equality contradicts the earliest-minimizer convention.

D.3 Proof of Theorem 4.4

The proof decomposes marginal risk changes into a nonnegative signal benefit and variance penalty, then shows exact risk decreases for small repetition counts and increases beyond a larger threshold. It also rules out any later decrease for r ≥ D when D is sufficiently large.

  • Both SD(r) and VD(r) are nonnegative, representing the marginal signal benefit and marginal variance penalty, respectively.
  • For sufficiently large D, SD(r) > VD(r) when 0 ≤ r ≤ c0D^(α/β), but SD(r) < VD(r) when r ≥ C′D^(α/β).
  • For sufficiently large D, the exact risk is increasing for every r ≥ D because σ² = o(1) when β > α.
  • The exact risk decreases up to c0D^(α/β) and increases after C′D^(α/β).
Loading 2608.14071v1…