PaperlayerView exact revision on arXiv

Source-linked AI summary

InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition

Fengze Liu, Weidong Zhou, Binbin Liu, Ping Guo, Zijun Wang, Bingni Zhang, Yifan Zhang, Yifeng Yu, Xiaohuan Zhou, Taifeng Wang

arXiv:2605.02364v1cs.CL

TL;DR

In data-constrained LLM pretraining, conventional scaling laws do not reliably extrapolate across mixture recipes and repetition. InfoLaw models information accumulation to predict loss, achieving 0.15% average and 0.96% maximum absolute error while extrapolating across scales and overtraining.

  • Problem

    Conventional scaling laws have limited reliability across data-mixture recipes and repetition in data-constrained LLM pretraining.

  • Method

    InfoLaw models learned information from data quality, mixture weights, train tokens, model size, and repetition-induced diminishing returns to predict loss.

  • Results

    0.15% average and 0.96% maximum absolute error were achieved when predicting unseen recipes and larger-scale runs, with reliable extrapolation to 25× overtraining.

  • Takeaways & Limitations

    InfoLaw supports efficient data-recipe selection across computational budgets without extensive additional experiments.

  • Takeaways & Limitations

    The framework assumes higher-quality buckets have larger quality-density values, imposing a decreasing form on the fitted quality function.

Abstract

from arXiv · show

Upweighting high-quality data in LLM pretraining often improves performance, but in datalimited regimes, especially under overtraining, stronger upweighting increases repetition and can degrade performance. However, standard scaling laws do not reliably extrapolate across mixture recipes or under repetitions, making the selection for optimal data recipes at scaling underdetermined. To solve this, we introduce InfoLaw (Information Scaling Laws), a data-aware scaling framework that predicts loss from consumed tokens, model size, data mixture weights, and repetition. The key idea is to model pretraining as information accumulation, where quality controls information density and repetition induces scaledependent diminishing returns. We first collect the model performance after training on datasets that vary in scale, quality distribution, and repetition level. Then we build up the modeling for information so that information accurately predicts those model performance. InfoLaw predicts performance on unseen data recipes and larger scale runs (up to 7B, 425B tokens) with 0.15% mean and 0.96% max absolute error in loss, and it extrapolates reliably across overtraining levels, enabling efficient data-recipe selection under varying compute budgets.

1. Introduction

InfoLaw addresses data-constrained LLM scaling by modeling information accumulation across heterogeneous data mixtures and repetition. It predicts loss across unseen recipes, larger scales, and 25× overtraining, achieving 0.15% mean and 0.96% maximum absolute error up to 7B parameters and 425B tokens.

  • Introduction: High-quality data is scarce, while excessive repetition from upweighting it can impair performance, especially under overtraining (Muennighoff et al., 2023; Touvron et al., 2023; Yang et al., 2025).Overtraining reduces inference costs relative to the compute-optimal regime (Hoffmann et al., 2022), intensifying the data-repetition tradeoff.
  • Introduction: InfoLaw models training as information accumulation, combining mixture-weight scaling with repetition-dependent diminishing returns to determine data recipes.High-quality data yields larger gains initially, but repeated exposure reduces its marginal benefit toward that of unseen low-quality data.
  • Introduction: The study fits InfoLaw using nine from-scratch models ranging from 252M to 1.2B parameters trained on LayerMix datasets varying in scale, quality, and repetition.LayerMix partitions the source dataset into quality buckets and samples them with different weights after downsampling to the target scale.
  • Introduction: 0.15% mean and 0.96% maximum absolute error are achieved when InfoLaw predicts loss across unseen recipes, larger scales, and 25× overtraining, including a 7B model trained on 425B tokens.The evaluation spans unseen LayerMix mixture weights, larger compute scales, and a higher overtraining ratio.

2. Related Work

Prior work established scaling laws for model size, training data, and compute allocation, then extended them to overtraining, data quality, repetition, and mixture design. These studies motivate InfoLaw’s focus on data-aware scaling under limited and repeated data.

  • Scaling Laws: Transformer language models exhibit predictable power-law scaling with model size and training data, motivating dense and mixture-of-experts systems.Compute-based laws formalize model-capacity and token allocation under fixed compute budgets.
  • Scaling Laws: Overtraining smaller models on more tokens has become common, while later scaling laws incorporate data quality, inference requirements, and overtrained regimes.Hoffmann et al. (2022) characterized compute-optimal training; Sardana et al. (2024) and Gadre et al. (2024) extended analysis beyond that setting.
  • Data-Aware Scaling: Repeated or upsampled data can yield diminishing returns and eventual degradation, although continued training on repetitions may sometimes outperform stopping early.These effects show that repetition is non-trivial and is not captured by classical scaling laws.
  • Data-Aware Scaling: Scaling-law methods increasingly optimize data recipes by predicting mixture-weight effects, searching ratios with proxy models, and analyzing model-scale-dependent mixing.Related work also applies scaling insights to continued pre-training and domain-mixture design.

3. Limitations of Conventional Scaling Laws

Conventional scaling laws are insufficient for modeling loss when training combines quality-weighted data mixtures with repetition, particularly for extrapolation. The section introduces LayerMix to vary data scale, quality composition, and repetition, then compares scaling behavior with and without repetition.

  • 3.1 LayerMix sampling: LayerMix uses quality scores from two classifiers to partition documents into six percentile buckets and enforces greater representation of higher-quality buckets.Five preset mixtures are used, with the lowest-quality bucket dropped; unless stated otherwise, K = S isolates repetition effects induced by mixture weights.
  • 3.1 LayerMix sampling: LayerMix constructs packed datasets whose scale, quality mixture, and repetition vary through sampling weights, source size, and training-set size.It defines bucket-level repetition as R_d = K_d/M_d, with R_d = 1 when all sampled tokens are unique and R_d > 1 otherwise.
  • 3.2 Loss–compute scaling: The experiments compare HQ data with more high-quality but more repetitive samples against MLQ data with more diverse and less repetitive samples under compute-optimal and overtrained regimes.Loss is measured as average perplexity over five downstream tasks in the loss–C_m view.
  • 3.2 Loss–compute scaling: Conventional scaling laws are insufficient to characterize scaling behavior under quality-weighted mixture data with repetition, especially for extrapolation.These observations motivate a modified scaling law that explicitly includes data-quality distribution and repetition degree.

4. Information Scaling Laws

InfoLaw models pretraining as information accumulation, combining data quality, repetition, model scale, and training tokens to predict validation loss. Its repetition- and quality-aware Information metric collapses varied training configurations onto a unified power-law curve, enabling extrapolation to larger models and token budgets.

  • Information-based formulation: Information accumulates from data quality, repetition, model scale, and training tokens, then maps monotonically to final validation loss through a power law.The metric is computed from LayerMix weights, training tokens, and fitted quality-density and repetition-rate functions, allowing loss prediction before new runs.
  • Repetition modeling: Repeated exposure yields diminishing information gains because the exponential decay model makes total learned information saturate toward each document’s full information content.The decay rate λ(N) depends on model non-embedding FLOPs/token, linking repetition effects to model scale.
  • Quality and repetition integration: Higher-quality buckets receive higher information density, while each bucket’s learned information combines packed-data content with the model’s repetition-dependent learning ability.For bucket d, the formulation uses unique-token count Md = min(wdK, BdS), density fd, and average repeat count Rd.
  • Unified scaling relationship: InfoLaw collapses experiments varying LayerMix weights, model non-embedding FLOPs/token, and training tokens onto a single unified power-law loss curve.Traditional compute-based scaling laws are unreliable for quality-weighted mixtures with repetition, motivating Information as an effective data signal.
  • Power-law fitting: The fitted loss–Information power law uses α = 3.7373 and β = 0.0441, with log-log slope −β and intercept log(α).The framework supports small-model comparisons and extrapolation to larger models trained with more tokens.

5. FITTING EXPERIMENTS

The fitting experiments estimate InfoLaw’s quality-density and model-capacity functions from 27 runs spanning model sizes, data-quality mixtures, and overtraining. The fitted capacity curve grows rapidly for smaller models, then saturates logarithmically and extrapolates strongly to larger models.

  • Experimental setup: 27 runs train nine 252M–1.2B models across HQ, MQ, and LQ LayerMix weights at a 3.6x over-trained ratio.The experiments use a transformer with SwiGLU activations, RoPE embeddings, and a 250k-vocabulary tokenizer.
  • Parameter fitting: The fitting objective maximizes Spearman correlation between modeled information and evaluation loss, while enforcing higher quality density for higher-quality data buckets.The quality-density parameterization is constrained to decrease with bucket index, with θ > 0.
  • Parameter fitting: 100,000 sampled combinations of θ and λ(N) identify optimal parameters, yielding fitted θ*=0.922 for the quality-density function.The fitted density is then combined with the capacity function to calculate information for arbitrary mixtures, token budgets, and model sizes.
  • Parameter fitting: a*=0.140 and b*=0.018 fit the λ(N)-N curve, whose logarithmic form shows strong extrapolation to larger models.The relationship is nonlinear, with rapid growth at smaller N and gradual saturation as N increases.

6. Extrapolation

InfoLaw extrapolates validation loss across unseen data recipes, larger model scales up to 7B, and higher overtraining degrees, while enabling recipe selection under varying training budgets. It also captures how optimal quality–diversity tradeoffs shift with model size and token count.

  • Unseen recipes: InfoLaw predicts unseen LayerMix recipes directly, whereas traditional scaling laws require additional experiments to fit new recipe-specific curves.Predicted losses align with experiments on unseen MLQ and MHQ mixtures and 25 additional sampled weights.
  • Larger model scales: InfoLaw accurately extrapolates loss to larger models, including 1.5B–7B scales, while traditional scaling laws become overly optimistic at high compute.The model-size extrapolation uses training data from 252M–1.2B models and remains accurate beyond that range.
  • Unseen recipes and scales: 0.15% validation-loss error was achieved across unseen LayerMix weights and model scales, including combined extrapolation to 7B models.These results cover MLQ, MHQ, 25 randomly sampled weight sets, and unseen model scales.
  • Higher overtraining: Using parameters fitted at m′ = 25, InfoLaw predicts a new higher-overtraining regime whose scaling curve is nearly parallel to the original, indicating mainly an intercept shift.The Cm′ prediction is generated from parameters fitted only on the Cm data.
  • Recipe optimization: Pearson correlation reached 0.76 on 25 held-out LayerMix configurations, supporting InfoLaw’s use for ranking recipes without additional search experiments.Optimal recipes shift from high-quality emphasis toward greater diversity as model size or total training tokens increases.

7. Conclusion

The paper proposes InfoLaw, a refined scaling law for predicting downstream performance under data-constrained, weighted-quality mixing. It accurately extrapolates to unseen recipes at larger computational scales and enables efficient data-recipe discovery without extensive additional experiments.

  • Conclusion: InfoLaw predicts downstream model performance under data-constrained settings with weighted-quality mixing.It is presented as a refined scaling law focused on this regime.
  • Conclusion: 0.15% average absolute error and 0.96% maximum error demonstrate accurate predictions on unseen data recipes at larger computational scales.These results support extrapolation beyond the evaluated data recipes and scales.
  • Conclusion: InfoLaw enables efficient discovery of optimal data recipes without extensive additional experiments.The prediction capability reduces the need for broad experimental search when selecting recipes.

8. Impact Statement · A. Training Dataset

The paper presents InfoLaw as a way to improve understanding of data mixing and repetition in LLM pretraining while reducing costly recipe experimentation. Its training data is deduplicated English Common Crawl spanning 96 snapshots and 3.7T tokens, and broader LLM risks remain relevant.

  • 8. Impact Statement: InfoLaw aims to improve understanding of LLM performance under different data-mixing and repetition strategies.
  • 8. Impact Statement: InfoLaw could make pretraining more efficient by reducing expensive trial-and-error over data recipes.
  • 8. Impact Statement: The authors do not anticipate direct negative societal consequences uniquely arising from this contribution.
  • 8. Impact Statement: Bias, misuse, and unsafe deployment remain important broader ethical issues associated with LLMs.
  • A. Training Dataset: The training corpus uses the English portion of Common Crawl across 96 snapshots from CC-MAIN-2013-20 to CC-MAIN-2024-18.
  • A. Training Dataset: Global fuzzy deduplication across all snapshots produced a dataset containing 3.7T tokens.

B. Justification for the Normalization Term log(K) · C. LayerMix Sampling Function

The authors justify log(K) as the normalization that best captures repetition decay across token budgets, unlike constant or power-law alternatives. They also specify LayerMix sampling, including the distinction between compute-optimal and overtrained runs.

  • B. Justification for the Normalization Term log(K): Equation 3 incorporates log(K) into the decay function to model the interaction between repetition decay and the total token budget.The authors selected this form after evaluating constant and power-law normalization alternatives.
  • B. Justification for the Normalization Term log(K): Constant normalization overestimates accumulated Information for large models trained with larger token budgets, producing overly optimistic loss predictions.The formulation fails to account for the scaling properties of information density and deviates significantly from experimental results.
  • B. Justification for the Normalization Term log(K): Power-law normalization cannot fit the relationship between Information and Validation Loss because its data points remain scattered without the required power-law correlation.The authors therefore could not derive a valid scaling law from this formulation.
  • B. Justification for the Normalization Term log(K): log(K) uniquely collapses varied (w, K, S) configurations onto one power-law curve while maintaining low extrapolation error from 252M to 7B model scales.This behavior is shown in Figure 3f and indicates logarithmically diminishing marginal utility for repeated data relative to the total training budget.
  • C. LayerMix Sampling Function: LayerMix’s sampling function is detailed in Algorithm 1.The passage identifies the algorithm as the specification of the LayerMix sampling procedure but provides no further implementation details.
  • C. LayerMix Sampling Function: m = 1 denotes a compute-optimal training run, whereas m > 1 denotes overtraining relative to the compute budget.This distinction defines how LayerMix characterizes training beyond the compute-optimal regime.

D. Training … K. Generalization to Refinedweb

Across training, repetition analysis, quality evaluation, and generalization tests, InfoLaw captures how data quality, repetition, model scale, and token mixtures affect loss and downstream performance. On RefinedWeb, it extrapolates an unseen quality configuration with 0.24% mean and 0.36% maximum absolute percentage error.

  • D. Training: Training uses 2048-token sequences, cosine decay, an initialized learning rate lr = round(0.3118 · C^-0.1250, 8), 0.5% warmup, AdamW, and specified β and weight-decay settings.The optimizer uses β1 = 0.9, β2 = 0.95, and weight decay = 0.1; repetition analysis distinguishes IST from LST regimes, where LST induces repetition and heavier repetition worsens final loss.
  • F. The relationship between benchmark validation loss and performance: Validation loss is near-linearly related to downstream benchmark performance, with lower loss consistently corresponding to higher performance within the studied operating regime.The relationship spans ARC-C, ARC-E, HellaSwag, MMLU-Lighteval, and TriviaQA, with Spearman correlations reported in Table 4.
  • G. Alternative Fits for λ: The logarithmic λ(N) model fits and extrapolates better than exponential and power-law alternatives, so it is adopted as the final parameterization.λ(N) relates the parameter λ to non-embedding FLOPs/token N.
  • H. Deviation of Traditional Scaling Law: LayerMix loss–C curves under both IST and LST sampling deviate clearly from the traditional scaling law fitted to the first three data points.The deviation is shown across different LayerMix sampling weights and repetition regimes.
  • I. Quality Score: High-quality FineWebEdu-selected subsets outperform random data, and the higher-quality subset performs better when training a 1.2B model on 30B tokens.The comparison uses the top 5%, top 20%, and random subsets from Penedo et al. (2023).
  • J. Optimizing Token Mix with InfoLaw: InfoLaw predicts that small models and budgets prioritize quality, whereas large models and budgets prioritize diversity in the optimal LayerMix token mix.These model- and budget-specific token-mix ratios are reported in Table 6.
  • K. Generalization to Refinedweb: InfoLaw’s fitted quality-density parameter θ was 0.93 on RefinedWeb, close to 0.92 on the primary dataset.The similarity is attributed to both datasets deriving from Common Crawl despite different filtering strategies.
  • K. Generalization to Refinedweb: 0.24% mean and 0.36% maximum absolute percentage error were achieved when InfoLaw extrapolated validation loss for unseen MLQ experiments on RefinedWeb.HQ and LQ configured fitting, while MLQ was held out for extrapolation; the λ(N) curve was not fit because only three model scales were available.

L. Limitation

The work’s data bucketing relies on a fixed empirical heuristic without ablations for the optimal number or boundaries of quality tiers, while the systematic effect of overtrain degree m still lacks theoretical explanation.

  • L. Limitation: Data bucketing uses a fixed empirical heuristic, with no ablation studies identifying the optimal number or boundaries of quality tiers.More systematic data partitioning could improve predictive accuracy.
  • L. Limitation: The systematic shift of the scaling-law curve with overtrain degree m is observed but not yet theoretically explained.Developing such an explanation remains an open direction.
  • L. Limitation: A more systematic approach to data partitioning is a clear avenue for improving the model’s predictive accuracy.
Loading 2605.02364v1…