PaperlayerView exact revision on arXiv

Source-linked AI summary

To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis

Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, Yang You

arXiv:2305.13230v2cs.LGcs.AIcs.CL

TL;DR

High-quality web text is approaching its scaling limit, while the effects of repeating pre-training data for LLMs remain unclear. This study empirically examines multi-epoch training and finds overfitting and degradation, with dropout effective and MoE useful for predicting dense-model behavior.

  • Problem

    High-quality web text is approaching its scaling limit, and prior findings leave unclear how repeated pre-training data affects LLMs.

    Source: §1 · p. 2

  • Method

    The paper empirically studies repeated-data pre-training, its degradation factors, and whether regularization can alleviate degradation.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    Larger models are more prone to overfitting on repeated data; larger datasets can alleviate multi-epoch degradation, while dataset quality and FLOPs have little effect, and dropout is effective but needs tuning at very large scales.

    Source: §1 · p. 3

  • Takeaways & Limitations

    MoE can predict the behavior of dense models with comparable parameters but much higher computational requirements, supporting more efficient LLM development.

    Source: §1 · p. 3, §6 · p. 9

    Experiments used models with 3B parameters and did not test state-of-the-art-scale models because repeated experiments on larger models were prohibitively expensive.

    Source: C Limitations · p. 13

Abstract

from arXiv · show

Recent research has highlighted the importance of dataset size in scaling language models. However, large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs. To further enhance LLMs, a straightforward approach is to repeat the pre-training data for additional epochs. In this study, we empirically investigate three key aspects under this approach. First, we explore the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting, leading to multi-epoch degradation. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives, while less influential factors consist of dataset quality and model FLOPs. Finally, we explore whether widely used regularization can alleviate multi-epoch degradation. Most regularization techniques do not yield significant improvements, except for dropout, which demonstrates remarkable effectiveness but requires careful tuning when scaling up the model size. Additionally, we discover that leveraging mixture-of-experts (MoE) enables cost-effective and efficient hyper-parameter tuning for computationally intensive dense LLMs with comparable trainable parameters, potentially impacting efficient LLM development on a broader scale.

1 Introduction

High-quality web text may not meet compute-optimal LLM token demand, making repeated pre-training data an appealing but uncertain response. The paper studies multi-epoch training, its contributing factors, and whether regularization or MoE can help.

1 Introduction

  • Using Chinchilla scaling-law modeling, the authors estimate fully pre-training PaLM-540B needs 10.8 trillion tokens, versus about 9 trillion tokens in high-quality text stock, which grows 4–5% annually.Source: §1 · p. 2
  • Compute-optimal token estimates omit inference costs, and the token shortage is more severe for non-English data, which represents 44% of web content.The paper notes that English content makes up over 56% of the web and cites non-English task performance as a concern.Source: §1 · p. 2
  • Prior findings disagree on repeated-data training: one study reported harm, while another observed improvement after four epochs, motivating further investigation.The cited examples are Hoffmann et al. [8] and Taylor et al. [30].Source: §1 · p. 2
  • The paper presents a systematic empirical study and reports that larger models are more prone to overfitting with repeated data, while larger datasets alleviate degradation.It also reports that data quality does not mitigate degradation, parameter count matters even at fixed computation, and FLOPs have negligible effect.Source: §1 · pp. 2–3
  • Most tested transformer regularization techniques struggle, whereas dropout is effective, can be introduced later, and needs additional tuning at very large model sizes.Later introduction is reported to preserve faster early learning; see also the scaling caveat in p09.Source: §1 · p. 3
  • MoE models can provide a faster, more affordable way to tune hyperparameters for dense models with comparable parameters but substantially higher FLOPs.The authors suggest this approach may have broader implications for efficient LLM development.Source: §1 · p. 3

2 What Are the Consequences of Repeating Pre-training Data?

Repeated-data pretraining can degrade model performance and downstream results; the section tests how dataset size, model scale, and training objectives affect that degradation.

2 What Are the Consequences of Repeating Pre-training Data?

  • T5 on C4 followed Chinchilla scaling: the number of tokens required for full training grew linearly with model size.The token requirement is operationalized by the smaller model's training-token count when a larger model outperforms it at matched computation budgets.Source: §2 · p. 4
  • Under token scarcity, repeated-data training degrades performance, and T5-XL can underperform T5-Large trained with 4× data (229 vs 227 tokens).Larger models are more susceptible to overfitting when training data is insufficient.Source: §2 · p. 4
  • A model pretrained on 227 tokens lost 1.9 points in SQuAD F1 despite a 2.9-point pretraining validation score.The table reports pre-fine-tuning C4 Top-1 accuracy and downstream SQuAD Exact Match and F1.Source: §2 · p. 4, Table 1 · p. 4

3.1 Data

  • A larger dataset alleviated multi-epoch degradation, while approximately 227-token Wikipedia and C4 subsets showed similar degradation.The 227-token and 229-token comparison also varied batch size; a fixed-batch ablation is noted as supporting the dataset-size finding.Source: §3.1 · p. 5

3.2 Model

  • At comparable FLOPs, the 0.5×-parameter ParamShare model was less affected by repeats than the 2×-parameter MoE model; increasing FLOPs alone showed no clear degradation increase.Figure 5 distinguishes one-epoch ample-data runs with dashed lines from limited-data multi-epoch runs with solid lines.Source: Figure 5 · p. 5, §3.2 · p. 6
  • Dense and MoE models with comparable trainable parameters showed similar overfitting trends, although the MoE models used fewer FLOPs.The comparisons reported 0.39× and 0.32× the FLOPs; the authors propose MoE as a lower-cost way to predict larger dense-model behavior.Source: §3.2 · p. 6

3.3 Training Objective

  • UL2 was more prone to overfitting and had a larger downstream performance drop than vanilla T5 under limited-data multi-epoch training.The authors say UL2's faster learning underlies its behavior; direct validation-accuracy comparisons between the objectives are not possible.Source: §3.3 · pp. 6–7, Table 3 · p. 7

4 Can Regularization Alleviate Multi-Epoch Degradation?

Dropout mitigates multi-epoch degradation, while the section tests staged use and scaling behavior. It also evaluates MoE-based dropout tuning and its cost relative to dense-model tuning.

Regularization ablations

  • Dropout alone effectively alleviated multi-epoch degradation; adding other regularizers brought no further improvement, and weight decay could destabilize training.The ablation covers dropout, droppath, label-smoothing, and weight decay.Source: §4 · p. 7

Regularization caveat

  • DropPath may slow training and reduce hardware utilization because tensor parallelism requires communication across GPU or TPU cores.The authors identify this as a possible reason for DropPath's limited effectiveness in their setting.Source: §4 · p. 8

Gradual dropout

  • Late-stage dropout performed comparably to dropout from the start and outperformed it during early pretraining.The tested Large-level model used dropout 0.1 after 215 of 219 steps.Source: §4 · p. 8, Figure 8 · p. 8

Dropout across scales

  • Dropout improved performance across model scales, but the XL model still had a slight late-training validation-accuracy drop.The reported scale comparison used dropout rates of 0.0 or 0.1.Source: §4 · p. 8, Figure 9 · p. 8

MoE hyper-parameter tuning

  • 18K USD was the reported total cost of MoE-assisted dropout tuning, 0.48× the expense of directly tuning Dense XL.The sweep found nearly identical Dense XL and MoE Large curves, with optimal dropout rates of 0.2 or 0.3; Dense XL training once cost 7.4K USD.Source: §4 · p. 9, Figure 10 · p. 9

Dropout-tuned scaling

  • With an appropriate dropout rate, training on about 227 tokens still improved performance up to 2.8B parameters, over 1700× the 16M scale indicated by Chinchilla scaling.Figure 11 compares a model trained with enough data against limited-data models with and without dropout.Source: §4 · p. 9, Figure 11 · p. 9

6 Conclusion

The paper examines repeated-token training under the token-crisis, including its effects, possible mitigations, and the use of MoE to predict dense-model behavior. Its conclusion presents MoE as a way to accelerate broader LLM development.

6 Conclusion

  • The study examines repeated-token training under the token-crisis, explores its degradation and mitigation, and demonstrates MoE's use in predicting more computationally expensive dense-model behavior.The authors describe this as a potential means of accelerating LLM development more broadly.Source: §6 · p. 9

A.1 Selection of Encoder-Decoder T5 Model and C4 Dataset

  • T5 with C4 was selected because the dataset is open-sourced and easier to prepare and maintain, and the pairing follows established practice that facilitates comparison with prior research.The subsection also notes continuing uncertainty about whether decoder-only models consistently outperform encoder-decoder models.Source: §A.1 · p. 12

A.2 Lack of Significant Improvement of UL2 in Table 3

  • The experiments' lack of significant UL2 improvement over vanilla T5 may reflect the omission of dropout used in the original UL2 implementation [29].The authors suggest this omission could have caused a performance drop.Source: §A.2 · p. 12

A.3 Reason for Not Training a Larger Scale Model (10B+)

  • Compute limits led the authors to study models up to approximately 3B parameters rather than 10B+ models.They cite the costs in computation, finances, time, and carbon emissions of comprehensive experiments.Source: §A.3 · p. 12

A.4 Performance Drop Without Enough Data Even If We Are Using Dropout

  • Even with suitable dropout, a substantial performance gap remains between full-token training and repetition of a limited token set.The comparison repeats only 227 tokens for 28 epochs; the authors expect a smaller gap in a more typical setting such as 1T tokens repeated for 10 epochs.Source: §A.4 · p. 12

B.1 Multi-Epoch LLM Pre-training

This section situates the paper's repeated-data work amid mixed findings on multi-epoch language-model training and distinguishes it from synthetic-data approaches to scarcity.

B.1 Multi-Epoch LLM Pre-training

  • Earlier findings differ: Hoffmann et al. [8] suggested multiple epochs may harm performance, while Tay et al. [29] reported no degradation in a 120B model trained for four epochs.Tay et al. attributed their result primarily to high-quality data; other work found no clear benefit from deduplication [2] and harm from repeating a small data fraction [7].Source: §B.1 · p. 13
  • The paper focuses on the token-crisis by studying repeated training on a fixed amount of data, and presents MoE prediction of dense-model behavior as a novel contribution.The authors describe their work as the first to explore the token-crisis while training LLMs for multiple epochs.Source: §B.1 · p. 13

B.2 Pre-training with Synthetic Data

  • Prior synthetic-data work includes artificial languages, simpler synthetic tasks, and domain-specific data-scarcity methods such as TAPEX [15].The paper treats this line as relevant to token scarcity but distinct from its focus on multiple epochs.Source: §B.2 · p. 13

C Limitations

The paper identifies limits in model scale, dataset-quality assumptions, and MoE-based predictions when models differ substantially in FLOPs.

C Limitations

  • Experiments reach 3B parameters, not state-of-the-art scales such as GPT-3's 175B, because repeated experiments on larger models are prohibitively expensive.The authors cite the computational resources required for extensive experimentation.Source: C Limitations · p. 13
  • The dataset-quality insight compares C4 as low-quality data with Wikipedia as high-quality data, so other dataset characteristics may yield different results.The authors acknowledge C4's quality is acceptable and explain their choice by prior LLM use of Wikipedia and claims of its better quality.Source: C Limitations · p. 13
  • MoE predictions may be imperfect across large FLOPs gaps: the 64-expert Base model does not perfectly predict XL dense behavior, though it is more overfit-prone than the 16-expert Base model.The authors call for further investigation, noting very sparse models may have under-trained parameters.Source: C Limitations · p. 13

D Ablation Study on Batch Size of Larger Dataset Size

The batch-size ablation reports little multi-epoch degradation with a smaller batch, supporting the finding that larger datasets can alleviate degradation.

D Ablation Study on Batch Size of Larger Dataset Size

  • The comparison holds batch size and the number of repeats fixed while training for four times more steps to process four times more tokens.Source: Figure 12 · p. 14
  • Even with a smaller batch size, the model shows little multi-epoch degradation, further verifying that a larger dataset can alleviate it.The ablation fixes batch size and repeat count, then uses four times more steps to process four times more tokens.Source: D Ablation Study on Batch Size of… · p. 14, Figure 12 · p. 14
  • The smaller-batch result supports the claim that increasing dataset size can alleviate multi-epoch degradation.Source: D Ablation Study on Batch Size of… · p. 14

E Ablation Study on Number of Experts for MoE

With limited training tokens, MoE models with more experts and trainable parameters show more severe multi-epoch degradation.

E Ablation Study on Number of Experts for MoE

F Encoder-Decoder vs Decoder-Only

With C4 and Span-Corruption held fixed, the encoder-decoder outperforms the decoder-only model, while a parameter-comparable MoE decoder-only model performs almost as well as the encoder-decoder. The authors attribute the behavioral difference more to training objective than architecture.

F Encoder-Decoder vs Decoder-Only

  • The encoder-decoder is clearly better than decoder-only, while a MoE decoder-only model with comparable trainable parameters performs almost the same as the encoder-decoder.The authors suggest that the behavioral difference is more attributable to training objective than architecture, motivating their exploration of UL2.Source: F Encoder-Decoder vs Decoder-Only · pp. 14–15, Figure 14 · p. 15

G Ablation Study on Batch Size of Larger Dataset Size

The larger-dataset experiment reports little multi-epoch degradation even with a smaller batch size, supporting dataset size as a mitigating factor. The supplied downstream-evaluation table caption identifies five-run standard deviations but gives no outcome values.

G Ablation Study on Batch Size of Larger Dataset Size

  • With fixed batch size and repeats, training for 4 times more steps to consume 4 times more tokens still yields little multi-epoch degradation at the smaller batch size, supporting the role of larger datasets.The passage says this further verifies that larger datasets can alleviate multi-epoch degradation.Source: Figure 15 · p. 15
  • The larger-dataset comparison changes token exposure by increasing training steps while holding batch size and repeats fixed.Source: Figure 15 · p. 15

H Downsteam Evaluation

  • The updated fine-tuning results are presented with standard deviations across 5 runs; the supplied caption does not state the results.Source: Table 5 · p. 15

I Verification of MoE Behavior Prediction with Dropout

The paper reports that MoE models can approximate dense-model behavior, while supplementary experiments show mixed effects from batch size, sequence length, embedding randomization, and weight masking. Its abstract also summarizes dropout’s effectiveness and tuning needs, and contrasts encoder-decoder with decoder-only behavior.

I Verification of MoE Behavior Prediction with Dropout

J Default Hyper-parameters; K Model Configurations

L.1 Training with Larger or Smaller Batch

  • Smaller batch sizes can reduce overfitting, but the authors consider this an inefficient way to address multi-epoch degradation.Source: §L.1 · p. 19

L.2 Training with Longer or Shorter Sequence

  • With the same total token usage, training on longer sequences achieves better performance than using shorter sequences.Source: §L.2 · p. 19

L.3 Randomly Activating One Embedding Layer from Multiple Embedding Layers

  • Randomly activating among multiple input embedding layers does not alleviate multi-epoch degradation, despite regularizing the layers to have different embeddings.Source: §L.3 · p. 19

L.4 Adding Dropout Before and After Linear Layers

  • Dropout before and after linear layers acts as weight masking and is too strong, producing inferior XL-level performance even at a small dropout rate.After-layer dropout is column-wise masking; applying it before and after a layer denotes element-wise masking.Source: §L.4 · p. 19

Abstract

  • The abstract reports that most regularization methods do not significantly alleviate degradation, whereas dropout is effective but needs careful tuning as model size scales.Source: Abstract · p. 20

A.1 Encoder-Decoder vs Decoder-Only

  • The encoder-decoder outperforms decoder-only, but a parameter-comparable MoE decoder-only model performs almost the same; the authors suggest training objective matters more than architecture.Source: §A.1 · p. 21

A.3 Downsteam Evaluation

Across the three reported C4 settings, Repeat28 has lower values than Repeat1 in all four listed downstream scores.

A.3 Downsteam Evaluation

  • For C4-SpanCorr, Repeat28 scores (73.8±1.0, 66.4±0.3, 80.0±0.6, 88.1±0.5) are below Repeat1 (77.0±0.7, 68.9±0.8, 82.3±0.9, 90.0±0.9).Source: §A.3 · p. 21
  • For C4-UL2, Repeat28 scores (74.5±0.6, 68.4±0.7, 79.3±1.1, 87.3±1.2) are below Repeat1 (78.8±0.9, 72.2±1.2, 82.3±0.8, 90.2±0.7).Source: §A.3 · p. 21
  • For C4-Wiki, Repeat28 scores (71.4±0.4, 64.9±0.5, 79.6±1.0, 87.9±1.3) are below Repeat1 (74.2±0.7, 69.3±1.1, 82.6±0.8, 90.1±0.6).Source: §A.3 · p. 21

Related papers

Loading 2305.13230v2…