Source-linked AI summary
Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning
Yang Zhang, Amr Mohamed, Hadi Abdine, Guokan Shang, Michalis Vazirgiannis
TL;DR
LLM pretraining is costly, and data ordering remains an underexplored efficiency direction. The paper systematically studies curriculum strategies and difficulty metrics, finding faster convergence and lasting performance gains when curriculum learning is used as a warmup before random sampling.
Problem
LLM pretraining is computationally and data intensive, while the ordering of training data remains largely overlooked and effective curriculum strategies are underexplored.
Source: §1 · p. 1Method
The paper compares vanilla, pacing-based, and interleaved curricula using six difficulty metrics across training scenarios and benchmarks.
Source: §1 · p. 1, §3.3 · p. 3, §3.3.1 · p. 4Results
Takeaways & Limitations
Compression ratio, MTLD, and Flesch Reading Ease were the most effective indicators, while curriculum warmup emerged as a robust and practical setting for lasting gains.
Source: §6 · p. 9Experiments use LLaMA3.2-like models and English data, and difficulty scores are static rather than adaptive to changing model capabilities.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Curriculum learning-organizing training data from easy to hard-has improved efficiency across machine learning domains, yet remains underexplored for language model pretraining. We present the first systematic investigation of curriculum learning in LLM pretraining, with over 200 models trained on up to 100B tokens across three strategies: vanilla curriculum learning, pacing-based sampling, and interleaved curricula, guided by six difficulty metrics spanning linguistic and information-theoretic properties. We evaluate performance on eight benchmarks under three realistic scenarios: limited data, unlimited data, and continual training. Our experiments show that curriculum learning consistently accelerates convergence in early and mid-training phases,reducing training steps by $18-45\%$ to reach baseline performance. When applied as a warmup strategy before standard random sampling, curriculum learning yields sustained improvements up to $3.5\%$. We identify compression ratio, lexical diversity (MTLD), and readability (Flesch Reading Ease) as the most effective difficulty signals. Our findings demonstrate that data ordering-orthogonal to existing data selection methods-provides a practical mechanism for more efficient LLM pretraining.
1 Introduction
The paper investigates whether ordering training data by difficulty can make LLM pretraining more efficient, an underexplored alternative to improving data quality or mixtures. It compares curriculum strategies and difficulty signals, finding faster convergence and lasting gains from curriculum warmup.
1 Introduction
- LLM pretraining remains costly, while prior efficiency efforts emphasize data quality and mixtures rather than training-data ordering.The paper frames curriculum learning as a way to structure data from easy to hard.Source: §1 · p. 1
- The study compares three curriculum strategies and six difficulty metrics across three training scenarios, models from 0.5B to 3B parameters, and up to 100B tokens.Evaluations cover eight benchmarks spanning commonsense reasoning, language understanding, and reading comprehension.Source: §1 · pp. 1–2
- 18–45% fewer training steps were needed to reach baseline performance, and curriculum warmup before random sampling improved performance by up to 3.5%.Advantages were especially consistent in early and mid-training phases and depended on combinations of difficulty metrics and curriculum strategies.Source: §1 · p. 2
- The work presents curriculum-based warmup as a practical mechanism for efficient training and reports a comprehensive study across three realistic scenarios.Its contributions also include analysis of six difficulty metrics and their effects on convergence and performance.Source: §1 · p. 2
2 Related Work
Prior efficiency methods reduce costs through filtering, reweighting, or data selection, while curriculum learning structures examples by difficulty. This paper positions pretraining data ordering and systematic difficulty-metric comparisons as complementary, less-explored directions.
Data-Efficient LLM Training
- Data-efficient pretraining methods use filtering, robust domain mixing, embedding or influence-based sampling, and quality or diversity signals to reduce token use.These approaches select or reweight data rather than order a prefiltered dataset by curriculum.Source: §2 · p. 2
Curriculum Learning
- Prior curriculum work covers NLP, LSTMs, transformers, and LLM fine-tuning, but largely focuses on fine-tuning or narrower setups rather than systematic LLM pretraining.The paper characterizes its work as the first systematic study of multiple pretraining paradigms and difficulty metrics at scale.Source: §2 · p. 2
Text Difficulty Estimation
- Difficulty estimation uses length and term-frequency heuristics, lexical-diversity measures such as MTLD, and information-theoretic measures such as compression ratio and entropy.The cited measures provide candidate signals for curriculum learning.Source: §2 · pp. 2–3
3 Methodology
The methodology selects difficulty signals and applies them through fixed-set ordering, paced sampling, or interleaving under distinct data-access scenarios. Experiments use an English pretraining corpus and evaluate models across benchmarks.
3 Methodology
- The methodology examines difficulty metrics, training scenarios, and curriculum settings for pretraining.The section evaluates candidate metrics for distinct linguistic and information-theoretic properties while considering large-scale training constraints.Source: §3 · p. 3, §3.1 · p. 3
3.1 Difficulty Metrics
- Six minimally correlated metrics represent information density, lexical diversity, readability, fertility, model-perceived difficulty, and sequence length.The selected signals are compression ratio, fertility, Flesch Reading Ease, MTLD, number of tokens, and perplexity; they are intended to be broadly representative and inexpensive to compute.Source: §3.1 · p. 3
3.2 Vanilla Curriculum Learning
- In the limited-data scenario, all fixed pretraining data must be used and is sorted by difficulty from easy to hard.This strict ordering introduces increasingly difficult samples during training.Source: §3.2 · p. 3
3.3 Curriculum Learning with Pacing
- With a large dataset and fixed token budget, pacing schedules control difficulty over time by sampling from difficulty groups.Linear pacing allocates tokens equally across groups; quadratic pacing shifts faster toward harder groups, while inverse quadratic pacing allocates more tokens to easier groups early.Source: §3.3 · pp. 3–4
3.3.1 Interleaved Curriculum Learning
- Interleaved curricula divide training into repeated token-budget segments, each progressing linearly from easier to harder groups.This repeatedly exposes the model to the full difficulty spectrum while preserving progression within each interleave.Source: §3.3.1 · p. 4
3.4 Experimental setting
- Experiments use CulturaX’s English subset and LLaMA3.2-like decoder-only models, evaluating eight benchmarks in zero-shot settings.The benchmark set covers commonsense reasoning, language understanding, reading comprehension, and world knowledge.Source: §3.4 · p. 4
4 Experiments
The experiments compare curriculum strategies using average benchmark accuracy against randomly shuffled baselines. Gains depend on the difficulty metric and strategy, with warmup results showing sustained advantages after switching to random data.
4.1 Experiment 1: Strict ordering
- 1.8% higher best performance with 17.9% fewer steps for MTLD; Number of Tokens reached similar performance with 27.5% fewer steps.Except for Flesch Reading Ease, strict-ordering curricula generally outperformed the random baseline early, with results converging or remaining slightly higher by training’s end.Source: Figure 2 · p. 5, §4.1 · pp. 5–6
4.2 Experiment 2: Pacing Functions
- 39.5% fewer steps for Compression Ratio to reach baseline peak accuracy, 31.7% fewer for MTLD, and 45.3% fewer for Fertility to reach 99.5% of baseline peak under linear pacing.These metrics tied to linguistic richness and redundancy benefited most from linear pacing; quadratic pacing worked best for Number of Tokens and Flesch Reading Ease.Source: Figure 3 · p. 5, §4.2 · p. 6
- 1.1% higher final accuracy for Number of Tokens with quadratic pacing; Flesch Reading Ease also favored quadratic pacing, finishing 1.6% above baseline.All pacing strategies beat baseline for Number of Tokens between 3B and 8B tokens; Flesch Reading Ease rose steadily across pacing functions.Source: Figure 3 · p. 5, §4.2 · p. 6
4.3 Experiment 3: Interleave Curriculum Learning
- 2.2% higher performance for interleaved Compression Ratio, reaching the best random-baseline performance with 41.7% fewer steps; Number of Tokens reached it with 23.5% less data.Other interleaved metrics were near baseline or slightly worse, and MTLD performed worse during training.Source: Figure 4 · p. 6, §4.3 · p. 7
4.1 Experiment 1: Strict ordering
- Perplexity ordering showed strong early gains but dropped later, while high-perplexity samples were hypothesized to be noisy and potentially destabilizing.Experiments report average benchmark accuracy because training or validation loss poorly captures generalization and downstream performance.Source: §4 · p. 5, §4.1 · p. 6
4.4 Curriculum Learning as model warmup
- 3.5% and 2.6% higher than baseline, respectively, for best- and last-checkpoint MTLD warmups; the baseline never surpassed the first-phase curriculum result with double the data.Both warmups continued on randomly shuffled data; the reported comparison is for MTLD.Source: Figure 5 · p. 7, §4.4 · p. 7
5 Ablation Study
The ablations examine difficulty-group choices, run-to-run robustness, data ordering versus filtering, and curriculum effects at larger scales. Gains persisted across repetitions and scales, while early curriculum data alone helped but did not match ordered training.
5.1 Number of difficulty groups
- Ten difficulty groups balanced smooth transitions with intra-group diversity; using three groups yielded nearly equal or worse performance, and 20 groups brought no early-stage improvement.The authors suggest fewer groups make transitions coarser, while more groups yield diminishing returns.Source: §5.1 · p. 8
5.2 Robustness of Curriculum Learning
- Across independent repetitions of sampling and training, the curriculum findings held on average with consistent trends.The baseline and six high-performing settings were repeated twice.Source: §5.2 · p. 8
5.3 Data sampling vs Curriculum Learning
- Early curriculum data acted as a filter and improved on the random baseline, but still underperformed the original ordered setting.The comparison shuffled data used before the best-performing curriculum checkpoint and retrained on that subset.Source: §5.3 · p. 8
5.4 Scale Curriculum Learning
- Curriculum benefits extended to larger models, with warmup delivering sustained performance improvements throughout training.The scale experiments trained 1B- and 3B-parameter models with 20B and 40B tokens, respectively.Source: §5.4 · p. 8
5.5 Scale Curriculum Learning
- Approximately 3.1% improvement at 100B tokens; warmup outperformed baseline across checkpoints and its gains extended to larger models and datasets.The reported scale experiments include 1B- and 3B-parameter models trained on 20B and 40B tokens, respectively.Source: §5.5 · p. 8
6 Discussion and Conclusion
Curriculum learning accelerates LLM pretraining and can produce lasting gains when used as a warmup before random sampling. Compression Ratio, MTLD, and Flesch Reading Ease are among the strongest difficulty signals, and ordering is complementary to data selection.
6 Discussion and Conclusion
- The study compares vanilla, pacing-based, and interleaved curricula across six difficulty metrics in its systematic exploration of LLM pretraining.The authors identify adaptive and model-aware curricula as future directions.Source: §6 · p. 9
- Curriculum learning reduces training steps by 18–45% and improves final performance by up to 3.5% when used as warmup before random sampling.The warmup strategy speeds early and mid-stage training while yielding lasting gains in later phases.Source: §6 · p. 9
- Compression Ratio, MTLD, and Flesch Reading Ease are the most effective examined difficulty signals, while perplexity may conflate difficulty with noise.Linear pacing and interleaved strategies further enhance generalization by gradually and repeatedly exposing the model to increasing difficulty.Source: §6 · p. 9
- Curriculum learning uses lightweight, precomputable difficulty metrics and data ordering, making it orthogonal to data selection and pruning.The authors state that it integrates into existing pretraining pipelines without computational overhead.Source: §6 · p. 9
Limitations
The study’s evidence is bounded by its model and data scope, static curricula, metric selection, and focus on pretraining rather than downstream fine-tuning.
Limitations
- Experiments use decoder-only LLaMA3.2-like models and English CulturaX data, which may limit generalizability to encoder-based models or multilingual settings.The work trains 0.5B and 1B parameter models.Source: Limitations · p. 9
- Difficulty scores are static and precomputed, so the study does not test curricula that adapt to the model’s evolving capabilities.Model-aware or task-informed pacing could yield further gains.Source: Limitations · p. 9
- The six tested metrics were selected through correlation analysis; syntactic depth and coherence were not explored, and perplexity may conflate difficulty with noise.Source: Limitations · p. 9
- The study examines curriculum learning during pretraining only, not its impact on downstream fine-tuning.Source: Limitations · p. 9
A Metrics Selection
The authors screened fifteen candidate difficulty metrics for redundancy and selected six signals spanning linguistic and information-theoretic properties.
A Metrics Selection
- Fifteen candidate metrics span information density, lexical diversity, readability, sequence length, fertility, and perplexity.The candidates include Compression Ratio, several lexical diversity and readability measures, Number of Tokens, Fertility Score, and KenLM perplexity.Source: A Metrics Selection · p. 13
- Readability formulas show strong positive correlations, diversity measures cluster moderately, and perplexity or fertility are near-orthogonal to most other metrics.Figure 6 visualizes Spearman correlations among the fifteen candidates, with cell shading encoding correlation strength.Source: A Metrics Selection · p. 13, Figure 6 · p. 13
- The final six signals are Compression Ratio, Flesch Reading Ease, MTLD, Perplexity, Fertility, and Number of Tokens.The selection retains one representative from highly correlated groups and preserves metrics capturing unique aspects of text complexity.Source: A Metrics Selection · p. 13
B Curriculum Learning as Warmup
In continual training on randomly shuffled data, curriculum warmup settings retain accuracy and reach baseline performance in fewer steps; using three rather than ten difficulty groups performs similarly or worse.
B Curriculum Learning as Warmup
- Flesch Reading Ease with quadratic pacing achieves 1.5% higher accuracy and reaches best baseline performance with 40.0% fewer steps.The model continues from its last checkpoint, which is also its best; Figure 7 compares average accuracy during continual training.Source: B Curriculum Learning as Warmup · p. 14, Figure 7 · p. 14
- Number of Tokens with vanilla curriculum learning achieves 2.2% higher accuracy and reaches best baseline performance with 20.7% fewer steps.Starting from the best checkpoint is slightly better than starting from the last; Figure 7 compares average accuracy during continual training.Source: B Curriculum Learning as Warmup · p. 14, Figure 7 · p. 14
C Number of difficulty groups
- With linear pacing, three difficulty groups perform generally worse or similarly to ten groups, and overall performance is less sensitive to group count.The authors hypothesize that three groups create coarser transitions and may lead to overfitting to each group; Figure 8 compares the two group counts.Source: C Number of difficulty groups · pp. 14–15, Figure 8 · p. 14
D Robustness of Curriculum Learning
Curriculum ordering outperforms shuffling the same selected data, and the tested curriculum settings remain robust across repeated runs. The section also describes the Cerebras hardware and communication systems used for training.
D Robustness of Curriculum Learning
- Across six high-performing settings, curriculum learning shows consistent improvements with little overlap in run-to-run variation with the baseline.The comparison averages accuracy over three runs; shaded regions show standard deviations.Source: Figure 9 · p. 15, D Robustness of Curriculum Learning · p. 15
E Sampling vs Ordering
- Ordered curriculum versions outperform randomly shuffled subsets of the same selected data, indicating that benefits extend beyond data selection.The shuffled subsets still beat the full random baseline, suggesting a filtering effect as well.Source: Figure 10 · p. 15, E Sampling vs Ordering · pp. 15–16
F Cerebras
- MemoryX stores weights, gradients, and optimizer states; SwarmX broadcasts layer weights to CS-2 systems and aggregates their gradients for updates.The systems use weight streaming and data parallelism across CS-2s.Source: F Cerebras · p. 16
F.1 Reproducibility on Nvidia GPUs
At scale, Flesch Reading Ease with quadratic pacing retains an improvement through training, while vanilla curriculum settings generally improve faster early and mid-training. A hardware comparison reports less than 0.5% relative performance differences across benchmarks.
F.1 Reproducibility on Nvidia GPUs
- Across eight benchmarks, the A100- and Cerebras-trained models differ by less than 0.5% in relative performance.The authors therefore consider the hardware-trained models equivalent with respect to performance.Source: §F.1 · p. 16, Table 1 · p. 16
G Scale Curriculum Learning
- Key curriculum settings were replicated using 1B and 3B Llama3.2-configured models with the previous experiments’ training hyperparameters.The vanilla curriculum experiment selects 20B tokens and creates six strictly ordered sets plus a baseline set.Source: G Scale Curriculum Learning · p. 16, §G.1 · p. 16
G.1 Vanilla Curriculum Learning
- MTLD consistently outperforms baseline; Number of Tokens improves faster early and mid-training, while Flesch Reading Ease ends well above baseline.Most vanilla curriculum settings reach their best performance faster than baseline.Source: §G.1 · pp. 16–17, Figure 11 · p. 17
G.2 Flesch Reading Ease with quadratic pacing
- With Flesch Reading Ease and quadratic pacing, the model achieves 2.5% higher accuracy and maintains stable performance gains through the end of training, with no sign of convergence.Source: §G.2 · p. 17, Figure 12 · p. 17
G.3 Curriculum Learning as Warmup
Curriculum warmup followed by randomly sampled training yields higher performance across the reported model scales. The paper also reports that computing difficulty metrics and ordering data adds negligible cost relative to pretraining.
G.3 Curriculum Learning as Warmup
- 3.2% (Flesch Reading Ease) and 2.1% (MTLD) higher peak performance, a 2.0% gain at 3B, and about 3.1% absolute improvement at 100B follow curriculum warmup.Warmup models continue with randomly sampled training; the 100B-token comparison reports superiority across checkpoints.Source: §G.3 · pp. 18–19, Figure 13 · p. 18, Figure 14 · p. 18
- CL warmup with Flesch Reading Ease and quadratic pacing or MTLD and vanilla CL used 20B pretraining tokens, followed by 10B randomly shuffled continuation tokens from either the best or last checkpoint, achieving 3.2% and 2.1% higher peak performance than baseline, respectively.Source: §G.3 · p. 18
H Computation overhead
- Difficulty scoring and ordering add $2 in data-loading and ordering costs, negligible beside the estimated 84 A100 GPU-hours for pretraining.The estimates are for computing difficulty metrics over 10B tokens on a 72-vCPU AWS machine.Source: H Computation overhead · p. 19, Table 2 · p. 19