Source-linked AI summary
Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning
Dawid J. Kopiczko, Sagar Vaze, Tijmen Blankevoort, Yuki M. Asano
TL;DR
The paper investigates whether unique-data scaling is necessary for effective reasoning SFT. Through controlled fixed-update experiments, it compares epoch scaling on smaller datasets with data scaling on larger ones. Repetition consistently improves performance, with memorization and training token accuracy indicating when gains saturate, while the underlying mechanism remains unresolved.
Problem
The paper examines whether the standard assumption that more unique training samples improve generalization applies to long-CoT SFT for reasoning models.
Method
The authors vary epochs and unique samples while holding total gradient updates and other parameters constant across models, datasets, and reasoning benchmarks.
Results
Under a fixed update budget, repeating smaller datasets substantially outperforms scaling unique samples, with gains plateauing at near-perfect training token accuracy and no additional catastrophic forgetting.
Takeaways & Limitations
Training token accuracy provides a practical stopping criterion for epoch scaling, while the repetition advantage remains a robust open problem in reasoning SFT.
Takeaways & Limitations
The study identifies correlates of improved performance but does not establish a definitive causal mechanism for the repetition advantage.
Abstract
from arXiv · showhide
Supervised fine-tuning (SFT) on chain-of-thought data is an essential post-training step for reasoning language models. Standard machine learning intuition suggests that training with more unique training samples yields better generalization. Counterintuitively, we show that SFT benefits from repetition: under a fixed update budget, training for more epochs on smaller datasets outperforms single-epoch training on larger datasets. On AIME'24/25 and GPQA benchmarks, Olmo3-7B trained for 128 epochs on 400 samples outperforms the equivalent 1 epoch on 51200 samples by 12-26 percentage points, with no additional catastrophic forgetting. We find that training token accuracy reliably signals when repetition has saturated; improvements from additional epochs plateau at full memorization, a pattern consistent across all settings. These findings provide a practical approach for reasoning SFT, where scaling epochs with token accuracy as a stopping criterion can replace expensive undirected data scaling. We pose the repetition advantage, where full memorization coincides with improved generalization, as a new open problem for the community in understanding the training dynamics of large language models.
1 Introduction
Reasoning SFT commonly uses long-CoT demonstrations, but the paper reports that repeating smaller datasets can outperform scaling unique samples under a fixed update budget. The authors link this advantage to memorization and propose training token accuracy as a practical stopping signal while identifying the phenomenon as an open problem.
- Long-CoT supervised fine-tuning is a post-training stage that primes reasoning models for later reinforcement-learning stages.
- Standard machine-learning intuition predicts that more unique samples should improve generalization.
- Under a fixed update budget, repeating smaller datasets can outperform training on larger datasets.
- Performance gains plateau when models reach near-perfect training token accuracy, even as validation loss continues to rise.
- The paper identifies epoch scaling, training token accuracy, and data properties as key factors in the repetition advantage, without additional catastrophic forgetting.
- Explaining why repetition improves generalization in long-CoT SFT remains an open problem.
2 Scaling Epochs on a Fixed Update Budget
The study compares supervised fine-tuning configurations at fixed update budgets, varying dataset size, repetition, and model or evaluation settings. Across these controlled comparisons, more epochs on fewer samples generally outperform scaling unique samples, with gains diminishing after higher repetition.
- Experimental design: Controlled experiments vary epochs and unique samples while holding total gradient updates and other parameters constant.The study trains base checkpoints on chain-of-thought data and evaluates them on challenging reasoning benchmarks.
- Results: The repetition advantage persists across models, benchmarks, and evaluation metrics, with normalized performance improving toward fewer samples and more epochs.Figure 2 covers Olmo3-7B and Qwen3-8B on AIME’24, AIME’25, and GPQA.
- Experimental design: Fixed update budgets isolate the effect of data repetition from differences in total optimization steps.For batch size one, the budget equals epochs multiplied by unique samples.
- Results: Across three models, 16 epochs on 3200 samples substantially outperforms 1 epoch on 51200 samples across all benchmarks.Table 1 compares configurations with equivalent update budgets while varying the epochs-to-samples ratio.
- Results: At 51200 updates, 32 epochs on 1600 samples reaches 39% average accuracy, versus 17% for 1 epoch on 51200 samples.The comparison is averaged across benchmarks and uses the same update budget.
- Results: Gains diminish around 32–64 epochs, suggesting a saturation point beyond which additional repetition provides limited benefit.The study also reports that the advantage persists for both the 0.6B and 8B teacher settings, although weaker-teacher data can have lower peak performance at a larger update budget.
3 Impact of Training Data
The repetition advantage persists across teacher quality and trajectory correctness, but teacher quality affects whether scaling unique data remains beneficial. Across settings, performance improves with epoch scaling until memorization saturates.
- Teacher Model Quality: Epoch scaling improves performance more reliably than data scaling for both 0.6B and 8B teachers.The repetition advantage persists regardless of teacher size.
- Teacher Model Quality: 54.0% versus 49.5%: with the 0.6B teacher, the highest average pass rate is higher for B = 6400 than B = 25600.Additional samples degrade average performance in this setting.
- Teacher Model Quality: 55.0% versus 66.6%: with the 8B teacher, the highest average pass rate increases when B grows from 6400 to 25600.Teacher quality determines whether data scaling remains beneficial, while repetition remains advantageous.
- Negative Trajectories: 40.0% versus 37.3% on AIME’24 and 29.3% versus 23.4% on GPQA: negative trajectories match or exceed positive trajectories.The repetition advantage persists when training traces have incorrect final answers.
- Memorization: Performance improves with epoch count until full memorization, after which gains plateau or degrade.Table 4 measures memorization as response-token top-prediction accuracy at fixed B = 51200.
4 Probing the Repetition Advantage
The paper probes repetition through memorization, termination, overfitting, and forgetting. Memorization and termination track improved reasoning performance, while standard overfitting indicators rise without preventing gains and epoch scaling causes less forgetting than data scaling.
- Scope of the Probe: The study identifies correlates of improved performance but does not establish a definitive causal mechanism.These observations are presented as empirical characterizations for future investigation.
- Memorization Signals Convergence: Near-perfect training-token accuracy occurs primarily through epochs, independently of dataset size, and downstream gains plateau near full memorization.Models trained for 16 epochs achieve near-perfect memorization across 200- and 3200-sample datasets.
- Termination Correlates with Performance: 24% versus 89%: termination rises from single-epoch models to 32-epoch models and strongly correlates with accuracy.Repeated exposure appears to internalize the convention of concluding long reasoning chains.
- Overfitting Paradox: Train loss approaches zero, validation loss rises, and prediction entropy decreases as downstream accuracy continues improving with epoch count.These standard metrics indicate overfitting despite monotonic downstream gains.
- Overfitting Paradox: The paper interprets repeated training as potentially eliciting latent capabilities rather than teaching genuinely new skills.This interpretation links confidence in divergent reasoning patterns with transfer to held-out benchmarks.
- Catastrophic Forgetting: Epoch scaling causes less catastrophic forgetting than data scaling under matched total gradient updates.Both strategies cause some forgetting relative to the base model, but epoch scaling combines less degradation with larger reasoning gains.
5 Related Work
Related work frames the repetition advantage against pretraining scaling laws, existing multi-epoch SFT practice, and research on memorization and generalization. These literatures motivate the paper’s focus on training dynamics.
- Data Repetition and Scaling Laws in Pretraining: Pretraining scaling laws relate validation loss to model size, training tokens, and compute, while commonly favoring additional fresh data over corpus repetition.The paper distinguishes this heuristic from its long-CoT SFT setting.
- Memorization, Overfitting, and Training Dynamics: Epoch scaling on a fixed small dataset causes less forgetting than single-epoch training on larger datasets under matched update budgets.This comparison connects the paper’s result to the broader concern that specialization can erase general capabilities.
- Multi-epoch SFT in Post-training Practice: Recent post-training pipelines often perform multiple SFT epochs without isolating epoch count as a studied variable.The paper positions its controlled epoch-versus-data comparison within this practice.
- Memorization, Overfitting, and Training Dynamics: Prior deep-learning and language-modeling studies show that memorization does not necessarily preclude generalization and depends on model, data, and optimization choices.These findings provide context for examining memorization during long-CoT SFT.
6 Conclusion
The study finds that repetition on smaller datasets can outperform unique-data scaling under a fixed update budget, while the best dataset size remains data- and model-dependent. It recommends treating epoch count and dataset size as joint decisions and identifies the underlying mechanism as an open problem.
- Under a fixed update budget, more epochs on smaller datasets substantially outperform training on larger datasets across the studied models, benchmarks, and data sources.
- Training token accuracy provides a practical stopping signal for epoch scaling, but it does not determine the optimal dataset size in advance.
- The mechanism by which memorization under repetition improves generalization in reasoning SFT remains poorly understood and is posed as an open problem.
C Termination-conditioned Results
The GPQA evaluation reports termination-conditioned accuracy for Olmo3-7B at a fixed update budget. Results are based on 198 problems with four generations per problem.
- 198 problems are used for the GPQA evaluation at a fixed update budget of B = 51,200.
- Each GPQA problem receives four generations during evaluation.
- “Correct (% terminated)” measures accuracy conditioned on generation reaching an end-of-sequence token.
D Training Loss.
Figure 6 compares training-loss curves for Olmo3-7B configurations under a fixed update budget. The configurations vary unique sample counts and epoch counts, and the curves are shown on linear and logarithmic scales.
- Figure 6 uses a fixed update budget of B = 51,200 for all Olmo3-7B configurations.
- The configurations vary the number of unique training samples and epochs while keeping the update budget fixed.
- Training-loss curves are shown on both linear and logarithmic scales.
E.1 Dolci Dataset
Dolci results show a consistent repetition advantage across three model backbones. More epochs on smaller datasets outperform fewer epochs on larger datasets.
- Across all three model backbones, more epochs on smaller datasets consistently outperform fewer epochs on larger datasets.
- Figure 7 reports Dolci dataset results for Olmo3-7B.
- Figure 8 reports Dolci dataset results for Qwen3-8B.
- Figure 9 reports Dolci dataset results for Qwen3-4B.
E.2 Qwen3 Distills
The distillation experiments show that repetition improves downstream accuracy across teacher strengths. Stronger teachers raise absolute performance, but multi-epoch training on smaller distilled datasets retains an advantage over scaling unique samples.
- Multi-epoch training on smaller distilled datasets consistently outperforms scaling unique samples.
- Stronger teachers improve absolute performance without eliminating the repetition advantage.
- Repetition continues to improve downstream accuracy despite weaker signals from a Qwen3-0.6B teacher.
- Qwen3 8B Distill; Pos. vs Neg.: Figures 12 and 13 compare distilled samples separated by correctness, using positive and negative samples from a Qwen3-8B teacher.