PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Curriculum Learning in Large Language Models via Cross-Difficulty Optimization Dynamics

Zhikai Ding, Ziyi Ye

arXiv:2608.17268v1cs.LGcs.AI

TL;DR

Curriculum learning’s effectiveness varies across reasoning tasks, but the mechanism determining when it works remains unclear. This paper analyzes cross-difficulty optimization transfer, formalizes Relative Transfer, and derives TDCS, which consistently outperforms fixed scheduling strategies across benchmarks and model scales.

  • Problem

    Existing evidence shows curriculum learning can help, fail, or harm across reasoning tasks, while the mechanism determining its effectiveness remains unresolved.

  • Method

    The paper analyzes cross-difficulty optimization interactions, formalizes them as Relative Transfer, and uses estimated transfer to dynamically adjust sampling through TDCS.

  • Results

    TDCS outperforms the strongest fixed scheduling strategy by 2.6% on Sudoku, 3.8% on iGSM, and 1.2% on KodCode.

  • Takeaways & Limitations

    Cross-difficulty transfer provides an optimization-based explanation of task-dependent curriculum effectiveness and supports adaptive sampling across reasoning settings.

Abstract

from arXiv · show

Curriculum learning has been widely adopted in the post-training of large language models by organizing training data from easy to hard. However, its effectiveness varies substantially across reasoning tasks, suggesting that no single curriculum is universally optimal and raising a fundamental question: what determines when curriculum learning works? In this paper, we answer this question by analyzing the optimization dynamics induced by different curriculum schedules. We show that the transfer relationship between different difficulty levels characterizes the optimization dynamics induced by curriculum learning, which in turn explains the effectiveness of different curriculum schedules, and formalize this relationship as Relative Transfer, a principled measure of cross-difficulty knowledge transfer. Based on this measurement, we derive Transfer-aware Dynamic Curriculum Sampling (TDCS), which dynamically adjusts the sampling distribution according to the estimated transfer relationship throughout training. Extensive experiments on multiple reasoning benchmarks demonstrate that TDCS consistently outperforms representative scheduling strategies across different tasks, model scales, and training paradigms. More importantly, our work provides a unified optimization-based explanation of curriculum learning through cross-difficulty transfer.

1 Introduction

The paper explains curriculum learning through cross-difficulty optimization dynamics, showing that transfer between difficulty levels determines when fixed schedules succeed or fail. It formalizes this mechanism as Relative Transfer and derives TDCS, which adaptively samples according to estimated transfer benefits.

  • Motivation: Easy-to-hard curriculum learning is widely adopted in LLM post-training, but its effectiveness varies substantially across reasoning tasks.It can outperform random sampling in some scenarios yet be ineffective or detrimental in others.
  • Motivation: No fixed scheduling strategy consistently performs best across different reasoning tasks.The paper establishes this through a systematic empirical study across multiple reasoning benchmarks.
  • Optimization Dynamics: Cross-difficulty transfer determines whether curriculum schedules succeed by facilitating or interfering with optimization across difficulty levels.The paper formalizes this optimization mechanism through Relative Transfer.
  • Method: TDCS dynamically adjusts the sampling distribution according to estimated transfer relationships instead of following a predefined easy-to-hard schedule.It allocates samples based on estimated transfer benefits across difficulty levels throughout training.
  • Results: TDCS consistently outperforms existing fixed scheduling strategies across multiple reasoning benchmarks, model scales, and self-improvement settings.The experiments also show generalization to downstream self-improvement settings.

2 Related Work

Prior work develops curricula by ordering samples according to difficulty and extends curriculum learning to LLM post-training through varied difficulty and optimization signals. This work instead explains curriculum effectiveness through cross-difficulty optimization and transfer relationships rather than predefined schedules or optimization heuristics alone.

  • Curriculum Learning: Curriculum learning orders samples from easy to hard using predefined difficulty measures, alongside variants such as self-paced and mentor-guided learning.These approaches aim to facilitate optimization through structured sample ordering.
  • Curriculum Learning: LLM post-training curricula use reasoning complexity, model confidence, training loss, or estimated sample difficulty, improving reasoning and instruction-following performance.Existing methods mainly emphasize designing difficulty metrics or curriculum schedules.
  • Research Gap: This work studies when and why curriculum schedules become effective through cross-difficulty optimization analysis rather than proposing another predefined schedule.Its focus is the optimization mechanism underlying curriculum learning.
  • Adaptive Data Scheduling: Adaptive data scheduling adjusts sampling distributions throughout training using loss, uncertainty, gradient, or reinforcement learning signals to improve optimization efficiency.These methods dynamically allocate training resources according to the current optimization state.
  • Research Gap: Unlike methods driven primarily by current-state optimization signals, this work uses transfer relationships between difficulty levels as a criterion for curriculum scheduling.The transfer-based criterion is intended to go beyond reliance solely on optimization heuristics.

3 Empirical Observation

Preliminary experiments across Sudoku, KodCode, and iGSM show that curriculum learning is not universally effective: the best schedule depends on the reasoning task and may change during training. These results motivate analyzing cross-difficulty optimization effects rather than schedule design alone.

  • Experimental Setup: Experiments cover Sudoku, KodCode, and iGSM, representing logical reasoning, code generation, and mathematical reasoning with naturally defined difficulty levels.Data are partitioned into four difficulty levels for Sudoku and KodCode, and five for iGSM.
  • Experimental Setup: The comparison includes Curriculum, Random, and Mix schedules, respectively training easy-to-hard, sampling uniformly, and combining curriculum with random samples.Mix replaces half of each stage’s samples with randomly sampled examples.
  • Final Performance: Curriculum performs best on Sudoku, Mix achieves the highest accuracy on iGSM, and Random performs best on KodCode.Thus, no single scheduling strategy consistently achieves the best performance across reasoning tasks.
  • Optimization Dynamics: Training dynamics differ across tasks: one strategy consistently dominates on Sudoku and KodCode, whereas the optimal strategy on iGSM changes as training progresses.Final performance alone does not capture these differing optimization behaviors.
  • Motivation: These findings motivate studying how optimization on one difficulty level influences optimization of others to identify the mechanism behind curriculum success.The paper therefore investigates the optimization process induced by curriculum learning.

4 Optimization Analysis of Curriculum Learning

This section analyzes curriculum learning through optimization dynamics induced by cross-difficulty updates. It defines Relative Transfer to quantify transfer strength and uses observed transfer patterns to explain why curriculum schedules differ across tasks.

  • Cross-Difficulty Transfer: Knowledge transfer measures how optimizing one difficulty level affects the loss of another difficulty level.The analysis treats the interaction between source difficulty j and target difficulty i as cross-difficulty knowledge transfer.
  • Optimization Dynamics: The loss effect of optimizing difficulty level j on difficulty level i is determined by the projection of their gradients.Positive gradient alignment reduces the target loss, whereas negative alignment creates optimization conflicts.
  • Relative Transfer: Relative Transfer compares loss reduction from optimizing difficulty level j with loss reduction from directly optimizing difficulty level i.It normalizes transfer by the target difficulty’s self-optimization effect, avoiding comparisons distorted by gradient magnitude.
  • Relative Transfer: Relative Transfer values of 1 indicate transfer equivalent to self-optimization, values between 0 and 1 indicate partial positive transfer, values above 1 indicate stronger transfer, and negative values indicate conflicts.These interpretations apply to transfer from difficulty level j to target difficulty level i.
  • Task-Dependent Transfer: Sudoku shows strong positive transfer from the hardest difficulty to earlier levels, whereas iGSM and KodCode show much weaker transfer.Strong transfer makes optimizing only the hardest examples sufficient for Sudoku, while weaker transfer motivates adapting the sampling strategy to observed transfer relationships.

5 Transfer-aware Dynamic Curriculum Sampling

Transfer-aware Dynamic Curriculum Sampling (TDCS) adapts training allocation to observed cross-difficulty transfer rather than using a fixed easy-to-hard schedule. It combines transfer-based current-difficulty allocation, replay of insufficiently transferred difficulties, and adjustments for exceptionally strong transfer to harder difficulties.

  • Design Principles: TDCS increases training on the current difficulty according to its transfer capability, while emphasizing replay for difficulties receiving insufficient transfer.These principles replace a predefined easy-to-hard schedule with allocation based on observed cross-difficulty transfer.
  • Current Difficulty Ratio Mapping: Difficulties with Relative Transfer below threshold τe are selected for additional replay, and their average transfer summarizes the current difficulty’s transfer capability to them.The selected set contains difficulty levels that receive insufficient transfer from the current difficulty.
  • Current Difficulty Ratio Mapping: A sigmoid mapping converts average transfer into the current-difficulty sampling ratio, with α controlling slope and β controlling midpoint.Stronger Relative Transfer yields more current-difficulty sampling, whereas weaker transfer encourages replay of previous difficulties.
  • Difficulty Allocation: The remaining probability, 1 −ρk, is allocated using reverse exponential weighting so weaker-transfer difficulties receive larger replay probabilities.This prioritizes difficulties less likely to benefit from the current optimization.
  • Harder Difficulty Adjustment: When transfer to a harder difficulty exceeds threshold τh, TDCS reallocates part of the current probability to that harder difficulty in proportion to its Relative Transfer, then normalizes all probabilities.This adjustment exploits exceptionally strong positive transfer to improve subsequent optimization.

6 Experiments

Experiments show that TDCS consistently improves reasoning performance over fixed scheduling strategies by adapting sampling to cross-difficulty transfer. Its advantages generalize across tasks, model scales, training paradigms, and both method components.

  • Main Results: Under matched LoRA fine-tuning budgets and hyperparameters, TDCS outperforms fixed scheduling by 2.6% on Sudoku, 3.8% on iGSM, and 1.2% on KodCode.Experiments use Qwen2.5-1.5B, four epochs, a learning rate of 1 × 10−5, and LoRA rank 64, with identical remaining hyperparameters.
  • Main Results: TDCS maintains superior training performance on Sudoku and iGSM and achieves the best final performance on KodCode despite a smaller intermediate-stage advantage.The training-dynamics comparison is against the strongest baseline.
  • Sampling Dynamics: TDCS adapts sampling differently by task: it shifts rapidly toward harder Sudoku difficulties but maintains moderate replay on iGSM according to estimated Relative Transfer.The sampling distributions reflect distinct cross-difficulty transfer characteristics rather than a predefined schedule.
  • Generalization Across Model Scales: TDCS consistently outperforms the strongest fixed scheduling strategy across Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Llama3.2-3B-Instruct.Improvements occur across both Qwen and Llama model families.
  • Generalization Across Training Paradigms: In self-improvement, TDCS outperforms all fixed scheduling strategies on GSM8K and KodCode when model answer accuracy defines difficulty.The curriculum is constructed from the model’s own generated responses rather than manually curated data.
  • Ablation Study: Removing either Harder Adjustment or Exponential Difficulty Allocation degrades performance across all benchmarks, with Harder Adjustment especially important on Sudoku and transfer-aware replay allocation outperforming uniform allocation.Harder Adjustment is particularly beneficial where harder-to-easier transfer is stronger, while Exponential Difficulty Allocation assigns replay using estimated transfer relationships.

7 Conclusion

The paper explains curriculum learning through optimization dynamics and cross-difficulty knowledge transfer. It introduces Relative Transfer and derives TDCS to adjust sampling according to estimated transfer relationships.

  • 7 Conclusion: Relative Transfer characterizes cross-difficulty knowledge transfer and explains when curriculum learning succeeds.It is presented as a principled measure grounded in the optimization dynamics induced by different curriculum schedules.
  • 7 Conclusion: TDCS dynamically adjusts the sampling distribution according to the estimated transfer relationship.The method is derived from the paper’s analysis of curriculum-induced optimization dynamics.

Appendix · 8 Dataset Details

The appendix documents the datasets used in the experiments through benchmark statistics, difficulty criteria, and representative input-output examples. It covers dataset-level information alongside examples from iGSM and KodCode.

  • Appendix: The appendix provides additional dataset details, including statistics, difficulty definitions, and representative examples.
  • 8.1 Dataset Statistics: Table 7 reports training and evaluation sample counts, difficulty-level counts, and difficulty criteria for each benchmark.
  • 8.2 Dataset Examples: Representative examples from three benchmarks pair each input with its expected output and corresponding difficulty level.
  • 8.4 iGSM Example: The iGSM example asks for NetVersity’s Microbiology Lab count using relationships among four laboratory quantities.
  • 8.4 iGSM Example: The iGSM example’s provided output defines G = 4, computes S = 19 + 4 = 0, then computes P = 4 * 0 = 0.
  • 8.5 KodCode Example: The KodCode example implements list permutation generation with a backtracking procedure and tests outputs for empty, singleton, repeated, and three-element lists.

9 Training Details

The experiments use fixed GPU configurations and a low-cost dynamic sampling procedure that updates difficulty-level weights every 25 steps. Sensitivity analysis shows that τe and τh affect replay and harder-example allocation differently, with τe = 0.5 and τh = 0.85 selected for the main experiments.

  • The 7B model is fine-tuned on 8 NVIDIA RTX 5090 GPUs, while smaller models use 8 NVIDIA RTX 4090 GPUs.
  • Dynamic sampling reweighs every difficulty level every 25 steps using 8 examples per level to estimate the Re matrix.This design is intended to preserve evaluation accuracy while adding very little computation.
  • Hyperparameter sensitivity: τe mainly affects iGSM and Sudoku by changing which easier problems are replayed, whereas Code performance remains relatively stable.The effect is especially apparent on Sudoku, where replaying easier examples is more beneficial.
  • Hyperparameter sensitivity: τh more strongly influences Sudoku by changing when harder difficulty levels receive additional sampling probability and altering the easy-hard example balance.This reflects the more pronounced transfer relationship between easier and harder Sudoku levels.
  • Hyperparameter sensitivity: The final method uses τe = 0.5 and τh = 0.85, which achieve consistently strong performance across all three benchmarks.These values are used throughout the main experiments to balance easier-problem replay with additional probability for harder problems.
Loading 2608.17268v1…