Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning

Shubham Parashar; Shurui Gui; Xiner Li; Hongyi Ling; Sushil Vemuri; Blake Olson; Eric Li; Yu Zhang; James Caverlee; Dileep Kalathil; Shuiwang Ji

Source-linked AI summary

Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning

Shubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling, Sushil Vemuri, Blake Olson, Eric Li, Yu Zhang, James Caverlee, Dileep Kalathil, Shuiwang Ji

arXiv:2506.06632v3Jun 7, 2025cs.LGcs.AIcs.CL

TL;DR

LLM reinforcement learning improves reasoning but is less effective on inherently difficult tasks with sparse rewards. E2H Reasoner progressively schedules decomposed tasks from easy to hard, and the paper reports improved reasoning, convergence guarantees, and sample-efficiency benefits, while noting scheduler limitations.

Problem
Reinforcement learning improves reasoning but struggles on inherently difficult tasks where models have low zero-shot performance and rewards are sparse.
Method
E2H Reasoner uses curriculum-based reinforcement learning with task decomposition and scheduling that progressively shifts training from easier to harder tasks.
Results
E2H enables models to learn initially failed tasks and reports strong performance across five reasoning tasks, with convergence guarantees and improved sample efficiency over direct RL.
Takeaways & Limitations
E2H provides a scalable, theoretically grounded, and practical approach for improving reasoning in small language models.
Takeaways & Limitations
Balanced scheduling can introduce hard tasks too early, producing sparser rewards and suboptimal curriculum reinforcement learning.

Abstract

from arXiv · show

We aim to improve the reasoning capabilities of language models via reinforcement learning (RL). Recent RL post-trained models like DeepSeek-R1 have demonstrated reasoning abilities on mathematical and coding tasks. However, prior studies suggest that using RL alone to improve reasoning on inherently difficult tasks is less effective. Here, we draw inspiration from curriculum learning and propose to schedule tasks from easy to hard (E2H), allowing LLMs to build reasoning skills gradually. Our method is termed E2H Reasoner. Empirically, we observe that, although easy tasks are important initially, fading them out through appropriate scheduling is essential in preventing overfitting. Theoretically, we establish convergence guarantees for E2H Reasoner within an approximate policy iteration framework. We derive finite-sample complexity bounds and show that when tasks are appropriately decomposed and conditioned, learning through curriculum stages requires fewer total samples than direct learning. Experiments across multiple domains show that E2H Reasoner significantly improves the reasoning ability of small LLMs (1.5B to 3B), which otherwise struggle when trained with vanilla RL alone, highlighting the effectiveness of our method. Our code can be found on https://github.com/divelab/E2H-Reasoning.

1 INTRODUCTION

RL improves LLM reasoning but struggles on difficult tasks with sparse rewards. E2H Reasoner addresses this by progressively scheduling decomposed tasks from easy to hard, with empirical and theoretical support.

Motivation: RL-based post-training struggles on inherently difficult tasks because correct-answer rewards provide sparse learning signals.This limitation is especially relevant when pretrained models have low zero-shot performance.
Method: A probabilistic scheduler gradually shifts training focus from easy to hard tasks instead of switching difficulty after a fixed number of iterations.This scheduling strategy is intended to support learning and generalization on complex problems.
Empirical results: E2H achieves state-of-the-art performance across five reasoning tasks, including Blocksworld, Countdown, and three arithmetic benchmarks.The reported task set spans planning, arithmetic, and related reasoning settings.
Method: E2H Reasoner decomposes training data into trivial, easy, and medium tasks, then schedules harder tasks as training progresses.The curriculum is designed to help models acquire core skills before tackling more difficult problems.
Theory: Theoretical analysis establishes convergence guarantees and finite-sample complexity bounds for curriculum-based reinforcement learning.The analysis uses an Approximate Policy Iteration framework and shows that a well-designed curriculum can require fewer total samples than direct learning.

2 BACKGROUND AND RELATED WORK

Prior work studies reasoning, supervised and reinforcement-learning post-training, and curriculum learning, but RL alone remains insufficient for difficult zero-shot tasks. Curriculum-based approaches provide a route toward more effective learning and generalization.

Reasoning definitions: LLM reasoning has been described as chain-of-thought generation, multi-step deduction, human-like problem solving, or generalization and abstraction.These perspectives differ in how they distinguish reasoning from pattern recognition.
LLM post-training: Supervised fine-tuning imitates curated reasoning examples but can overfit to surface-level patterns, limiting generalization.This contrasts with reinforcement learning, which updates policies using task-specific rewards.
LLM post-training: Reinforcement-learning post-training has greater potential for reasoning improvement than imitation, yet remains insufficient on inherently difficult zero-shot tasks.The cited literature motivates methods that provide more effective learning signals for hard problems.
Curriculum learning: Curriculum learning organizes tasks by increasing difficulty so agents can master simpler behaviors before acquiring complex ones.Recent work applies curriculum-based reinforcement learning to reasoning and generalization in LLMs.

3 METHOD

E2H Reasoner decomposes difficult reasoning tasks into progressively harder curriculum stages and uses adaptive scheduling to address sparse rewards, task forgetting, and easy-task overfitting. Its API-based analysis provides convergence and finite-sample guarantees, including fewer total samples than direct learning under suitable curriculum conditions.

Motivation: RL struggles on difficult tasks because low zero-shot accuracy and correctness-only rewards create large distribution gaps and sparse learning signals.The paper identifies distribution shift between pre-training and target-task data as a source of low accuracy and sparse rewards.
Task Decomposition: Task decomposition partitions training data into increasing difficulty levels, reducing distribution shift and breaking complex skill acquisition into simpler steps.The proposed curriculum uses trivial, easy, medium, and hard stages, with intermediate distributions bridging the source and target tasks.
Scheduling Challenges: Traditional sequential scheduling can cause task forgetting, while balanced scheduling avoids forgetting but exposes the model to hard tasks too early and produces sparser rewards.Fixed task order can also overfit the model to easier tasks; equal-probability sampling trades forgetting for premature exposure to difficult examples.
Theoretical Setup: Approximate Policy Iteration alternates approximate policy evaluation and policy improvement across interpolated curriculum distributions to analyze sequential adaptation toward the final policy.The framework accommodates function approximation and inexact updates, making it suitable for analyzing practical reinforcement-learning algorithms.
Convergence Guarantee: The convergence guarantee decomposes the final performance gap into actor convergence bias, evaluation and sampling errors, policy-update error, and curriculum approximation error.Actor bias decreases geometrically with more updates; update error can be reduced with sufficiently large or geometrically increasing stepsizes.
Finite-Sample Analysis: Under appropriately designed curricula, easier tasks improve estimation accuracy and stable policy updates, yielding finite-sample learning with fewer total samples than direct learning.The result depends on intermediate distributions remaining close to the target and on progressively optimized accuracy allocation across curriculum stages.

4 EXPERIMENTS

Experiments evaluate E2H Reasoner across reasoning and planning benchmarks, task decompositions, schedulers, and baselines. Results indicate that gradual easy-to-hard scheduling improves transfer and generalization, especially for harder and OOD tasks, while appropriate decay of easy-task exposure helps prevent overfitting.

Evaluation setup: Experiments cover Blocksworld, Countdown, MATH, GSM8K, AQuA, AIME24, and OlympiadBench using small instruction-tuned LLMs.The study includes Qwen 2.5 1.5B and Llama 3.2 3B models, with additional models reported in the appendix.
Task decomposition: Including trivial and easy examples helps models build core skills that transfer to harder tasks and improve OOD performance.This finding is reported for Qwen-1.5B-Instruct under a balanced scheduler.
Task scheduling: E2H-C and E2H-G address scheduling weaknesses by shifting training from easy toward hard tasks rather than using fixed-order or balanced schedules.E2H-C uses cosine scheduling, while E2H-G rapidly decays trivial and easy-task sampling in sparse-reward settings.
Hard-task learning: Direct training on difficult examples largely fails, with Qwen-2.5 1.5B trained on Level 5 MATH underperforming its CoT baseline.The result motivates curriculum-based reinforcement learning for small-model reasoning.
Generalization: E2H improves performance as task difficulty rises, strengthens OOD generalization, and remains effective when error rates replace human difficulty labels.The method also reports gains over baselines and does not require extensive hyper-parameter tuning for E2H-G.
Ablations and extensions: Combining E2H with DAPO reduces zero-advantage batches and yields the strongest performance across benchmarks, indicating complementary sampling effects.E2H shapes the difficulty-conditioned distribution from which DAPO resamples.

5 CONCLUSION

The paper concludes that E2H Reasoner is a curriculum-based post-training method that schedules tasks from easy to hard for LLM reasoning. Its theoretical analysis uses approximate policy iteration to provide convergence and finite-sample guarantees, including improved sample efficiency under suitable task decomposition and conditioning.

Conclusion: E2H Reasoner schedules tasks from easy to hard so models can learn tasks they initially fail to solve.The method is presented as curriculum-based reinforcement learning for LLM post-training.
Theory: The theoretical analysis studies curriculum reinforcement learning through approximate policy iteration.The proof development follows prior approximate-policy-iteration analyses and extends them to the curriculum setting.
Conclusion: Under suitable task decomposition and conditioning, curriculum learning requires fewer total samples than direct learning.This theoretical result is presented as consistent with the empirical observations.
Proof assumptions: When later curriculum stages subsume earlier ones, the analysis uses C_k ≤ 1 to simplify distribution-norm bounds without loosening them unnecessarily.Here, the d_K norm is bounded by the sup norm, and the curriculum support is assumed to be the same or growing across stages.

A.1.1 THEOREM A.1 AND PROOF

Theorem A.1 analyzes curriculum reinforcement learning through approximate policy iteration, providing convergence and finite-sample guarantees under stage-wise assumptions. The proof decomposes the final performance gap into actor, critic, approximation, importance-sampling, and curriculum-discrepancy terms, then derives conditions under which curriculum learning is more sample-efficient than direct learning.

Theorem and proof: Theorem A.1 bounds the final policy’s performance gap after K curriculum stages under assumptions on all stages.The proof proceeds by induction, initializing stage k + 1 with the final policy from stage k.
Theorem and proof: The error decomposition includes actor-update error, function-approximation error, importance-sampling bias, critic convergence bias, and curriculum discrepancy.Function-approximation error vanishes with a complete basis, exact importance-sampling factors remove the associated bias, and critic bias decreases geometrically with inner-loop iterations.
Theorem and proof: Smoother curriculum trajectories and smaller finite-sample approximation errors can improve final policy performance.The relevant approximation errors depend on the stepsize and the number of updates per iteration.
Sample complexity: The proof controls the total error by allocating an error budget across curriculum stages and deriving per-stage and aggregate sample-complexity expressions.The analysis considers uniform error allocation and incorporates dependence on the bootstrapping parameter.
Sample complexity: A curriculum efficiency factor compares total curriculum sample complexity with the sample complexity of learning only the final task.Under geometric error allocation and progression of task parameters, CRL is more sample-efficient than direct learning when the derived efficiency condition holds.

B SAMPLE EFFICIENCY GAINS WITH CRL

The experiments compare curriculum and non-curriculum reinforcement-learning methods under a fixed training budget and count samples by difficulty level. Consistent with the theory, curriculum methods achieve strong performance while using substantially fewer hard-task samples than non-curriculum baselines.

Experimental setup: 12,800 samples are used for every method, enabling direct sample-efficiency comparisons under the same training budget.The budget comes from 1,600 iterations with an effective batch size of 8.
Sample efficiency: E2H-G uses 3,580 hard samples versus 12,800 for GRPO (HARD) while attaining strong performance.Table 8 reports this comparison as consistent with the theoretical guarantees.
Sample efficiency: Training exclusively on OOD tasks with GRPO-OOD uses 12,800 OOD samples and performs poorly.This result is reported in comparison with the curriculum methods in Table 8.

C.1 DATASETS WITH HUMAN ANNOTATED DIFFICULTIES

The evaluation uses datasets with human-annotated or task-structured difficulty levels, organizing examples into progressively harder splits. These splits are based on plan length, operand count, or problem level, while implementation uses several small instruction-tuned LLMs and GRPO with LoRA.

Difficulty construction: Blocksworld difficulty is determined by plan length, with longer action sequences requiring more complex planning.The dataset evaluates transitions between initial and target block configurations.
Difficulty construction: Countdown difficulty is increased by the number of input operands, using Trivial (2), Easy (3), Medium (4), Hard (5), and OOD (6) splits.More inputs expand the space of possible operation sequences.
Difficulty construction: MATH uses existing problem levels to define Trivial through OOD splits, mapping Levels 1 through 5 from easiest to hardest.The benchmark contains high-school mathematics problems spanning algebra, geometry, number theory, and probability.
Difficulty construction: GSM8K evaluates multi-step arithmetic reasoning, with difficulty splits based on model error rates rather than explicit annotations.AQuA is similarly divided into Trivial, Easy, Medium, and Hard groups using model error rates, without an OOD category.
Models and training: Experiments use LLaMA 3.2 3B Instruct, Qwen 2.5 1.5B Instruct, and Qwen 2.5 3B Instruct.Training uses GRPO with LoRA and a reward design that gives partial format rewards and full rewards for correct format and answers.

E.3 DIFFICULTY-BASED TRAINING SPLIT CREATION FOR GSM8K AND AQUA

For GSM8K and AQuA, difficulty levels are constructed from repeated model-error estimates because neither dataset provides explicit difficulty annotations. The experiments use deterministic inference for reporting, while comparisons include supervised fine-tuning and reinforcement-learning post-training methods.

Difficulty split creation: Difficulty is estimated by querying a prompted model 20 times per question and computing the fraction of incorrect answers.The setup uses chain-of-thought prompting with one-shot in-context learning and assigns samples to difficulty groups from the resulting errors.
Difficulty split creation: GSM8K and AQuA use model-error-based difficulty splits and exclude an OOD category.The absence of explicit difficulty annotations motivates this construction.
Evaluation: Inference uses temperature 0.0 for deterministic and reproducible outputs, while pass@k uses temperature 0.7, top_p 0.9, and top_k 50.The sampling settings apply specifically to pass@k evaluations.
Baselines: SFT performance varies significantly across tasks, performing well on Blocksworld when training and test distributions have a small gap.The comparison includes vanilla SFT and RL-based post-training methods.
Baselines: Additional results on more LLMs are reported in the subsection.

G.2 RESULTS ACROSS GAUSSIAN PARAMETERS

The appendix reports Gaussian scheduling variants and robustness analyses for E2H Reasoner. Across these analyses, the method remains robust to difficulty-split choices, while combining E2H with DAPO produces more informative training batches.

Gaussian scheduling results: Gaussian scheduling variants are reported for Qwen 2.5 1.5B Instruct on GSM8K and AQuA, alongside balanced and cosine-scheduling comparisons.The appendix includes all three Gaussian parameter settings and expanded variant results.
Comparison results: The appendix provides comparison tables for RL-based post-training and SFT, including Qwen-2.5 3B Instruct results.The SFT comparison notes noticeable performance variation across tasks.
Difficulty-split robustness: The number of difficulty splits is treated as a hyperparameter for datasets without human-annotated difficulty labels, including GSM8K and AQuA.The method uses four splits, consistent with the 3–5 range in curriculum-learning literature.
Difficulty-split robustness: E2H Reasoner remains robust across different choices of the difficulty-split hyperparameter.This robustness is reported in Table 15.
Batch informativeness: Combining E2H with DAPO consistently lowers the fraction of advantage-zero batches.The authors interpret this as selecting more informative batches whose difficulty better matches model competence.

G.5 QUALITATIVE ANALYSIS OF SCHEDULING TECHNIQUES

The qualitative analysis compares scheduling strategies and emphasizes that their strengths depend on the task and model scale. E2H variants are evaluated against balanced scheduling, traditional curriculum learning, and chain-of-thought reference results.

Qualitative scheduler comparison: Different scheduling techniques are suited to different tasks and model scales.The qualitative comparison highlights distinct strengths and weaknesses across schedulers.
Future directions: The analysis motivates exploring more sophisticated scheduling strategies for LLM post-training.This direction is presented as a future research motivation based on E2H Reasoner’s effectiveness.
Qualitative scheduler comparison: Table 13 compares balanced scheduling, traditional curriculum learning, E2H-G, and E2H-C, with CoT as a reference.E2H-G denotes Gaussian scheduling and E2H-C denotes cosine scheduling.
Expanded Gaussian results: Table 14 expands Qwen 2.5 1.5B Instruct results across all E2H-G variants on GSM8K and AQuA.The comparisons include Balanced and E2H-C baselines.
Batch-level analysis: Figure 7 tracks the fraction of advantage-zero batches over training steps for DAPO, DAPO+E2H-G, and DAPO+E2H-C.The figure provides a training-step comparison of batch informativeness across these methods.

I LIMITATIONS

E2H Reasoner’s schedulers are effective but use fixed Gaussian or cosine functions rather than adapting during training. The paper identifies adaptive scheduling as a promising direction while cautioning that maximizing learnability may not always improve reasoning.

Scheduler limitations: The schedulers are simple Gaussian and cosine functions that do not adapt during training.The authors describe these choices as effective but limited.
Future directions: Advantage-based scheduling could offer further improvements over the fixed scheduler choices.The paper presents adaptive strategies as a future improvement direction.
Adaptive-curriculum caveat: Maximizing learnability does not always lead to stronger reasoning, depending on dataset problem structure and difficulty.This insight comes from comparisons with adaptive curriculum methods.
Future directions: Combining E2H Reasoner with adaptive approaches is identified as a promising direction for future work.The proposed combination is not presented as an evaluated solution in this passage.