Self-Evolving Curriculum for LLM Reasoning

Xiaoyin Chen; Jiarui Lu; Minsu Kim; Dinghuai Zhang; Jian Tang; Alexandre Piché; Nicolas Gontier; Yoshua Bengio; Ehsan Kamalloo

Source-linked AI summary

Self-Evolving Curriculum for LLM Reasoning

Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, Ehsan Kamalloo

arXiv:2505.14970v4May 20, 2025cs.AIcs.LG

TL;DR

RL fine-tuning benefits from effective curriculum ordering, but random curricula are imperfect and alternatives can rely on heuristics or costly filtering. SEC jointly learns curriculum selection with RL by treating problem categories as non-stationary bandit arms and using absolute advantage as a learning-gain proxy. Across planning, inductive reasoning, and mathematics, SEC improves harder out-of-distribution performance and balances skills across multiple domains, while requiring additional hyperparameter tuning.

Problem
RL fine-tuning depends on curriculum ordering, while random curricula remain imperfect and manually designed or online-filtering approaches have practical limitations.
Method
SEC learns a curriculum policy during RL fine-tuning by treating problem categories as non-stationary Multi-Armed Bandit arms and rewarding categories with absolute advantage.
Results
SEC improves reasoning and generalization across planning, inductive reasoning, and mathematics, with reported gains including 13% on Countdown, 21% on Zebra, and up to 33% on AIME24.
Takeaways & Limitations
SEC provides an automatic curriculum strategy that improves generalization and balances learning across multiple reasoning domains simultaneously.
Takeaways & Limitations
SEC introduces extra hyperparameters, including temperature and learning rate, that require tuning.

Abstract

from arXiv · show

Reinforcement learning (RL) has proven effective for fine-tuning large language models (LLMs), significantly enhancing their reasoning abilities in domains such as mathematics and code generation. A crucial factor influencing RL fine-tuning success is the training curriculum: the order in which training problems are presented. While random curricula serve as common baselines, they remain suboptimal; manually designed curricula often rely heavily on heuristics, and online filtering methods can be computationally prohibitive. To address these limitations, we propose Self-Evolving Curriculum (SEC), an automatic curriculum learning method that learns a curriculum policy concurrently with the RL fine-tuning process. Our approach formulates curriculum selection as a non-stationary Multi-Armed Bandit problem, treating each problem category (e.g., difficulty level or problem type) as an individual arm. We leverage the absolute advantage from policy gradient methods as a proxy measure for immediate learning gain. At each training step, the curriculum policy selects categories to maximize this reward signal and is updated using the TD(0) method. Across three distinct reasoning domains: planning, inductive reasoning, and mathematics, our experiments demonstrate that SEC significantly improves models' reasoning capabilities, enabling better generalization to harder, out-of-distribution test problems. Additionally, our approach achieves better skill balance when fine-tuning simultaneously on multiple reasoning domains. These findings highlight SEC as a promising strategy for RL fine-tuning of LLMs.

1 INTRODUCTION

RL fine-tuning depends critically on curriculum design, yet random, manually designed, and online-filtering strategies have important limitations. SEC learns curriculum selection during RL fine-tuning and improves reasoning and out-of-distribution generalization across planning, inductive reasoning, and mathematics.

Motivation: Curriculum order critically influences RL fine-tuning effectiveness and should align with the model’s evolving learning progress.The paper treats curriculum as the order in which training data is presented and links alignment with current progress to learning outcomes.
Motivation: A reverse difficulty curriculum produces poor test performance and minimal generalization, whereas random sampling improves both outcomes.The controlled Countdown experiment orders problems from hard to easy for the reverse curriculum and contrasts it with uniformly random sampling.
Approach: SEC automatically learns a curriculum policy concurrently with RL fine-tuning by modeling categories such as difficulty or problem type as arms in a non-stationary Multi-Armed Bandit.The method uses absolute advantage as the arm reward and updates the curriculum policy with TD(0).
Results: SEC significantly improves reasoning across planning, inductive reasoning, and mathematics, especially on harder out-of-distribution problems.Reported relative gains include 13% on Countdown, 21% on Zebra, 22% on ARC-1D, and up to 33% on AIME24.

2 METHOD

SEC frames curriculum selection as a non-stationary bandit over problem categories and updates category values using locally measurable learning gains. It samples categories through a Boltzmann policy, trains the LLM on the resulting batch, and updates rewards from absolute advantages.

Curriculum selection: Training problems are partitioned into categories, and curriculum selection learns an expected return for each category as the LLM changes.Categories may represent domains, problem types, or in-domain difficulty levels; selected categories are followed by uniform within-category problem sampling.
Learning objective: The curriculum policy uses a locally measurable proxy because final task performance requires completing the full RL fine-tuning process.The proxy is intended to avoid the computational cost of frequent intermediate evaluations.
Learning objective: Immediate learning outcomes are approximated through gradient magnitude, whose policy-gradient weighting depends on the absolute advantage.The paper motivates absolute advantage as a proxy for learning gain from selected training data.
Reward: SEC defines each category’s reward as the batch-wise expectation of absolute advantage across associated rollouts.The reward is computed as r(c) = E|A_t| for problems drawn from category c.
Sampling and updates: Categories are sampled from a Boltzmann distribution over current category values, balancing exploration and exploitation through temperature.Problems are then sampled uniformly from selected categories until the desired batch size is reached.
Sampling and updates: After the LLM policy update, sampled-category rewards update their Q-values, refining future curriculum selection.The complete procedure performs category sampling, LLM updating, reward computation, and Q-value updating at each RL step.

3 EXPERIMENTS

SEC is evaluated across planning, inductive reasoning, and mathematics using adaptive curriculum selection during RL fine-tuning. It improves challenging OOD generalization, adapts difficulty to model progress, and balances multiple reasoning tasks.

Method: SEC selects curriculum categories through a non-stationary bandit procedure while RL fine-tuning updates the language model.Categories are treated as arms; batches sample categories according to the curriculum policy and problems uniformly within categories, followed by rollout, advantage estimation, and Q-value updates.
Main Results: SEC improves Qwen2.5-3B OOD accuracy by approximately 13% on Countdown, 21% on Zebra, and 33% on AIME24 versus random curricula.The reported comparisons are Countdown 0.48 → 0.54, Zebra 0.29 → 0.35, and AIME24 0.075 → 0.10.
Curriculum Analysis: SEC initially samples easier problems and progressively increases difficulty, with the stronger Qwen2.5-7B receiving harder problems than Qwen2.5-3B.This pattern is reported across tasks and models as alignment between selected difficulty and improving model capability.
Multiple Curriculum Categories: SEC-2D balances simultaneous multi-task training more reliably than random curricula, whose Countdown OOD performance collapses midway through training.SEC-2D uses one arm for each combination of three problem types and three difficulty levels, yielding nine arms; it consistently outperforms random across the three reasoning tasks.
Robustness and Category Design: SEC also improves Countdown ID and OOD performance under PPO and RLOO, and remains effective with curriculum categories inferred from empirical success rates.These results extend SEC beyond GRPO and reduce reliance on manually assigned curriculum labels.

4 RELATED WORKS

Prior work applies RL fine-tuning and curriculum learning to language-model reasoning, while recent methods increasingly adapt curricula automatically. SEC differs by framing category selection as a dynamic bandit problem rather than relying on filtering or predefined difficulty orderings.

RL fine-tuning: RL fine-tuning has improved language-model alignment, mathematical reasoning, and self-training through outcome or process rewards.These approaches model language models as sequential decision-making policies and use rewards during or after generation.
RL algorithms: Policy-gradient, PPO, off-policy, and value-based algorithms provide the broader RL algorithmic context for these language-model fine-tuning methods.The related work identifies REINFORCE variants and PPO as widely adopted for their relative stability, alongside DPO and other alternatives.
Curriculum learning: Curriculum learning organizes training examples or tasks to address optimization, sparse-reward, and exploration challenges in supervised and reinforcement learning.Examples include easy-to-hard sequencing, reverse-curriculum generation, and Teacher-Student Curriculum Learning.
Curriculum learning for LLMs: Recent language-model curricula include reverse curricula, self-evolving task generation, online problem filtering, and adaptive difficulty adjustment.These methods differ in whether they reveal longer reasoning sequences, generate tasks from failures, estimate difficulty online, or use ordered difficulty levels.
SEC: SEC uses a general non-stationary Multi-Armed Bandit formulation to dynamically adjust curriculum categories during RL fine-tuning.Unlike online filtering and methods requiring explicit difficulty ordering, SEC dynamically adjusts the curriculum through category-level arms.

5 CONCLUSION

The paper presents SEC as an automatic curriculum framework that adapts category selection to the model’s changing capabilities during RL fine-tuning. Experiments across reasoning tasks report improved generalization and balanced learning across domains, while the method requires additional hyperparameter tuning.

Contribution: SEC formulates adaptive curriculum selection as a non-stationary Multi-Armed Bandit problem that dynamically adjusts problem difficulty as the model evolves.The framework’s components include curriculum rewards, sampling methods, and update rules; the reported implementation uses absolute advantage, Boltzmann sampling, and TD(0).
Findings: Experiments across planning, inductive reasoning, and mathematics show consistent improvements in generalization and performance balance across multiple reasoning domains.The conclusion states that SEC improves generalization while balancing learning across tasks trained simultaneously.
Limitations: SEC introduces extra hyperparameters, including temperature and learning rate, that require tuning.The paper identifies this tuning requirement as a limitation and suggests more flexible curriculum definitions for future work.

A INTERPRETING ABSOLUTE ADVANTAGE IN RL WITH VERIFIABLE REWARDS.

The analysis explains why absolute advantage is a useful curriculum reward under binary verifiable rewards. Its expected magnitude is highest for problems with a 0.5 success rate, motivating selection of neither-too-easy-nor-too-hard problems, while experiments also test broader settings.

Motivation: Under binary correctness rewards and GRPO, expected absolute advantage is maximized when the expected reward, or success rate, equals 0.5.This identifies intermediate-difficulty problems as producing the highest expected learning gain under the analyzed setting.
Advantage estimation: GRPO estimates each rollout’s advantage from its binary reward relative to the mean and standard deviation across n rollouts from the same problem.The reward is modeled as Bernoulli, with mean p and standard deviation sqrt(p(1 − p)).
Derivation: For Bernoulli rewards, the expected absolute advantage depends on p(1 −p), which is symmetric and strictly concave on [0, 1].The derivation considers the two possible binary reward outcomes and their probabilities.
Implication: Maximizing expected absolute advantage is therefore equivalent to prioritizing problems with a success rate of 0.5.This connects the curriculum objective to selecting problems at an intermediate difficulty for the current model.
Scope: The derivation analyzes a common binary-reward setting but does not restrict SEC to binary rewards or a single RL algorithm.The paper reports strong SEC performance with non-binary rewards and alternative RL algorithms.

B SEC ACROSS BASE MODEL FAMILIES

SEC improves over a random curriculum on both Countdown evaluation splits when applied to a Llama-3.2-1B variant. This result supports generalization across model families and scales.

Cross-family evaluation: SEC improves over the random curriculum on both Countdown splits for a Llama-3.2-1B variant.The model includes special mid-training intended to improve RL fine-tuning performance.

C IMPLEMENTATION DETAILS

The experiments use GRPO-based fine-tuning with Qwen2.5 models, specified training schedules, and task-specific hyperparameters and datasets. Additional settings cover multi-task and alternative-algorithm experiments.

GRPO fine-tunes the models through the Volcano Engine verl library implementation.
Qwen2.5-3B and Qwen2.5-7B are trained with task-specific gradient-step schedules and batches of 256.Advantages are estimated from 8 rollouts, and the study omits the KL divergence loss.
The multi-task experiment fine-tunes Qwen2.5-3B for 720 steps on a mixed dataset using α = 0.5 and τ = 0.2.
Alternative RLOO and PPO experiments use 120 training steps with separately specified SEC and GAE hyperparameters.These experiments also omit the KL divergence loss.
The mathematics experiments use MATH500, AMC22-23, and AIME datasets.

D DATA EXAMPLES

The data examples present prompt formats for mathematics, Countdown, Zebra Puzzle, and ARC-1D tasks. They illustrate arithmetic, constraint-solving, and grid-transformation problems requiring final answers in boxed form.

Countdown: Countdown prompts require using each provided number exactly once with basic arithmetic operations to reach a target.The example uses [5, 17, 91] and targets 113.
Zebra Puzzle: Zebra Puzzle prompts define entities with distinct names, cigars, and animals, then provide clues constraining their assignments.The example asks for the person living in House 1.
ARC-1D: ARC-1D prompts provide input-output examples and ask the model to infer a common rule for transforming a test grid.The prompt requests a detailed derivation before returning only the test output grid in boxed form.
Mathematics: Mathematics prompts ask the assistant to solve a question step by step and place the final answer within boxed notation.The example asks for a remainder when an expression is divided by 17.