PaperlayerView exact revision on arXiv

Source-linked AI summary

VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models

Guochao Jiang, Wenfeng Feng, Guofeng Quan, Chuzhan Hao, Yuewei Zhang, Guohua Liu, Hao Wang

arXiv:2509.19803v1cs.LGcs.CL

TL;DR

Existing rollout-based RL methods do not explicitly match training-sample difficulty to changing LLM abilities. VCRL uses rollout reward variance for dynamic curriculum sampling and replay learning, achieving state-of-the-art results across five mathematical benchmarks and two models.

  • Problem

    Existing rollout-based reinforcement learning methods do not explicitly match sample difficulty to the model’s changing abilities during training.

  • Method

    VCRL dynamically selects moderately difficult samples using group-reward variance and uses replay learning with a memory bank to retain high-value samples.

  • Results

    VCRL achieves state-of-the-art performance across five mathematical benchmarks on both Qwen3-4B-Base and Qwen3-8B-Base, outperforming all listed baselines.

  • Takeaways & Limitations

    Variance-based curriculum sampling and replay learning provide a way to control training-sample difficulty while improving reinforcement-learning efficiency and stability.

  • Takeaways & Limitations

    PPO-based comparisons involve value-model costs and accuracy limitations, especially for long responses and sparse rewards in complex LLM tasks.

Abstract

from arXiv · show

Policy-based reinforcement learning currently plays an important role in improving LLMs on mathematical reasoning tasks. However, existing rollout-based reinforcement learning methods (GRPO, DAPO, GSPO, etc.) fail to explicitly consider LLMs' learning ability for samples of different difficulty levels, which is contrary to the human cognitive process of mathematical reasoning tasks from easy to difficult. Intuitively, we find that the variance of the rollout group's reward in RLVR partly reflects the difficulty of the current sample for LLMs. Samples that are too easy or too difficult have a lower variance, while samples with moderate difficulty have a higher variance. Based on this, we propose VCRL, a curriculum reinforcement learning framework that dynamically controls the difficulty of training samples based on the variance of group rewards. Experiments on five mathematical benchmarks and two models reveal the advantages of VCRL over the current LLM RL baselines.

1 INTRODUCTION

Existing rollout-based RL methods do not adapt training-sample difficulty to changing model abilities. VCRL uses reward variance to select moderately difficult samples and replay high-value samples, with consistent gains across five benchmarks and two models.

  • RLVR enables test-time scaling through model exploration and verifiable rewards, supporting reinforcement-learning improvements in LLM reasoning.The introduction situates rollout-based RL within broader progress from long chain-of-thought reasoning.
  • Existing rollout-based RL methods do not match sample difficulty to changing model abilities, unlike curriculum learning that progresses from easier to harder tasks.Because model skills change during training, pre-sorting samples by fixed difficulty is ineffective.
  • VCRL dynamically adjusts training difficulty using rollout-group reward variance, prioritizing samples with moderate difficulty and high learning value.Very easy and very difficult samples tend to have low variance, while uncertain samples have higher variance.
  • Replay Learning maintains high-variance samples in a memory bank to improve training stability and efficiency.The memory bank helps keep valuable samples in the training batch as training proceeds.

2 PRELIMINARIES

The preliminaries describe PPO and rollout-based policy-optimization methods for LLMs. GRPO removes PPO’s value model through group-relative advantages, while DAPO and GSPO modify clipping, sampling, or importance-ratio granularity.

  • PPO constrains policy updates to a proximal region of the old policy using clipping and token-level importance ratios.Its advantage estimates use a value model and generalized advantage estimation.
  • PPO’s value model adds computational and memory costs, and its accuracy can limit performance for long responses with sparse rewards.The limitation is especially relevant to complex LLM tasks.
  • GRPO eliminates PPO’s value model by estimating each response’s relative advantage within a group generated for the same query.The group contains G responses, and advantages are computed relative to that group.
  • DAPO extends GRPO with removed KL regularization, clip-higher, and dynamic sampling using a token-level loss.Its clipping uses separate lower and upper ranges for the importance ratio.
  • GSPO replaces token-level importance ratios with sequence-level ratios to better match sentence-level rewards.The method retains group-based advantage estimation.

3 VARIANCE-BASED CURRICULUM REINFORCEMENT LEARNING

VCRL dynamically matches training-sample difficulty to the model by selecting queries with high rollout-reward variance and replaying valuable samples from a memory bank. It then applies the standard RL update to the resulting batch.

  • Variance-based curriculum reinforcement learning: VCRL calculates each query’s normalized group-reward variance, filters low-variance queries, replenishes the batch from a memory bank, and performs the standard RL update.The procedure uses a predefined threshold κ to retain queries with p ≥ κ and replace removed queries from the memory bank.
  • Variance-based dynamic sampling: Dynamic difficulty tracking is necessary because model parameters change during training and samples that are initially hard can become easier later.Therefore, fixed difficulty ordering is ineffective for rollout-based RL training.
  • Variance-based dynamic sampling: High reward variance identifies samples that are neither too easy nor too hard for the current model, making them more valuable for training.Easy samples tend to receive reward near 1 and hard samples reward near 0, so both produce low group-reward variance.
  • Variance-based dynamic sampling: The method computes variance after costly rollouts, then uses threshold-based dynamic sampling and replay to focus training on high-value samples.This design addresses the computational cost of calculating p for every training sample solely for sampling.
  • Replay learning: VCRL uses a momentum-maintained priority queue as a memory bank to replay removed high-value queries and preserve the training-batch size.The priority combines the query’s variance value with the number of steps since it was last accessed.

4 EXPERIMENTS

VCRL is evaluated on five mathematical reasoning benchmarks using Qwen3-4B-Base and Qwen3-8B-Base against GRPO, DAPO, and GSPO. It achieves state-of-the-art performance across both model sizes, improves training trends and final results, and benefits from both proposed components.

  • 4.1 EXPERIMENTAL SETUP: The evaluation uses AIME-2024, AIME-2025, MATH500, OlympiadBench, and AMC23 to test VCRL across multiple mathematical difficulty levels.AIME-2024 and AIME-2025 are designated as high-difficulty benchmarks.
  • 4.2 MAIN RESULTS: VCRL achieves state-of-the-art performance across all five benchmarks on both Qwen3 model sizes, outperforming GRPO, DAPO, and GSPO.The main comparison reports consistent superiority across the full benchmark suite and both Qwen3-4B-Base and Qwen3-8B-Base.
  • 4.2 MAIN RESULTS: 57.76 average score on Qwen3-8B-Base exceeds GSPO by over 4.67 points, while Qwen3-4B-Base reaches 49.43 versus 26.68 for the base model.The reported gains are especially pronounced on the challenging AIME-2024 and AIME-2025 datasets.
  • 4.3 PERFORMANCE TREND: VCRL improves performance faster during early training and retains significantly better final results than the RL baselines.Its performance curve stays above other methods during the first 100 training steps, while methods generally converge later.
  • 4.4 ABLATION STUDY: 49.43 average score after adding Replay Learning follows 44.73 with Variance-based Dynamic Sampling and 41.76 for Naive GRPO.The ablation trend is also reported for the larger Qwen3-8B-Base model, supporting positive contributions from both components.

5 RELATED WORK

Related work develops policy-based RL methods for improving LLM reasoning, efficiency, stability, and curriculum selection. Existing efforts include RL mechanisms, reasoning-trace exploration, and difficulty-aware training approaches.

  • Policy-based reinforcement learning: Recent RL methods improve LLM reasoning by using model exploration and rollout-based optimization rather than relying only on labeled supervision.The literature includes GRPO and several variants designed for LLM reasoning tasks.
  • RL efficiency and stability: Efficiency and stability methods modify RL objectives, clipping, sampling, token weighting, or off-policy reasoning traces.Examples include DAPO, Dr. GRPO, SimpleRL-Zoo, and LUFFY.
  • Curriculum learning: Curriculum-oriented work selects training examples using reward design or predicted difficulty scores to improve LLM training.These approaches include self-adaptive curriculum selection and reward functions balancing correctness, length, and formatting.
  • RLVR for reasoning: RLVR research also examines reasoning behavior, chain-of-thought structure, and critical tokens as factors related to reasoning improvement.Several studies investigate how exploration or KL penalties affect decision points in generated reasoning traces.

6 CONCLUSION

VCRL dynamically controls training-sample difficulty using group-reward variance and combines variance-based sampling with replay learning. Across five mathematical benchmarks, it achieves state-of-the-art results against LLM RL baselines.

  • 6 CONCLUSION: VCRL filters training samples by group-reward variance, retaining moderately difficult examples while removing samples that are too easy or too difficult.Replay Learning maintains high-value samples in a memory bank to improve training stability.
  • 6 CONCLUSION: VCRL achieves state-of-the-art results on five mathematical benchmarks compared with LLM RL baselines.Training-dynamics analysis and ablation studies further confirm the method’s effectiveness.

A TRAINING DYNAMICS

Training-dynamics analysis compares VCRL with GRPO using reward score, response length, and entropy for Qwen3-4B-Base and Qwen3-8B-Base. VCRL shows higher and more stable rewards, longer responses, and sustained entropy during training.

  • Monitoring indicators: Figure 4 tracks reward score, response length, and entropy as monitoring indicators of RL training dynamics and potential issues.Reward curves use moving averages and rolling standard deviations with a window size of 20 for stability visualization.
  • Reward score: VCRL maintains higher reward scores and smaller fluctuations than GRPO after stabilization, indicating more stable training dynamics.For Qwen3-4B-Base, VCRL’s reward is much higher before about 270 steps; for Qwen3-8B-Base, it remains significantly higher throughout training.
  • Response length: VCRL’s response length grows faster early and remains noticeably longer than GRPO’s after stabilization.The analysis relates longer responses to greater exploration during reasoning training.
  • Entropy: VCRL keeps entropy at a reasonable level throughout training, whereas GRPO falls below 0.1 within 50 steps.The reported entropy behavior is associated with continued exploration during training.

B VARIANCE AS A DIFFICULTY METRIC

Group reward variance provides a task-level signal of sample difficulty: it is low for samples the model almost always solves or fails, and highest near a 50% success rate. This makes variance more directly relevant to learning difficulty than generation entropy.

  • VCRL uses group reward variance to identify samples at the cusp of the model’s current capabilities.This supports using variance to estimate current learning difficulty in VCRL or GRPO.
  • Maximum variance occurs when approximately half of the G = 16 rollouts succeed, marking samples near the model’s capability boundary.Too-easy samples yield nearly all rewards of 1, while too-hard samples yield all rewards of 0, producing near-zero variance in both cases.
  • Group reward variance measures task-level outcome uncertainty, whereas generation entropy measures token-level action diversity.High entropy can accompany nonsensical responses that all receive reward 0, while reward variance directly reflects final task success.

C POLICY GRADIENT REDUCTION

VCRL is theoretically guaranteed to have no larger expected policy-gradient norm than GRPO, and Figure 6 empirically compares their gradient-norm dynamics on two Qwen3 model scales. The reported results characterize VCRL as producing a more stable optimization trajectory.

  • VCRL has a no-larger expected policy-gradient norm than GRPO, providing a theoretical guarantee of greater training stability.Theorem 1 states E_VCRL[∥∇θ log πθ∥] ≤ E_GRPO[∥∇θ log πθ∥].
  • The gradient comparison is derived by expressing both GRPO and VCRL objectives through policy-gradient forms and aligning VCRL with importance sampling.The VCRL expression includes memory-bank-related sampling terms and an indicator controlling policy-gradient contribution.
  • The proof bounds VCRL’s gradient contribution using the sampling-probability and indicator-function constraints in its objective.The derivation uses P(x ∈ D ∪ M) ≤ P(x ∈ D) and I(p_i ≥ κ) ≤ 1.
  • VCRL’s gradient norm remains in a lower, narrower band than GRPO’s on Qwen3-4B-Base and Qwen3-8B-Base.The reported comparison interprets this pattern as more measured and stable policy updates.
Loading 2509.19803v1…