PaperlayerView exact revision on arXiv

Source-linked AI summary

CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs

Yongcheng Zeng, Zexu Sun, Bokai Ji, Erxue Min, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Haifeng Zhang, Xu Chen, Jun Wang

arXiv:2510.01037v2cs.LGcs.AI

TL;DR

Uniform prompt sampling can waste computation because prompts differ in difficulty and training utility. CurES uses gradient analysis to adapt prompt sampling and rollout quantities, refining difficulty estimates with Bayesian posterior updates. It matches or surpasses strong baselines across settings and accelerates convergence by up to 5.5 times versus uniform-sampling baselines like GRPO.

  • Problem

    Uniform sampling treats training prompts identically despite differences in difficulty and training utility, while existing methods may not fully account for those variations.

    Source: §1 · p. 1

  • Method

    CurES uses gradient analysis to guide difficulty-based prompt sampling and rollout allocation, refining prompt-accuracy estimates through Bayesian posterior updates.

    Source: §1 · p. 2

  • Results

    CurES matches or surpasses the strongest baselines across settings and accelerates convergence by up to 5.5 times compared with uniform-sampling baselines like GRPO.

    Source: §1 · p. 2, §5.1 · p. 9

  • Takeaways & Limitations

    CurES offers reasoning-model training with enhanced efficiency and stability at minimal computational overhead.

    Source: §1 · p. 2

    The prompt-success model assumes rollout correctness is binomial and models its success probability with a Beta distribution.

    Source: §4.3 · p. 6

Abstract

from arXiv · show

Curriculum learning plays a crucial role in enhancing the training efficiency of large language models (LLMs) on reasoning tasks. However, existing methods often fail to adequately account for variations in prompt difficulty or rely on simplistic filtering mechanisms to select prompt datasets within a narrow criterion range, resulting in significant computational waste. In this work, we approach the problem from the perspective of reinforcement learning gradient optimization, offering a systematic and theoretical investigation into how to improve the training efficiency of LLMs. We identify two key factors influencing training efficiency: the selection of training prompts and the allocation of rollout quantities across different prompts. Our theoretical analysis reveals that the sampling distribution of prompts dictates the convergence rate of gradient descent, while the allocation of the rollout quantity influences the consistency and stability of overall gradient updates. Based on these insights, we propose CurES, an efficient training method that accelerates convergence and employs Bayesian posterior estimation to minimize computational overhead. Experiments demonstrate that our CurES outperforms Group Relative Policy Optimization (GRPO) by +3.30 points and +4.82 points with 1.5B and 7B models, respectively, and exceeds the best prior sample efficient methods by +2.12 points on average across eight math reasoning benchmarks. Additionally, CurES exhibits faster convergence compared to baselines, including GRPO.

1 INTRODUCTION

Uniform prompt sampling can waste resources on examples that are too easy or too difficult, while existing curricula and filtering methods do not fully adapt to prompt difficulty. CurES grounds prompt selection and rollout allocation in gradient analysis and Bayesian estimation.

1 INTRODUCTION

  • Prompt difficulty varies in training utility, so uniform sampling can waste computation on trivial prompts or examples where the model makes negligible progress.The introduction frames this as a limitation of approaches that treat all training instances identically.Source: §1 · p. 1
  • Hand-crafted difficulty stages can be too coarse for changing model capabilities, while online filtering generates and prunes samples.The passage contrasts these approaches with the need to account for training’s inherent dynamism.Source: §1 · p. 1
  • CurES analyzes gradient efficiency in relation to prompt sampling probabilities and rollout quantities, then reallocates both using estimated question-answering accuracy.It progressively refines confidence in accuracy estimates through Bayesian posterior estimation.Source: §1 · p. 2, Figure 1 · p. 2
  • CurES accelerates convergence by up to 5.5 times compared with uniform-sampling baselines like GRPO.The introduction also reports state-of-the-art performance across diverse mathematical reasoning datasets and model scales.Source: §1 · p. 2

2 RELATED WORKS

Prior work applies gradient analysis to optimization stability, learning-rate adaptation, data selection, and anomaly detection. Curriculum and data-selection methods for RLVR face challenges in adapting to model progress and reducing rollout overhead.

2 RELATED WORKS

3 PRELIMINARIES

The preliminaries describe RLVR as reinforcement learning for tasks with programmatically verifiable rewards and present its objective and binary reward formulation. The reward design is intended to partially mitigate reward hacking.

3 PRELIMINARIES

  • RLVR is a reinforcement-learning paradigm for reasoning tasks where rewards can be deterministically verified programmatically.The paper identifies mathematical reasoning, code generation, and logical deduction as suitable domains with objectively defined correctness criteria.Source: §3 · p. 3
  • The RLVR objective maximizes expected reward while constraining policy updates within a trust region.The trust-region boundary δ limits excessive policy divergence.Source: §3 · p. 3
  • RLVR uses a binary reward: 1 for a correct answer and 0 otherwise.The paper states that this straightforward reward formulation partially mitigates reward hacking.Source: §3 · p. 3

4 METHODOLOGY

CurES connects prompt difficulty to optimization potential, uses prompt sampling and rollout allocation to improve gradient efficiency, and updates difficulty estimates through Bayesian inference. Its implementation separates lightweight estimation from large-scale training.

4.1 PROMPT DIFFICULTY CAPS OPTIMIZATION POTENTIAL

  • Prompt difficulty is defined as the model’s question-answering accuracy, and the loss function’s optimization potential is related to that difficulty.The analysis uses this relationship to motivate difficulty-dependent prompt sampling while retaining exploration through an entropy constraint.Source: §4.1 · pp. 4–5
  • The optimal prompt sampling distribution varies prompt probabilities by difficulty to influence loss convergence while balancing exploration.The derivation uses a trust-region formulation and Fisher information to study update efficiency.Source: §4.1 · pp. 4–5

4.2 CLOSING THE GAP WITH THEORETICAL BOUND

  • With a fixed rollout budget, CurES allocates rollout quantities to minimize total gradient-estimator variance within a KL-bounded policy update.The variance formulation reuses prompt-difficulty estimates and distinguishes correct from incorrect answers when estimating gradient variance.Source: §4.2 · pp. 5–6

4.3 PROMPT DIFFICULTY ASSESSMENT AND ALGORITHMIC IMPLEMENTATION

  • CurES models each prompt’s success probability with a Beta distribution and updates its parameters using observed correct and incorrect rollouts.The model assumes rollout correctness is binomial; the posterior is refined as new samples arrive.Source: §4.3 · p. 6
  • Each iteration resets difficulty and gradient-variance estimates across non-overlapping dataset subsets, then runs lightweight estimation followed by large-scale training.This design is described as alleviating distribution shift without significant overhead; the practical update assumes a learning rate that keeps policy changes within the KL constraint δ.Source: §4.2 · p. 5, §4.3 · p. 7

5 EXPERIMENTS

Experiments compare CurES with strong baselines across mathematical reasoning benchmarks and examine overall performance, adaptive sampling, and sample efficiency.

5 EXPERIMENTS

  • CurES is evaluated on mathematical-reasoning benchmarks including MATH500, AIME 2024 and 2025, AMC 2023, GSM8K, Gaokao-EN 2023, Mineva, and OlympiadBench, and compared with GRPO, RPP, Speed-RL, and GVM.Numina-Math is partitioned into 15 subsets, resulting in 15 training iterations.Source: §5 · p. 8

5.1 OVERALL PERFORMANCE (RQ1)

  • CurES consistently outperforms GVM and other baselines with GRPO or RPP, matching or surpassing the strongest baselines across both model scales and all settings.Table 1 reports quantitative results across datasets.Source: Table 1 · p. 9, §5.1 · p. 9

5.2 SAMPLING BEHAVIOR (RQ2)

  • As training proceeds, estimated accuracy shifts higher and concentrates; success rates are mainly bimodal, supporting redistribution over uniform sampling followed by accuracy filtering.The reported bimodality is concentrated at high and low success rates.Source: §5.2 · p. 9
  • CurES allocates more rollouts to intermediate-accuracy prompts, sharpening that allocation as such prompts diminish to keep informative prompts in each batch.The authors connect coupling difficulty-based sampling with adaptive sample sizing to performance gains per step.Source: §5.2 · pp. 9–10

5.3 EFFICIENCY ANALYSIS (RQ3)

  • Increasing either the accuracy-estimation sample size N ′ or budget coefficient n does not yield proportional performance gains relative to added computation.The authors report that relatively small values of both are sufficient for effective training.Source: §5.3 · p. 10
  • 5.5× fewer steps let CurES-GRPO reach GRPO’s peak performance, while CurES-RPP reaches RPP’s peak 1.75× faster.These comparisons are reported on MATH500.Source: §5.3 · p. 10

6 CONCLUSION

CurES combines gradient-based curriculum analysis with Bayesian prompt-difficulty estimation to adapt prompt selection and rollout budgets. Across mathematical reasoning benchmarks, it consistently outperforms strong baselines in accuracy and convergence speed.

  • CurES links training efficiency to prompt accuracy, adaptively prioritizing prompts and dynamically allocating rollout budgets.Its analysis connects sampling distribution with gradient efficiency and rollout allocation with update consistency and stability.Source: §6 · p. 10
  • A Bayesian framework models each prompt’s success rate with a Beta distribution and updates its posterior as new rollouts are observed.The approach uses prior information to track prompt difficulty in a lightweight, adaptive manner.Source: §6 · p. 10
  • CurES consistently outperforms strong baselines in accuracy and convergence speed across a wide range of mathematical reasoning benchmarks.Source: §6 · p. 10

A MATHEMATICAL DERIVATIONS

The appendix states the assumptions for its Cramér–Rao inequality lemma, including regularity and unbiased estimation.

A.1 PROMPT DIFFICULTY CAPS OPTIMIZATION POTENTIAL

The derivation connects prompt difficulty to optimization potential and uses that relationship to motivate an optimal sampling distribution.

  • The derivation treats the model’s question-answering accuracy pθ(x) as the quantity estimated by the reward.It states that r(x, y) is an unbiased estimator of pθ(x).Source: §A.1 · p. 16
  • The analysis states that a prompt’s optimization potential is inherently related to its difficulty.Source: §A.1 · p. 16
  • The optimal prompt sampling distribution is derived to balance exploration and exploitation under a maximum-entropy constraint with hyperparameter α.Source: §A.1 · p. 16

A.2 CLOSING THE GAP WITH THEORETICAL BOUND

The appendix derives a rollout-allocation strategy for approaching a gradient-update bound within a KL-constrained trust region. Because the theoretical gradient direction is unknown, it minimizes total estimator variance under a fixed rollout budget.

A.2 CLOSING THE GAP WITH THEORETICAL BOUND

  • The practical update seeks to approach the theoretical efficiency limit within a KL-divergence trust region of δ, while allocating rollouts under a fixed total budget N.The natural-gradient method is often avoided in practice because of its high computational cost.Source: §A.2 · p. 17
  • Because the theoretical gradient direction is unknown, the analysis minimizes total estimator variance, Tr(V(ĝ)), to reduce variance uniformly across directions.This replaces minimizing uncertainty in a particular direction with a direction-independent objective.Source: §A.2 · p. 19
  • CurES variants exhibit higher average gradient norms in three of four algorithm-and-model-scale combinations.The figure compares CurES-GRPO and CurES-RPP with GRPO and RPP.Source: Figure 7 · p. 21
  • The allocation derivation differentiates the objective under the rollout constraint to obtain a strategy for distributing rollouts across prompts.The total sampling budget is N.Source: §A.2 · p. 21

B EXTENDED EXPERIMENTAL RESULTS

Extended experiments report faster convergence and stronger performance for CurES, alongside analyses of gradient signals, rollout allocation, sampling configurations, component contributions, and additional model families.

B EXTENDED EXPERIMENTAL RESULTS

Related papers

Loading 2510.01037v2…