PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Reinforcement Finetuning via Adaptive Curriculum Learning

Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, Jieyu Zhao

arXiv:2504.05520v4cs.LGcs.CL

TL;DR

RFT can improve mathematical reasoning but remains sample- and compute-inefficient. AdaRFT dynamically matches problem difficulty to the model’s evolving skill using reward feedback. Experiments report consistent convergence-speed gains, especially on imbalanced distributions, while the reported 2× speedup excludes upfront difficulty estimation.

  • Problem

    RFT remains sample-inefficient and computationally expensive because training repeatedly generates rollouts, computes rewards, and updates the policy.

  • Method

    AdaRFT selects problems near a reward-responsive target difficulty and updates that target as the model’s average reward changes.

  • Results

    Experiments across multiple data regimes and model sizes show consistent gains in convergence speed, especially for imbalanced training distributions.

  • Takeaways & Limitations

    AdaRFT provides a lightweight, scalable curriculum strategy for improving RFT efficiency without modifying the reward function or underlying RL algorithm.

  • Takeaways & Limitations

    The reported 2× speedup covers subsequent RFT training after difficulty scoring, excluding the upfront cost of scorer rollouts.

Abstract

from arXiv · show

Reinforcement finetuning (RFT) has shown great potential for enhancing the mathematical reasoning capabilities of large language models (LLMs), but it is often sample- and compute-inefficient, requiring extensive training. In this work, we introduce AdaRFT (Adaptive Curriculum Reinforcement Finetuning), a method that significantly improves the efficiency of RFT through adaptive curriculum learning. AdaRFT dynamically adjusts the difficulty of training problems based on the model's recent reward signals, ensuring that the model consistently trains on tasks that are challenging but solvable. This adaptive sampling strategy accelerates learning by maintaining an optimal difficulty range, avoiding wasted computation on problems that are too easy or too hard. AdaRFT requires only a lightweight extension to standard RFT algorithms like Proximal Policy Optimization (PPO), without modifying the reward function or model architecture. Experiments on competition-level math datasets demonstrate that AdaRFT improves convergence efficiency and reasoning performance. Given problem-level difficulty annotations, AdaRFT reduces RFT training time by up to 2 times across data distributions and model scales, offering a more scalable and effective RFT framework.

1 Introduction

RFT offers targeted learning for mathematical and coding tasks, but repeated rollouts, reward computation, and policy updates make it sample-inefficient and costly. AdaRFT addresses these limitations by adapting problem difficulty to the model’s evolving skill level.

  • RFT uses task-success rewards to align LLMs with domain-specific goals, offering more targeted learning than SFT alone.
  • Repeated rollout generation, reward computation, and policy updates make RFT expensive and difficult to scale.
  • Existing algorithmic and data-centric efficiency methods can trade off stability, depend on brittle scoring, or require model-specific processing.
  • Fixed datasets, schedules, and manually staged curricula can be suboptimal when training data are non-uniform or imbalanced.
  • AdaRFT dynamically adjusts training difficulty from recent rewards, selecting tasks near a target level that are challenging but solvable.
  • AdaRFT is a lightweight, model-agnostic extension applicable to standard RL algorithms such as PPO, evaluated on competition-level mathematics across distributions and model sizes.

2 Related Work

Prior RFT efficiency work simplifies reinforcement learning or reduces and selects data, while curriculum methods organize task difficulty. AdaRFT instead continuously adapts difficulty from reward feedback without fixed schedules or repeated rollout filtering.

  • RFT variants such as RAFT, ReMax, GRPO, and REINFORCE++ simplify RL components to reduce computational overhead.
  • Data-centric methods show that small or selected datasets can support strong performance, but scoring may require full retraining and recomputation for each model.
  • Reducing training samples does not necessarily reduce optimization steps or wall-clock convergence time.
  • Classical RL curriculum learning includes difficulty sorting, teacher-student task selection, and self-play, but its application to LLM RFT remains limited.
  • Existing RFT curricula use hand-designed difficulty tiers or repeated rollout-based filtering, limiting adaptability or increasing computational overhead.
  • AdaRFT continuously adjusts task difficulty from rewards, avoiding fixed schedules and repeated rollouts while remaining model-agnostic.

3 AdaRFT

AdaRFT trains on problems near a reward-responsive target difficulty, updating that target as model performance changes. Its intermediate-success motivation links reward variance to informative updates, while explicitly limiting the claim to intuition rather than a global optimization guarantee.

  • 3 AdaRFT: AdaRFT aims to train a policy on mathematical problems whose difficulty matches the model’s current capability.
  • 3 AdaRFT: Each problem has a precomputed human-annotated or model-estimated difficulty score, and the curriculum is controlled by a target difficulty T.
  • 3.1 Dynamic Curriculum Sampling: AdaRFT selects the B problems closest to T, focusing training on tasks that are neither too easy nor too hard.
  • 3.1 Dynamic Curriculum Sampling: The selected batch generates responses, receives correctness-based rewards, and updates the policy through PPO, GRPO, or REINFORCE++.
  • 3.3 Target Difficulty Update: High average reward increases T, while poor reward decreases it, adapting the curriculum to the model’s evolving performance.
  • 3.3 Target Difficulty Update: The clipped tanh update provides smooth, bounded target-difficulty changes, with η controlling step size, α sensitivity, and β the target reward.
  • 3.4 Motivation for Target Reward β: Reward variance is maximized at a 50% success rate, motivating β = 0.5 as an informative intermediate target without proving globally maximal learning progress.

4 Experiments

The experiments evaluate difficulty estimation, dataset-distribution effects, and AdaRFT against PPO-based baselines across six math benchmarks and two Qwen model sizes. Results indicate that adaptive curriculum learning supports efficient training across uniform and skewed difficulty distributions.

  • 4.1 Difficulty Estimation: AdaRFT estimates difficulty using Qwen 2.5 MATH 7B, selecting a balanced model because excessively strong or weak estimators provide poor discrimination.Each problem’s score is computed from 128 attempts as d_i = 100(1 − s_i/n).
  • 4.1 Difficulty Estimation: 86.0% at level 1 versus 52.7% at level 5, with AoPS difficulty correlating negatively with model success rates (r = −0.34, p < 0.05).The alignment supports using labeled difficulty levels and empirical estimates for curriculum learning.
  • 4.2 Dataset: Each skew-difficult, skew-easy, and uniform training distribution contains 10,000 samples, with problems concentrated toward hard, easy, or evenly balanced difficulty levels.The distributions are constructed because collected training data may not have a controllable difficulty profile.
  • 4.2 Dataset: Evaluation averages accuracy across MATH 500, GSM8K, AIME 24, AMC 23, OlympiadBench, and Minerva Math, spanning elementary through advanced competition mathematics.AMC 23 and AIME 24 use eight sampled responses per problem for more stable estimates.
  • 4.3 Training Setup: The comparison includes standard PPO, PPO with filtered data, PPO with a fixed curriculum, and AdaRFT (PPO) across uniform, skew-easy, and skew-difficult setups.Filtering removes problems solved at rates ≤10% or ≥90%, while fixed curriculum difficulty increases linearly regardless of learning speed.

5 Results and Analysis

Across two Qwen model sizes and multiple data distributions, AdaRFT improves RFT efficiency while maintaining or improving final accuracy. Its adaptive difficulty progression also generalizes beyond PPO and remains effective with approximate difficulty signals.

  • Overall Results: AdaRFT reaches comparable accuracy in fewer RFT training steps and achieves higher final accuracy in single-seed runs across two model sizes.The evaluation covers Qwen 2.5 MATH 1.5B and Qwen 2.5 7B, with results averaged across six benchmarks.
  • Training Efficiency: 43 extra steps (+71.7%) were needed by standard PPO to match AdaRFT on skew-difficult Qwen 2.5 MATH 1.5B, versus 34 steps (+56.7%) in the uniform setting.PPO with filtered data required +49 steps (81.7%) and +52 steps (86.7%) in the same settings.
  • Training Efficiency: 24 steps (60.0%) and 25 steps (62.5%) were additionally required by PPO and filtered-data PPO, respectively, to match AdaRFT on skew-difficult Qwen 2.5 7B.PPO with a fixed curriculum also trailed AdaRFT in convergence speed.
  • Model Performance: AdaRFT achieved higher final accuracy across all configurations, including 37.41% to 40.48% on skew-difficult Qwen 2.5 MATH 1.5B.For Qwen 2.5 7B, accuracy increased from 44.17% to 46.83% on skew-difficult data, from 44.70% to 46.92% in the uniform setting, and from 45.07% to 45.94% on skew-easy data.
  • Ablation on Target Reward β: β = 0.5 produced the highest accuracy throughout training and maintained reward near 0.5, whereas β = 0.8 emphasized easy problems and β = 0.2 emphasized difficult ones.Response length was shortest for β = 0.8 and longest for β = 0.2, consistent with their sampled problem complexity.
  • Data Difficulty: Uniform + AdaRFT achieved the highest overall accuracy, while hard-extreme training followed a flatter, lower trajectory and overly easy or hard problems provided weak learning signals.Uniform + AdaRFT maintained reward near 0.5 and steadily increased problem difficulty and response length during training.
  • Generality and Difficulty Estimation: AdaRFT improved convergence speed and final accuracy with both REINFORCE++ and GRPO, and LLM-judged difficulty still improved over standard PPO despite being less precise than rollout-based estimates.These results support compatibility across optimization methods and approximate difficulty sources.

6 Discussion

AdaRFT benefits most from adaptive curricula when data are imbalanced or model capacity is limited, while its reported speedups exclude the upfront difficulty-estimation cost.

  • When Does Curriculum Learning Help Most?: AdaRFT most improves curriculum learning under imbalanced training distributions and limited model capacity.Benefits are smaller for strong models or well-balanced data.
  • When Does Curriculum Learning Help Most?: AdaRFT prioritizes easier problems in skewed settings, enabling reward-sparse models to bootstrap before progressing to harder tasks.The target difficulty is adjusted using reward feedback.
  • Use of Offline Difficulty Estimation: AdaRFT uses fixed, solver-independent offline difficulty estimates while dynamically updating the target difficulty during training.This provides the functional outcome of online difficulty estimation without recomputing problem scores.
  • Efficiency Accounting: The reported efficiency results measure only policy optimization after curriculum scores are assigned, excluding curriculum-construction compute.The authors report curriculum construction and policy optimization separately.
  • Efficiency Accounting: 1.28M scorer rollouts are required to assign pass@128 difficulty scores to a 10,000-example pool in the main rollout-based setup.The cost can be reduced with fewer rollouts or LLM-judged difficulty and amortized across reuse of the scored dataset.

7 Conclusion

AdaRFT adaptively matches problem difficulty to evolving model skill using reward feedback. Across model sizes and data regimes, it improves convergence efficiency without changing the reward function or underlying RL algorithm.

  • 7 Conclusion: AdaRFT dynamically matches problem difficulty to a model’s evolving skill level.It adjusts a target difficulty using reward feedback.
  • 7 Conclusion: AdaRFT improves sample and compute efficiency during RFT with problem-level difficulty annotations.The method is presented as lightweight and scalable across data distributions and model sizes.
  • 7 Conclusion: AdaRFT shows consistent convergence-speed gains, especially in imbalanced training distributions, across multiple data regimes and model sizes.The approach does not modify the reward function or underlying RL algorithm.

A.1 Training Dynamics: AdaRFT vs. Fixed Curriculum vs. PPO

Training dynamics compare standard PPO, fixed-curriculum PPO, and AdaRFT using accuracy, reward, response length, and sampled difficulty over the first 100 steps. Fixed curricula can help initially but may escalate difficulty faster than the model learns, whereas AdaRFT adapts to learning progress.

  • A.1 Training Dynamics: Figure 8 tracks accuracy, reward, response length, and average sampled difficulty across training steps for three PPO-based methods.Curves are exponentially smoothed with α = 0.3.
  • A.1 Training Dynamics: PPO with a fixed curriculum initially converges slightly faster than standard PPO by exposing the model to easier problems.Its fixed schedule later increases difficulty independently of actual learning progress.
  • A.1 Training Dynamics: Fixed-curriculum PPO ultimately underperforms AdaRFT because its schedule pushes the model into harder problem regimes too quickly.The resulting misalignment appears in the reward dynamics.
  • A.1 Training Dynamics: AdaRFT keeps response lengths moderate and gradually increasing, unlike the rapid response-length increase under fixed-curriculum PPO.The paper treats longer responses as typically corresponding to more difficult reasoning tasks.
  • A.1 Training Dynamics: A fixed curriculum cannot remain aligned as perceived problem difficulty changes during training, causing over-challenged or under-challenged periods.AdaRFT addresses this mismatch by adjusting difficulty during training.

A.2 Detailed Breakdown of Model Performance

Across six mathematical reasoning benchmarks, AdaRFT matches or outperforms PPO-based baselines on most individual benchmarks and consistently improves average accuracy across model scales and data distributions.

  • A.2 Detailed Breakdown of Model Performance: AdaRFT matches or outperforms PPO-based baselines on most benchmarks across both model scales and all data distributions.The evaluation covers GSM8K, MATH 500, AMC 23, AIME 24, OlympiadBench, and Minerva Math.
  • A.2 Detailed Breakdown of Model Performance: AdaRFT produces consistent gains in overall average accuracy across models, setups, and data distributions.Table 2 reports final accuracy at step 100.
  • A.2 Detailed Breakdown of Model Performance: For Qwen 2.5 Math 1.5B, improvements are particularly pronounced on GSM8K, MATH 500, AMC 23, and AIME 24 under skew-difficult and uniform settings.Performance remains stable on harder benchmarks in the skew-easy setting.
  • A.2 Detailed Breakdown of Model Performance: For Qwen 2.5 7B, gains are more modest but remain consistent, with higher average accuracy across all three data distributions.Performance remains stable on the most challenging benchmarks.

B.1 Training Configuration

Experiments used PPO training on a single 8-A100 node, with Qwen2.5-7B and Qwen2.5-MATH-1.5B variants. The comparison covered PPO, GRPO, and REINFORCE++ hyperparameter configurations in veRL.

  • PPO training used both actor and critic models on a single node with 8 A100 GPUs.
  • The experiments evaluated Qwen2.5-7B and Qwen2.5-MATH-1.5B, training each model for approximately 100 optimization steps.The shorter-context Qwen2.5-MATH-1.5B variant required adjusted maximum response length and sequence parallel size.
  • Table 3 compares shared and algorithm-specific hyperparameters for PPO, GRPO, and REINFORCE++ using veRL.The settings include KL treatment modes, rollout configurations, and critic configurations.

B.2 Deriving the Target-Difficulty Update Rule from a Linear Mapping

The target-difficulty controller begins with linear reward-to-difficulty mappings, then updates the current target from reward deviation relative to a desired success rate. AdaRFT smooths this controller with a saturating hyperbolic tangent, bounds the update, and uses annotated difficulty scores for sampling.

  • Target-difficulty update: The target difficulty T is updated from the model’s observed average reward performance.The final controller is presented as a smooth, stabilized variant of a standard linear mapping between reward and difficulty.
  • Linear mapping: Linear interpolation maps a value from [x, y] into [a, b], providing the basis for converting reward values into difficulty values.
  • Linear mapping: Average reward Ravg is mapped from its reward interval to the difficulty range [dmin, dmax].In the main setting, reward spans [0, 1] and difficulty spans [0, 100].
  • Reward deviation: Rather than reassigning difficulty from absolute reward at every step, the controller adjusts the current target using deviation from target success rate β.With β = 0.5, the deviation range is [−0.5, 0.5].
  • Reward deviation: The naive linear controller increases difficulty above the target reward, decreases it below the target, and remains stable when Ravg = β.A symmetric difficulty-change interval [−∆, ∆] is obtained by mapping the reward-deviation interval to difficulty adjustments.
  • Difficulty estimation: Difficulty annotations are estimated by GPT-4o on a revised 1–5 scale after clipping because over 95% of problems received scores from 1 to 5.Half-point scores such as 1.5 are permitted, and the prompts and examples are sourced from the AoPS Wiki.
Loading 2504.05520v4…