Source-linked AI summary
Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning
Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, Donghyun Kwak
TL;DR
Prior RLVR work lacked a theoretical foundation connecting difficulty-aware data curation with online reinforcement learning. This paper develops such a foundation and validates it empirically, showing that intermediate-difficulty tasks and balanced filtering support policy improvement, performance, and sample efficiency.
Problem
Recent RLVR work applies difficulty-aware data curation but lacks detailed theory linking it to online reinforcement learning algorithms.
Method
The paper uses sample reward variance as a proxy for online filtering and introduces asynchronous sampling to replace filtered items while preserving a fixed batch size.
Results
Intermediate-difficulty tasks maximize the lower bound of policy improvement, while experiments across multiple reasoning benchmarks show consistent gains in performance and sample efficiency.
Takeaways & Limitations
The results provide theoretical and empirical guidelines for difficulty control in RLVR, connecting prior empirical heuristics with formal learning principles.
Takeaways & Limitations
Empirical validation was conducted solely on math reasoning tasks, leaving coding, scientific reasoning, larger-scale, and broader model-family applicability for future work.
Abstract
from arXiv · showhide
Recent advances in reinforcement learning with verifiable rewards (RLVR) show that large language models enhance their reasoning abilities when trained with verifiable signals. However, due to reward sparsity, effectiveness depends heavily on selecting samples of appropriate difficulty. In this work, we present a formal analysis of online difficulty-aware filtering and establish its theoretical foundations. We show that expected policy improvement is lower-bounded by the variance of task-level success probabilities, implying that selecting tasks of intermediate difficulty maximizes learning efficiency. Building on this, we demonstrate that balanced filtering maximizes this lower bound, leading to superior performance and sample efficiency. Evaluations across multiple math reasoning benchmarks validate that balanced filtering consistently enhances convergence speed and final performance, achieving up to +12% gains in less than half the training steps of standard GRPO. By extending our analysis to various reward distributions, we provide a principled foundation for future RLVR curriculum strategies, confirmed through both theoretical analysis and extensive empirical results.
1 Introduction
The paper develops theoretical and empirical foundations for online difficulty-aware filtering in RLVR, targeting training-efficiency bottlenecks by retaining appropriately difficult prompts. It connects intermediate difficulty and balanced filtering to improved learning efficiency, with gains validated across math reasoning benchmarks.
- Online difficulty-aware filtering addresses the lack of theoretical analysis connecting intermediate-difficulty curation to online RLVR algorithms.The paper frames this as a missing theoretical foundation for difficulty-aware filtering.
- Sample reward variance lower-bounds policy separation and serves as a proxy for filtering prompts by learnability.The proposed foundation links the variance proxy to the reversed KL divergence between initial and optimal policies.
- +10% on AIME and +4.2% average performance are reported for 3B models across five math reasoning benchmarks.
- Less than half the gradient updates are needed to outperform plain GRPO with filtering.
- The analysis extends the reward-variance learnability proxy to Gaussian and Multinomial reward distributions.
2 Preliminaries
The preliminaries describe KL-regularized reinforcement learning for language models and GRPO, which estimates policy gradients from grouped sampled responses without a parameterized value model.
- RL optimizes a training policy initialized from a reference policy to maximize reward while penalizing excessive divergence.
- The optimal policy is defined relative to the initial policy, with a partition function normalizing action probabilities.
- GRPO excludes a parameterized value model and uses a PPO-style clipped surrogate objective.
- GRPO weights each trajectory’s log-likelihood by its advantage, computed from sampled responses and their rewards.
- For verifiable tasks, rewards are represented with binary correctness signals.
3 Learnability in GRPO and Online Difficulty Filtering
The analysis shows that binary-reward prompts at intermediate difficulty provide the strongest learning signal, motivating balanced online filtering with asynchronous sampling and fixed batch size.
- 3.1 Background: For binary rewards, a prompt’s pass rate p(x) determines Bernoulli reward variance p(x)(1 − p(x)).
- 3.1 Background: Zero variance occurs exactly at p(x)=0 or p(x)=1, corresponding to absolute-hard and absolute-easy prompts.
- 3.2 Prompt-level learnability: theoretical analysis: When p(x) approaches the extremes, reverse KL divergence tends to zero and provides no effective update.
- 3.2 Prompt-level learnability: theoretical analysis: When p(x)≈0.5, reward variance and learnability are maximized, motivating retention of an intermediate-difficulty band.
- 3.3 Method: online difficulty filtering with fixed batch size: Online filtering dynamically assesses difficulty during each training step rather than following a predefined static ordering.
- 3.3 Method: online difficulty filtering with fixed batch size: Asynchronous sampling replaces filtered examples while maintaining the fixed batch size |B|=N.
- 3.3 Method: online difficulty filtering with fixed batch size: Balanced filtering uses symmetric thresholds around success rate 0.5, whereas skewed filtering removes only easy or hard prompts.
4 Experiments
The experiments evaluate online difficulty filtering for RLVR using GRPO across five math reasoning benchmarks and compare it with plain GRPO and offline filtering methods. The design also tests balanced and skewed threshold strategies and evaluates scalability with 3B and 7B models.
- Experimental setup: GRPO trains on NuminaMath prompts by generating 16 rollouts for each of 16 prompts and assigning correctness-based rewards.The setup also includes format and language rewards, with 1,024 problems held out for validation.
- Filtering strategies: The study compares plain GRPO, balanced online filtering, skewed online filtering, and two offline filtering strategies.Offline strategies include data curation before training and scheduling that additionally orders training batches.
- Filtering strategies: Online filtering thresholds are varied across balanced settings from (0, 1) to (0.4, 0.6) and skewed settings that independently sweep TLow or THigh.The balanced and skewed sweeps test how threshold choices affect training.
- Evaluation: Offline filtering uses Qwen2.5-7B-Instruct or the SFT model as difficulty proxies, enabling comparison between external and initial-model assessments.The comparison covers both offline data curation and offline scheduling.
- Evaluation: Performance is measured with pass@1 on MATH500, AIME, AMC, MinervaMath, and OlympiadBench, spanning different difficulty levels.Table 1 reports Qwen2.5-3B results and identifies pass rate p(x) as average rollout correctness.
5 Results
Balanced online difficulty filtering outperforms plain GRPO and offline filtering across the reported math reasoning evaluations. Stricter balanced thresholds improve benchmark averages, while skewed filtering remains below plain GRPO; the trend also holds at 7B scale.
- 5.1 Balanced and skewed filtering: Balanced filtering raises the five-benchmark average from 26.3% with plain GRPO to over 30%, with up to 10% point improvement on AIME.The improvement is reported across all five threshold choices and benchmarks.
- 5.1 Balanced and skewed filtering: The five-benchmark average increases from 27.3% at (0, 1) to over 30% at (0.3, 0.7) as balanced thresholds become stricter.Removing examples that do not contribute to learning produces a slight improvement over the baseline.
- 5.1 Balanced and skewed filtering: Skewed filtering reaches around 24.9% to 25.9%, below plain GRPO’s 26.3% across every tested threshold choice.Skewed filtering nevertheless improves up to 5.7% over the SFT checkpoint.
- 5.2 Offline and online filtering: Offline curation and scheduling provide at most 2.1% improvement over plain GRPO, whereas balanced online filtering consistently outperforms them.Offline proxies based on an external model exceed the initial-model proxy on average, with benchmark-specific variation.
- 5.3 Scalability: At 7B scale, thresholds satisfying 0.3 < p(x) < 0.7 produce the strongest results, increasing AIME by 3% and AMC by 5%.The ascending performance trend aligns with the 3B experiments.
6 Analysis
Balanced online filtering maintains intermediate difficulty during training, while theoretical extensions relate learnability to reward fluctuations across broader reward distributions. Empirically, the approach improves reward, performance, and efficiency relative to plain GRPO.
- 6.1 Learning dynamics analysis: Balanced filtering targets intermediate difficulty and maintains an average difficulty around 0.5 during GRPO.Difficulty is defined as 1 − p(x), so higher difficulty corresponds to lower sample accuracy.
- 6.1 Learning dynamics analysis: External-proxy offline filtering provides partially meaningful but imperfectly aligned difficulty estimates, whereas balanced online filtering yields better performance and efficiency.The comparison is supported by marginal improvements over plain GRPO for offline filtering and stronger results for online filtering.
- 6.1 Learning dynamics analysis: 0.4 < p(x) < 0.6 achieved the highest overall reward and surpassed plain GRPO in fewer than 50% of the gradient updates.Figure 3 compares validation reward against both training steps and wall-clock-relative time.
- 6.3 Theoretical generalizability: The reverse KL between the initial and optimal policies can be expressed through the cumulant generating function of centered rewards.The formulation uses the centered reward r − µ and evaluates its CGF at t = 1/β.
- 6.3 Theoretical generalizability: For Gaussian rewards, reverse-KL learnability is exactly proportional to reward variance, making sample variance an unbiased and consistent proxy with sufficient rollouts.The variance term is the leading component governing learnability in the broader CGF view.
- 6.3 Theoretical generalizability: For multinomial rewards, variance provides tight second-order control of reverse KL, while higher cumulants capture skewness and tail behavior.The binary reward case is recovered as the special instance N=1, extending the analysis to multiple verifiable objectives.
7 Conclusion
The paper establishes theoretical foundations for online difficulty-aware filtering in RLVR and tests them across multiple reasoning benchmarks. It concludes that intermediate difficulty and theoretically grounded difficulty control are associated with improved sample efficiency and performance.
- Intermediate-difficulty tasks maximize the lower bound of policy improvement, providing a theoretical explanation for difficulty-based data curation.
- Extensive ablations across multiple reasoning benchmarks verify the theoretical insights and demonstrate consistent gains in sample efficiency and performance.
- The results connect empirical difficulty-curation heuristics with formal learning principles for RLVR.
Limitations
The empirical validation is limited to math reasoning tasks, while the theoretical analysis assumes binary rewards and conditions supporting second-order expansions. Experiments also use a specific model, dataset, and hardware configuration.
- Empirical validation covers only math reasoning tasks, leaving coding and scientific reasoning for future work.
- The binary-reward proof assumes β > 0 with 1/β ≪ 1 so that second-order Taylor expansions are valid.
- Experiments use Qwen2.5-3B, a 1,107-problem SFT dataset, and 8×NVIDIA A100 GPUs.
- The reported reinforcement-learning configuration runs for 256 steps with 16 rollouts and 16 distinct prompts per batch.
- The reward combines format, language, and binary accuracy components into a score ranging from 0 to 3.
E Evaluation Benchmarks
The evaluation uses five challenging math reasoning benchmarks spanning contest, undergraduate or Olympiad, and open-ended competition problems. Inference uses fixed sampling and output-length settings.
- MATH500 contains 500 problems sampled from a source dataset while maintaining topic and difficulty balance.
- AIME uses 30 problems from the 2024 official competition, and AMC uses 40 from the 2023 official competition.
- MinervaMath evaluates quantitative reasoning with complex undergraduate- or Olympiad-level mathematical problems.
- OlympiadBench contributes 674 open-ended text-only competition problems from its specified English mathematics subset.
- Inference uses top-p 0.95, temperature 0.6, and a maximum output length of 8,192 tokens.
F Difficulty-Aware Performance Analysis
Difficulty-aware filtering improves performance most clearly on harder tasks and difficulty levels. Tighter thresholds generally help challenging benchmarks, while balanced filtering outperforms plain GRPO across most MATH500 levels.
- Benchmark-Level Difficulty Spectrum: The SFT checkpoint spans 0.0%–49.8% across benchmarks ordered from AIME to MATH500 by increasing performance.
- Benchmark-Level Difficulty Spectrum: Tighter difficulty thresholds generally improve performance, especially on challenging benchmarks such as MinervaMath and AIME.
- Benchmark-Level Difficulty Spectrum: AIME shows more than 300% relative improvement over SFT, while MinervaMath improves by 35%.
- Difficulty-Level Breakdown within MATH500: Balanced-filtering GRPO outperforms plain GRPO across most MATH500 difficulty levels, especially Levels 3–5.
- Difficulty-Level Breakdown within MATH500: Table 3 reports accuracy percentages for GRPO-trained models on MATH500 by difficulty level, with the highest score for each level bolded.
G.3 Proof of Corollary 6.3.
The corollary extends the variance-based analysis to bounded or multinomial rewards using a cumulant-generating-function expansion. The variance term dominates at second order, yielding a lower-order control that transfers to reverse KL divergence.
- The proof writes the centered reward as Z := rM − µ and expands its cumulant-generating function around t = 0.
- For bounded multinomial rewards, all moments and cumulants of the centered reward are finite.
- The variance term dominates at second order and provides lower control up to O(β^-3) for sufficiently large β.
- Applying Proposition 6.1 translates the resulting bound into a reverse-KL statement.
- The broader analysis reports an exact relationship for Gaussian rewards and a tight second-order lower bound for multinomial or bounded rewards.