Source-linked AI summary
Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Haizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, Beidi Chen
TL;DR
Scaling rollouts improves data selection for LLM reinforcement learning but incurs substantial computational overhead. The paper introduces GRESO, which uses reward dynamics to probabilistically filter uninformative prompts before rollout. Across math benchmarks and models, GRESO achieves up to 2.4× rollout and 2.0× total-training speedups without accuracy degradation.
Problem
Rollout scaling improves reinforcement-learning data quality but creates substantial computational overhead, while existing selection methods do not efficiently estimate prompt value before rollout.
Method
GRESO is an online pre-rollout filtering algorithm that uses reward-training dynamics to predict and probabilistically skip zero-variance prompts.
Results
Across six math reasoning benchmarks and three models, GRESO achieves up to 2.4× rollout speedup and 2.0× total training-time speedup while maintaining comparable accuracy.
Takeaways & Limitations
Reward dynamics can support efficient selective rollout by reducing sampling of uninformative prompts while retaining online adaptation and occasional exploration.
Takeaways & Limitations
GRESO uses a fixed 25% target zero-variance ratio and an 1:2 easy-to-hard allocation, although a more optimal allocation may exist.
Abstract
from arXiv · showhide
Reinforcement learning, such as PPO and GRPO, has powered recent breakthroughs in LLM reasoning. Scaling rollout to sample more prompts enables models to selectively use higher-quality data for training, which can stabilize RL training and improve model performance. However, this comes at the cost of significant computational overhead. In this paper, we show that a substantial portion of this overhead can be avoided by skipping uninformative prompts before rollout. Our analysis of reward dynamics reveals a strong temporal consistency in prompt value: prompts that are uninformative in one epoch of training are likely to remain uninformative in future epochs. Based on these insights, we propose GRESO (GRPO with Efficient Selective Rollout), an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts using reward training dynamics. By evaluating GRESO on a broad range of math reasoning benchmarks and models, such as Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B, we show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation.
1 Introduction
Rollout scaling can improve reinforcement-learning data quality and model performance, but it creates substantial computational overhead. GRESO addresses this by predicting uninformative prompts before rollout using reward-training dynamics, achieving major speedups without accuracy degradation.
- Motivation: Scaling rollouts lets RL selectively use higher-quality data, improving converged model performance but making rollout a major computational bottleneck.The paper frames selective rollout as a way to retain the benefits of additional sampling while reducing its cost.
- Limitations of Existing Methods: Existing static pruning lacks conclusive evidence for improving overall RL efficiency, while online Dynamic Sampling filters uninformative prompts only after rollout.Static methods also overlook that data value changes across models and training stages.
- GRESO: GRESO predicts and skips prompts likely to be uninformative before rollout, using historical reward dynamics and online training states.Its design targets selective rollout without training a separate offline data-selection model.
- Target Prompts: Zero-variance prompts contribute no learning signal because identical rewards across responses produce zero advantages.GRESO targets these prompts as the uninformative cases to filter.
- Results: 2.4× rollout speedup and 2.0× total training-time speedup were achieved while maintaining comparable accuracy across six math benchmarks and three models.The evaluation included Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B.
2 Related Work
Related work improves LLM training through reinforcement learning and data selection. These directions include reward-based reasoning fine-tuning and strategies for pruning or selecting training examples.
- RL for LLM Reasoning: Reinforcement learning evolved from preference alignment into a common approach for fine-tuning LLMs with human-feedback reward signals.The related work places RL within the broader development of helpful, harmless, and honest language-model behavior.
- Data Selection for LLM: Data-selection research seeks to improve LLM training efficiency and effectiveness by pruning or selecting examples for supervised fine-tuning.The passage cites approaches that reduce large training sets to smaller selected subsets.
3 Observation
The paper identifies zero-variance prompts as a source of ineffective GRPO training data and finds that their value is temporally consistent across epochs. This motivates pre-rollout filtering with occasional exploration rather than post-rollout selection or static pruning.
- GRPO and Prompt Value: GRPO normalizes rewards within response groups, so identical rewards yield zero advantages and no learning signal; such prompts are called zero-variance prompts.Prompts with non-identical rewards across responses are termed effective prompts.
- Reduction of Effective Prompts: The effective-prompt ratio can fall to around 20% late in GRPO training, potentially hurting training stability and final model performance.Figure 3 reports that the ratio decreases as training proceeds.
- Dynamic Sampling: Dynamic Sampling improves performance by oversampling and retaining effective prompts, but an 80% zero-variance ratio requires around five times as many rollouts.This wastes rollout computation on prompts that ultimately provide no learning signal.
- Temporal Correlation: Zero-variance prompts show strong temporal correlation: over 90% of current zero-variance prompts were identified as such earlier, while approximately 20% later became effective again.The findings support prediction from historical reward dynamics while preserving some exploration.
- GRESO Motivation: GRESO predicts zero-variance prompts before rollout from training dynamics, unlike Dynamic Sampling, which filters them after rollout.Its probabilistic filtering occasionally resamples zero-variance prompts so potentially valuable prompts can re-enter training.
4 Methodology: GRESO
GRESO performs online pre-rollout filtering using reward dynamics to predict and skip zero-variance prompts, while retaining exploration. It also adapts exploration probabilities and rollout batch sizes to reduce unnecessary computation.
- Pre-rollout filtering: GRESO predicts and skips zero-variance prompts before rollout using reward training dynamics.The method is an online selective rollout algorithm designed to avoid expensive rollouts of prompts expected to provide no learning signal.
- Pre-rollout filtering: Each prompt’s training-dynamics trace records its sampling epochs and response-reward sets for predicting future zero-variance behavior.The prediction target is whether all responses to a prompt will receive identical rewards.
- Pre-rollout filtering: Probabilistic filtering balances exploitation and exploration by occasionally sampling prompts that may become effective again.The filtering probability is based on recent zero-variance rollouts, while exploration preserves opportunities to revisit potentially valuable prompts.
- Adaptive exploration: GRESO automatically adjusts separate exploration probabilities for easy and hard zero-variance prompts toward target zero-variance percentages.The base exploration rate changes each iteration according to whether the observed zero-variance percentage is above or below target; evaluations use a 1% step size.
- Adaptive batching: Adaptive rollout batch sizing reduces computation when only a small number of effective prompts are needed to complete the training batch.GRESO uses the current zero-variance ratio and a safety factor to estimate the rollout batch size needed to collect sufficient valid examples.
5 Experiment
Across six math benchmarks and three models, GRESO matches Dynamic Sampling accuracy while reducing rollout and training costs. Its online selection and adaptive batch sizing improve effective rollout efficiency throughout training.
- End-to-end efficiency: GRESO achieves comparable accuracy to Dynamic Sampling across six math benchmarks while using up to 3.35× fewer rollouts.On Qwen2.5-Math-7B trained on DAPO + MATH, average accuracy is 57.5% versus 57.8%, with rollouts reduced from 13.1M to 6.3M.
- End-to-end efficiency: GRESO reduces rollout time and total training time by up to 2.4× and 2.0×, respectively, compared with Dynamic Sampling.The comparison uses end-to-end training-time breakdowns across three models trained on DAPO + MATH.
- Training dynamics: GRESO maintains a higher effective prompt ratio than Dynamic Sampling during training by filtering zero-variance prompts before rollout.When Dynamic Sampling’s ratio falls to around 20% late in training, GRESO maintains a ratio above 70%.
- Training dynamics: Under the same rollout-time budget, GRESO reaches 2 million effective rollouts in around 25 hours, versus over 40 hours for Dynamic Sampling.This comparison measures effective rollouts generated for training over time.
- Selection dynamics: GRESO selects or skips prompts across epochs, occasionally exploring easy prompts while increasingly skipping prompts that become easier.Some hard prompts become solvable later, whereas others remain unsolved throughout training.
- Ablation study: Adaptive batch size reduces unnecessary computation by rolling out only a smaller batch when few effective prompts are needed.Combined with selective prompt filtering, ABS yields fewer samples per step than Dynamic Sampling and DS with ABS.
6 Conclusion
The paper presents GRESO as a selective rollout algorithm for LLM reinforcement learning. By filtering zero-variance prompts using reward dynamics, it reports faster rollout and overall training without accuracy degradation.
- Conclusion: GRESO selects effective prompts before rollout to avoid sampling overhead from uninformative prompts.The method uses reward dynamics to filter zero-variance prompts.
- Conclusion: GRESO achieves up to 2.4× rollout speedup and 2.0× overall training speedup.These are reported as end-to-end efficiency improvements in the empirical evaluation.
B Limitations
GRESO filters obvious zero-variance prompts but does not assess the remaining prompts’ training value. The paper identifies finer-grained scoring or ranking as future work.
- GRESO filters zero-variance prompts that provide no learning signal.
- Remaining prompts may still be uninformative and contribute only limited training value.
- Future work could extend binary filtering with finer-grained scoring or ranking based on estimated training utility.
- The method is presented as a foundation for more adaptive and efficient reinforcement learning in LLM training.
E Detailed Experimental Setting
Experiments use three math-focused models, two training-data settings, and six reasoning benchmarks. Training uses specified rollout and hardware configurations, with one fixed hyperparameter set across models and tasks.
- Models & Datasets: Experiments use Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B.
- Models & Datasets: Training uses DAPO+MATH and a 30,000-example OPEN-R1 subset.
- Hyperparameters: GRESO uses one fixed hyperparameter set across models and tasks.
- Evaluation: Evaluation covers Math500, AIME24, AMC, Minerva Math, Gaokao, and Olympiad Bench.
F.1 Impact of Targeted Zero-variance Percentage
The study varies the targeted zero-variance percentage and finds nearly identical accuracy across settings. Rollout counts at 0% remain similar to the 25% default because exploration and first-batch oversampling persist.
- Nearly identical accuracy is observed across targeted zero-variance percentages of 0, 25%, 50%, and 100%.
- At a 0% target, rollouts per step remain similar to the 25% setting.
- A minimum exploration rate of 5% prevents the actual zero-variance percentage from reaching 0.
- Oversampling in the first rollout batch adds redundancy and can avoid a second rollout batch.
F.2 Alternative Design: Linear Backoff
Linear backoff skips prompts for future epochs after repeated zero-variance rollouts, but deferred prompts cannot respond directly to changing training dynamics. This produces periodic rather than smooth zero-variance-ratio changes.
- Linear backoff skips a prompt for the next k training epochs after k recent zero-variance rollouts.
- Deferred prompts cannot be directly affected when exploration intensity changes with new observations or evolving training dynamics.
- Linear backoff can produce periodic fluctuations in zero-variance prompt ratios, unlike probabilistic filtering’s smoother dynamics.
F.3 Case study of Filtered Examples
The case study groups MATH prompts by whether GRESO frequently skips or selects them during training. It illustrates these groups with easy, hard, and selected examples.
- Prompt categories: GRESO’s case study categorizes prompts into Frequently Skipped Prompts (Easy), Frequently Skipped Prompts (Hard), and Frequently Selected Prompts.These categories organize examples according to their observed filtering behavior during training.
- Frequently Skipped Prompts (Easy): Additional frequently skipped easy examples ask for a sum of consecutive integers or the final value in an alternating division-and-multiplication sequence.Their stated solutions are 5n+10 and 512, respectively.