Source-linked AI summary
APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation
Yuzhen Zhou, Jiajun Li, Yusheng Su, Gowtham Ramesh, Zilin Zhu, Xiang Long, Chenyang Zhao, Jin Pan, Xiaodong Yu, Ze Wang, Kangrui Du, Jialian Wu, Ximeng Sun, Jiang Liu, Qiaolin Yu, Hao Chen, Zicheng Liu, Emad Barsoum
TL;DR
Long-tail rollout generation, which accounts for over 90% of RL training runtime, can leave GPUs idle as batches wait for lengthy responses. APRIL over-provisions requests and resumes incomplete rollouts; it improves throughput across algorithms and reports faster convergence and higher final accuracy.
Problem
Rollout generation accounts for over 90% of RL training runtime, and a few long responses can stall batches and leave faster workers idle.
Source: §1 · p. 2Method
APRIL over-provisions rollout requests, stops once the target number completes, and buffers unfinished responses for continuation in later iterations.
Source: §1 · p. 2Results
APRIL improves rollout throughput by 22.5% on average across commonly used RL algorithms, accelerates convergence, and averages 2.1% higher final accuracy across three tasks.
Source: §1 · p. 3, §5 · p. 10Takeaways & Limitations
APRIL offers a practical step toward more efficient RL training and has been deployed across diverse hardware devices.
Source: §5 · p. 10APRIL relaxes strict on-policy training because update batches can include trajectories generated by earlier policy versions.
Source: §2.2 · p. 4, §3.2.2 · p. 5
Abstract
from arXiv · showhide
Reinforcement learning (RL) has become a cornerstone in advancing large-scale pre-trained language models (LLMs). Successive generations, including GPT-o series, DeepSeek-R1, Kimi-K1.5, Grok 4, and GLM-4.5, have relied on large-scale RL training to enhance reasoning and coding capabilities. To meet the community's growing RL needs, numerous RL frameworks have been proposed. However, RL training remains computationally expensive, with rollout generation accounting for more than 90% of total runtime. In addition, its efficiency is often constrained by the long-tail distribution of rollout response lengths, where a few lengthy responses stall entire batches, leaving GPUs idle and underutilized. As model and rollout sizes continue to grow, this bottleneck increasingly limits scalability. To address this challenge, we propose Active Partial Rollouts in Reinforcement Learning (APRIL), which mitigates long-tail inefficiency. In the rollout phase, APRIL over-provisions rollout requests, terminates once the target number of responses is reached, and recycles incomplete responses for continuation in future steps. This strategy ensures that no rollouts are discarded while substantially reducing GPU idle time. Experiments show that APRIL improves rollout throughput by 22.5% on average (at most 44%) across commonly used RL algorithms (GRPO, DAPO, GSPO), accelerates convergence, and achieves 2.1% on average(at most 8%) higher final accuracy across tasks. Moreover, APRIL is both framework and hardware agnostic, already integrated into the slime RL framework, and deployable on NVIDIA and AMD GPUs alike. Taken together, this work unifies system-level and algorithmic considerations in proposing APRIL, with the aim of advancing RL training efficiency and inspiring further optimizations in RL systems. Our codebase is available at https://github.com/RLsys-Foundation/APRIL
1 INTRODUCTION
RL training has become important for advancing large language models, but long-tail rollout generation leaves GPUs idle and limits training efficiency. APRIL recycles incomplete rollouts to reduce this inefficiency while preserving accuracy.
Motivation
- Rollouts account for over 90% of RL training runtime, and a few unusually long responses can stall a batch while faster workers wait idle.The long-tail bottleneck grows more consequential as rollout inference scales.Source: §1 · p. 2
- Existing RL frameworks combine distributed inference, training, and orchestration engines, while the paper identifies rollout generation as the remaining dominant bottleneck.The introduction names OpenRLHF, verl, Areal, and slime among these frameworks.Source: §1 · p. 2
APRIL
- APRIL over-provisions rollout requests, stops when the target number completes, and resumes unfinished responses in later iterations rather than discarding them.Completed rollouts update the model while partial responses are buffered for continuation.Source: §1 · p. 2
Contributions
- 22.5% average throughput improvement across widely used RL algorithms accompanies faster convergence and around 2.1% higher final accuracy.APRIL is also reported as deployable across RL frameworks and NVIDIA and AMD GPUs.Source: §1 · p. 3
2 METHOD
The method formalizes RL training as rollout generation followed by policy optimization, then uses partial rollouts to reduce waiting on long responses. Unfinished trajectories are resumed later, introducing mixed-policy data that the authors report does not destabilize training.
2.1 PRELIMINARY ABOUT GENERAL RL TRAINING
- The preliminary REINFORCE formulation separates sampled rollout actions from the score-function gradient that updates policy parameters toward higher expected reward.The reward may come from a learned model or a reward function.Source: §2.1 · p. 3
2.2 PARTIAL ROLLOUT FOR MITIGATING THE LONG-TAIL PROBLEM
- Synchronous training waits for every rollout in a batch, so faster workers sit idle until the longest sequences finish.This waiting increases idle time and slows the training cycle.Source: §2.2 · p. 3
- APRIL launches more rollouts than needed, stops after N complete, and buffers unfinished requests for prioritized resumption in the next step.Completed rollouts are used to optimize the policy; Figure 2 depicts this process.Source: Figure 2 · p. 3, §2.2 · p. 4
- APRIL's update batches combine newly completed trajectories with continuations that may have been generated under earlier policies.The resulting advantage estimate is computed over a sequence generated by a mixed policy.Source: §2.2 · p. 4
- APRIL's mixed-policy modification does not destabilize training and occasionally yields modest but consistent gains across several common RL algorithms.The cited algorithms include PPO, GRPO, DAPO, and GSPO.Source: §2.2 · p. 4
3 EXPERIMENTS
The experiments test APRIL’s rollout efficiency, accuracy and convergence, off-policy trade-offs, and deployment across GPU platforms. APRIL improves throughput while maintaining comparable or sometimes better accuracy.
3.1 EXPERIMENTAL SETUP
- APRIL requests 512 rollouts for a 256-response target (2× oversampling), stopping after the first 256 complete; the standard setting requests 256 per step.The experiments primarily use 8× NVIDIA H100 or 8× AMD MI300 GPUs, with reported results focused on AMD MI300.Source: §3.1 · p. 5
3.2 THE LONG-TAIL PROBLEM IN ROLLOUT
- Across datasets, more than half of rollout responses terminate within a few thousand tokens, while outliers extend near the maximum limit and stall synchronous batches.The resulting inference idle time creates training bubbles.Source: §3.2 · p. 5, Figure 3 · p. 5
3.2.1 PERFORMANCE - TOKEN THROUGHPUT
- 22.5% overall average rollout-throughput improvement over the non-partial baseline across models, algorithms, and datasets.Table 1 compares the baseline and APRIL across dapo-math-17k, DeepScaler, and DeepMath-103K.Source: §3.2.1 · p. 5, Table 1 · p. 6
3.2.2 CONVERGENCE SPEED AND ACCURACY
- 2.1% overall average accuracy improvement; APRIL does not degrade performance overall and sometimes achieves notable gains.The authors report no cases requiring more than five successive policy versions; possible adverse effects from greater off-policy influence remain an open question.Source: Table 2 · p. 6, §3.2.2 · p. 6
3.2.3 ANALYSIS OF PERCENTAGE OF PARTIAL ROLLOUTS AND ROLLOUT LENGTH
- Approximately 40% of rollouts incorporate sequences generated by preceding policy models in each RL training step.Separately, carried-over tokens account for approximately 40% of tokens in subsequent steps; training remains stable and achieves slightly higher final accuracy.Source: §3.2.3 · p. 6, Figure 4 · p. 7
3.3 HARDWARE-AGNOSTIC
- APRIL was integrated into slime and evaluated on NVIDIA H100 and AMD MI300; AMD deployment also included a HIP implementation of torch memory saver.The memory saver supports co-allocated inference and training by reducing GPU memory pressure.Source: §3.3 · pp. 7–8
3.2.3 ANALYSIS OF PERCENTAGE OF PARTIAL ROLLOUTS AND ROLLOUT LENGTH
- Within-instance response-length standard deviation is typically below 1,500 tokens for GRPO and 2,400 for DAPO, versus 4,000–4,500 tokens at batch level.The authors conclude instance-level completion does not reintroduce a severe intra-group long-tail effect; reported results here are for Qwen-4B.Source: §3.2.3 · p. 7, Figure 5 · p. 8
4 RELATED WORK
The section contrasts synchronous RL’s on-policy stability with asynchronous RL’s utilization gains and staleness risks, then positions APRIL as a way to improve synchronous rollout efficiency while retaining on-policy benefits.
RL Framework Architectures: Synchronous vs. Asynchronous
- Synchronous RL waits for every rollout before updating, preserving fresh on-policy data but leaving resources underused when long responses delay a batch.The section frames this waiting cost as a consequence of long-tail rollout lengths.Source: §4 · p. 9
- Asynchronous RL streams rollouts through a shared buffer and can improve hardware utilization, but rollout staleness may destabilize training or reduce final accuracy.The risk arises when more off-policy rollouts are consumed.Source: §4 · p. 9
Rollouts: On-policy vs. Off-policy
- PPO requires fresh rollouts and substantial model memory, while GRPO removes the explicit value network and DAPO and GSPO extend GRPO.Source: §4 · p. 9
- APRIL reuses partial rollouts as a mild relaxation of strict on-policy training, seeking some off-policy efficiency benefits while retaining on-policy stability.The section presents APRIL as improving synchronous RL without full asynchronous decoupling or managing off-policy staleness.Source: §4 · pp. 9–10
5 CONCLUSION
APRIL reuses partial rollouts to reduce long-tail inefficiency in LLM reinforcement learning. It improves throughput and convergence, with higher final accuracy in some cases.
5 CONCLUSION
- APRIL improves throughput by 22.5% on average across commonly used RL algorithms and accelerates convergence.The conclusion reports these gains alongside reuse of partial rollouts and reduced wasted computation.Source: §5 · p. 10
- APRIL achieves an average of 2.1% higher final accuracy in some cases across three tasks.Source: §5 · p. 10
- Engineering efforts support deployment across diverse hardware devices, and the work presents APRIL as a practical step toward more efficient and sustainable RL training.Source: §5 · p. 10
ETHICS STATEMENT
The work targets RL efficiency for large language models through APRIL, using publicly available datasets and aiming to reduce computational overhead. It notes broader risks from powerful language models and emphasizes responsible use.
ETHICS STATEMENT
- The work focuses on mitigating long-tail inefficiencies in rollout generation for large-scale language-model reinforcement learning.Source: ETHICS STATEMENT · p. 11
- Experiments use publicly available datasets that are widely adopted in prior research, and APRIL is described as framework- and hardware-agnostic.Source: ETHICS STATEMENT · p. 11
- The statement identifies potential misuse of powerful language models for harmful content and emphasizes responsible deployment aligned with ethical AI guidelines.Source: ETHICS STATEMENT · p. 11
REPRODUCIBILITY STATEMENT
The reproducibility materials describe the code and hyperparameter settings, alongside the experimental models, datasets, algorithms, hardware, and rollout configurations.
Reproducibility Materials
- The paper provides its codebase and complete hyperparameter settings to support reproduction of the reported results.Table 3 distinguishes inference hyperparameters on the left from training hyperparameters on the right.Source: REPRODUCIBILITY STATEMENT · p. 11, Table 3 · p. 11
Algorithms, Model, and Datasets
- The experiments use Qwen3-4B and Qwen3-8B with three math-reasoning training datasets, evaluate on AIME-2024, and apply GRPO and DAPO.The training datasets are DAPO-Math-17k, DeepMath-103K, and DeepScaleR.Source: REPRODUCIBILITY STATEMENT · p. 11
Hardware Platform
- The evaluation setup uses a single node with either 8 NVIDIA H100 GPUs or 8 AMD MI300 GPUs, with reported results focused mainly on AMD MI300.Source: REPRODUCIBILITY STATEMENT · p. 11
Hyperparameter Setting
- APRIL requests 512 rollouts using an oversampling batch size of 64, then stops after the first 256 are completed for a 32-instance batch with 8 samples per prompt.Source: REPRODUCIBILITY STATEMENT · p. 12
A.1 USE OF LLMS
ChatGPT-5 was used only to rephrase sentences and correct grammar and typographical errors. It was not involved in the research or its conclusions.
A.2 USE OF LLMS
- ChatGPT-5 was used solely to rephrase sentences and correct grammar and typographical errors.The authors state that it was not involved in research ideation, experimental design, data analysis, or drawing conclusions.Source: §A.1 · p. 17
- The authors take full responsibility for the paper’s content.Source: §A.1 · p. 17
- ChatGPT-5 was not involved in research ideation, experimental design, data analysis, or drawing conclusions.Source: §A.1 · p. 17
A.2 EXPERIMENTAL SETUP
Experiments use Qwen3-4B and Qwen3-8B on three mathematical reasoning training datasets, with AIME-2024 for final evaluation. They test GRPO and DAPO on NVIDIA H100 and AMD MI300 hardware.
A.2 EXPERIMENTAL SETUP
- Experiments evaluate APRIL with GRPO and DAPO using Qwen3-4B and Qwen3-8B across three math reasoning training datasets.The training datasets are DAPO-Math-17k, DeepMath-103K, and DeepScaleR.Source: §A.2 · p. 17
- AIME-2024 is used to evaluate final performance on challenging math reasoning problems.Source: §A.2 · p. 17
- Experiments run on a single node with either 8 NVIDIA H100 GPUs or 8 AMD MI300 GPUs, with results focused mainly on AMD MI300.Source: §A.2 · p. 17
- Each step collects 256 samples from 32 inputs with 8 samples per prompt; APRIL requests 512 and stops after 256 rollouts complete.The oversampling batch size is 64, twice the rollout batch size of 32.Source: §A.2 · p. 17
A.3.1 PERFORMANCE - TOKEN THROUGHPUT
APRIL improves rollout throughput over non-partial rollouts and reports higher average accuracy, while the appendix situates its scheduling approach among synchronous and asynchronous RL designs and rollout optimizations.
A.3.1 PERFORMANCE - TOKEN THROUGHPUT
- 22.5% average rollout-throughput improvement is reported for APRIL over non-partial rollouts across two models, three datasets, and GRPO and DAPO.For Qwen3-8B with GRPO on DeepMath-103K, throughput improves by 49.5%.Source: Figure 6 · p. 18, §A.3.1 · p. 18
A.3.2 CONVERGENCE SPEED AND ACCURACY
- 2.1% average accuracy improvement is reported overall, although some model-algorithm-dataset comparisons show declines.The authors also report stable training and suggest that mildly off-policy rollouts add diversity.Source: Figure 7 · p. 19, §A.3.2 · p. 19
A.3.3 ANALYSIS OF PERCENTAGE OF PARTIAL ROLLOUTS AND ROLLOUT LENGTH
- Batch-level and instance-level response-length variation are analyzed separately using per-iteration standard deviations.Figure 8 defines batch-level variation; Figure 9 defines instance-level variation.Source: Figure 8 · p. 20, Figure 9 · p. 21
A.4 MOTIVATION AND MORE BACKGROUND
- The appendix expands APRIL’s motivation by surveying RL architectures, rollout optimizations, and rollout categories.Source: §A.4 · p. 22
A.4.1 RL FRAMEWORK ARCHITECTURES: SYNCHRONOUS VS. ASYNCHRONOUS
- Synchronous RL waits for every rollout in a batch, so long-tail responses can create idle time; asynchronous RL improves utilization but risks stale, off-policy data.The appendix frames this as a trade-off between training stability and hardware utilization; APRIL adds asynchronous rollout scheduling without full decoupling.Source: §A.4.1 · pp. 22–23
A.4.2 ROLLOUT OPTIMIZATION
- APRIL schedules rollout generation across RL iterations, complementing inference-level methods that optimize token generation or within-batch processing.The appendix contrasts this with length-aware scheduling and pipeline fusion as system-level approaches.Source: §A.4.2 · p. 23
A.4.3 ALGORITHM: ON-POLICY VS. OFF-POLICY
- On-policy methods use fresh rollouts, while off-policy methods can reuse older-policy data but risk instability; APRIL is described as a mild move away from strict on-policy learning.Source: §A.4.3 · pp. 23–24