Source-linked AI summary
RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
Wei Gao, Yuheng Zhao, Dakai An, Tianyuan Wu, Lunxi Cao, Shaopan Xiong, Ju Huang, Weixun Wang, Siran Yang, Wenbo Su, Jiamang Wang, Lin Qu, Bo Zheng, Wei Wang
TL;DR
Synchronous RL post-training can waste GPU time when long-tail responses stall rollout batches, while relaxing synchronization may compromise accuracy and stability. RollPacker uses tail batching to group long-response prompts into designated rounds and adds optimizations across rollout, reward, and training. It reports faster end-to-end training than veRL and RLHFuse for Qwen2.5 models.
Problem
Synchronous RL post-training suffers GPU bubbles from imbalanced response lengths, while relaxing synchronization to reduce idle time can compromise accuracy and stability.
Source: §1 · p. 1, §2.3 · pp. 3–4Method
RollPacker schedules long-response prompts into designated long rounds and combines tail batching with optimizations for rollout, reward, and training.
Source: §1 · p. 2, §9 · p. 13Results
2.03×–2.56× end-to-end training speedup over veRL and up to 2.24× over RLHFuse for Qwen2.5 models.
Source: §1 · p. 2Takeaways & Limitations
RollPacker is designed to expedite synchronous RL training while tail batching preserves accuracy by changing only sample order.
Source: §1 · p. 2, §9 · p. 13Existing overlap of rollout with reward and reference inference masks limited work because those computations typically account for less than 15% of step runtime.
Source: §2.3 · p. 3
Abstract
from arXiv · showhide
Reinforcement Learning (RL) is a pivotal post-training technique for enhancing the reasoning capabilities of Large Language Models (LLMs). However, synchronous RL post-training often suffers from significant GPU underutilization, referred to as bubbles, caused by imbalanced response lengths within rollout steps. Many RL systems attempt to alleviate this problem by relaxing synchronization, but this can compromise training accuracy. In this paper, we introduce tail batching, a novel rollout scheduling strategy for synchronous RL that systematically consolidates prompts leading to long-tail responses into a small subset of rollout steps (long rounds), while ensuring that the majority of steps (short rounds) involve only balanced, short rollouts. By excluding long responses from short rounds and rescheduling them into a few designated long rounds, tail batching effectively reduces GPU idle time during rollouts and significantly accelerates RL training without sacrificing accuracy. We present RollPacker, a system that fully harnesses the benefits of tail batching through holistic optimizations across all three RL stages: elastic parallelism adaptation for rollout, dynamic resource allocation and scheduling for reward, and stream-based training. Empirical results show that RollPacker achieves a 2.03x-2.56x end-to-end training time reduction compared to veRL and up to 2.24x speedup compared to RLHFuse for the Qwen2.5 family of LLMs on up to 128 H800 GPUs.
1 Introduction
Synchronous RL post-training suffers GPU bubbles because long-tail responses keep GPUs waiting, while overlapping other stages cannot fully mask rollout delays. RollPacker’s tail batching groups long-response prompts into designated rounds and combines this scheduling with system optimizations across the RL pipeline.
1 Introduction
- Tail batching consolidates long-response prompts into a few long rounds, leaving most short rounds with balanced, short responses.It changes sample order, preserving training accuracy according to cited algorithmic research.Source: §1 · p. 2
- Short rounds use speculative execution to retain the first P0 prompts and R0 responses to finish; excluded long prompts wait for a dedicated long round.Once the long-prompt queue reaches P0, RollPacker runs those prompts without speculative execution.Source: §1 · p. 2
- RollPacker adapts tensor parallelism to batch size, sequence length, and online response lengths, reducing rollout latency by up to 21.9%.The planner profiles memory footprints and selects a tensor-parallel configuration for each training step.Source: §1 · p. 2
- A reward scheduler pipelines per-sample evaluation with rollouts and adjusts compute budgets to reduce reward and end-to-end latency by an average of 23.9%.Examples include adjusting code-execution timeouts and dynamically sharing GPUs for judge models.Source: §1 · p. 2
- A stream trainer starts training when partial results arrive, reallocates idle rollout GPUs, and defers updates to preserve on-policy semantics.It overlaps gradient computation and optional reference-logit evaluation with ongoing rollouts.Source: §1 · p. 2
- RollPacker achieves 2.03×–2.56× end-to-end training speedup over veRL and up to 2.24× over RLHFuse for Qwen2.5 models.The evaluation trains 7B–32B models on up to 128 H800 GPUs.Source: §1 · p. 2
2 Background and Motivation
Synchronous on-policy RL uses the latest actor weights but can create pipeline bubbles, particularly when rollout responses have highly skewed lengths. The characterization identifies rollout as the dominant stage, while prior overlap and relaxed-synchronization approaches leave a trade-off between bubbles and on-policy guarantees.
2.1 RL for LLM Post-Training
- Synchronous on-policy RL completes rollout before training and updates actor weights only after training, preserving use of the latest parameters but often causing pipeline bubbles.This synchronization can reduce utilization, especially during rollout.Source: §2.1 · p. 3
2.2 Characterization of RL Post-Training
- Across math, code, and LLM-as-a-Judge tasks, rollout takes about 70% of step time; response lengths are long-tailed, with P75 of 755–1.1k tokens and maxima up to 16k.Short-request GPUs can become idle until the longest responses in a batch finish.Source: §2.2 · p. 3
2.3 Existing Solutions and Limitations
- Existing stage overlap leaves long-tail rollout delays insufficiently masked, while relaxing synchronization can compromise accuracy and stability; neither resolves the inefficiency while preserving synchronous RL guarantees.Reward and reference inference typically account for less than 15% of step runtime.Source: §2.3 · pp. 3–4
3 Tail Batching
Tail batching separates long-tail prompts from balanced short rounds without excluding them from training. This reduces rollout costs while preserving the required batch size and on-policy sample distribution.
3 Tail Batching
- Speculative execution over-provisions prompts and responses, retaining the first P0 prompts and R0 responses to form balanced, shorter batches.In the reported setting, η = 1.25, so the short round launches ηP0 prompts and ηR0 responses before accepting P0 × R0 completions.Source: §3 · p. 4
- Aborted long prompts enter a queue and are scheduled together in dedicated long rounds, so no prompt is permanently excluded.Because long prompts are rare, these rounds occur infrequently between the more frequent short rounds.Source: §3 · p. 4
- Tail batching reorders short and long prompts without changing the sample distribution or relaxing synchronization, and its accuracy curves are nearly identical to standard synchronous RL.Figure 4 compares response-length distributions and training-time breakdowns across five rounds: four short rounds and one long round.Source: §3 · p. 4, Figure 4 · p. 4
- 1.48× shorter end-to-end training time; tail batching also reduces the maximum response length in short rounds by up to 8.9× versus veRL.Long responses are deferred to a dedicated fifth round, with outputs capped at the same 16k tokens as the baseline.Source: §3 · p. 5
- The different resource-use profiles of short and long rounds motivate system optimizations tailored to individual RL pipeline stages.With rollout overhead reduced, reward and training make more pronounced relative contributions to training time.Source: §3 · p. 5
4 RollPacker System Design
RollPacker combines adaptive rollout parallelism, resource-aware reward scheduling, and streamed training to improve tail-batching efficiency while preserving on-policy semantics. Its components adapt to the distinct resource demands of short and long rounds.
4.1 System Overview
- RollPacker combines a parallelism planner, reward scheduler, and stream trainer to address distinct bottlenecks in rollout, reward, and training.Offline profiling informs online orchestration of rollout, reward evaluation, GPU reassignment, and synchronized updates.Source: §4.1 · p. 5
4.2 Parallelism Planner
- Short rounds incur up to 1.79× more preemptions than long rounds; increasing TP to 2 reduces short-round duration by 25%.The planner adapts TP per step using preemption counts: a sudden increase can trigger doubling TP, while four steps without preemptions can trigger halving it.Source: §4.2 · p. 6, Figure 6 · p. 6
4.3 Reward Scheduler
- Asynchronous reward evaluation overlaps rollout, while adaptive sandbox timeouts terminate responses exceeding a threshold and assign zero reward.The timeout is relaxed to account for potential CPU contention and avoid overly aggressive cutoffs.Source: §4.3 · p. 6
- RollPacker colocates the judge LLM with the actor using MPS, and offloads most judge layers to host memory to manage GPU contention and memory pressure.Judge parameters are streamed over PCIe alongside GPU activation computation, with the number of offloaded layers adjusted for input sequence length.Source: §4.3 · pp. 6–7
4.4 Stream Trainer
- The stream trainer reassigns rollout GPUs for training as requests complete, overlapping gradient computation with ongoing rollout while reducing idle time.It selects GPUs without splitting tightly coupled communication groups and scales down only when projected peak KV-cache demand fits memory limits after migration.Source: §4.4 · p. 7
- During streaming, gradients are buffered without parameter updates; after rollout, sample-count renormalization makes the final update equivalent to standard on-policy training.Updates are deferred until rollout completes, preserving on-policy semantics.Source: §4.4 · pp. 7–8
5 Implementation
RollPacker is implemented as an extension of existing LLM infrastructure, with separate backends and mechanisms for rollout, reward, and training.
5 Implementation
- RollPacker is implemented in ∼6.6k lines of Python on ROLL, using vLLM for rollout, Ray workers for reward evaluation, and Megatron-LM for actor training.It extends vLLM request interfaces for termination and resubmission, and uses GPU streams for activation computation and parameter transfers.Source: §5 · p. 8
6 Evaluation
The evaluation tests RollPacker on Qwen2.5-7B, 14B, and 32B across math, code, and multi-subject QA, using synchronous RL on up to 128 H800 GPUs. Across the reported settings, it reduces training time relative to veRL and RLHFuse while maintaining validation accuracy.
6.1 End-to-End Evaluation
- RollPacker is 2.03×, 2.22×, and 2.56× faster than veRL, and 1.14×, 1.68×, and 2.24× faster than RLHFuse, across the three evaluated model sizes.Validation scores show no accuracy compromise across model sizes and response lengths; the paper also reports faster early convergence, hypothesized to reflect more balanced response lengths.Source: §6.1 · p. 8, Figure 9 · p. 9, Figure 8 · p. 9
6.2 Performance Breakdown
- Tail batching reaches 2.21× speedup at Qwen2.5-32B/32k; the reward scheduler, planner, and stream trainer add gains in specified settings.The reward scheduler improves speedup from 1.30× to 2.01× at 8k and from 2.21× to 2.48× at 32k; the planner adds 4% (2.48× to 2.52×) at 32k. The stream trainer improves speedup by 20% (2.02× to 2.22×) at Qwen2.5-14B/16k, and gives positive gains in other settings.Source: Table 2 · p. 9, §6.2 · p. 9
- RollPacker reduces maximum response length in short rounds, yielding up to a 7.8× speedup in average rollout time.Rollout times are comparable to veRL in long rounds, where rollout increasingly dominates step time; shorter rollouts lower average step time across rounds.Source: §6.2 · p. 9, Figure 10 · p. 10
6.3 Sensitivity Analysis of Tail Batching
- With η = 1.25 for both prompts and responses, tail batching improves average rollout speed by up to 3.9×.It outperforms fixed-P0 and fixed-R0 settings by up to 1.5× and 1.6×, respectively; at 32k, raising prompt oversampling η from 1.25 to 1.5 increases average rollout time as long rounds become more frequent.Source: §6.3 · p. 10, Figure 11 · p. 10
6.4 Parallelism Planner
- Adaptive tensor-parallel sizing yields 1.9× average speedup over fixed TP size 1 and reduces short-round preemptions by 13.8%.For Qwen2.5-14B, the planner raises TP to 2 and 4 at specified steps; short-round rollout speedups are 1.11×–1.28×.Source: §6.4 · p. 10, Figure 12 · p. 11
6.5 Reward Scheduler
- Asynchronous reward computation is 1.48×, 1.35×, and 1.18× faster than synchronous computation across three LLMs.MPS yields up to 1.25× speedup; pipelined judge execution gives up to 1.4× speedup at 32k. Adaptive timeout reduces unnecessary short-round timeouts and achieves 1.6× average speedup across steps.Source: §6.5 · pp. 10–11, Figure 13 · p. 11
6.6 Performance Analysis of Stream Trainer
- Adaptive GPU scaling achieves 1.08× speedup over no scaling, while asynchronous fetching reduces end-to-end step time by up to 14%.GPU migration overhead is at most 3 seconds, and decoding throughput decreases by less than 1%; adaptive scaling outperforms fixed-trigger baselines. The fetching comparison uses fixed ratios capped at 20%–50% of samples.Source: §6.6 · pp. 11–12, Table 3 · p. 12, Table 4 · p. 12
6.7 Performance Scalability
- At Qwen2.5-14B/16k, RollPacker provides 2.2× throughput over veRL and about 1.5× throughput when resources double.The analysis scales batch size from 128 to 512 with corresponding compute resources, up to 128 GPUs; the smaller-than-linear gain is attributed to increased training time for larger batches.Source: §6.7 · p. 12
7 Discussion
The discussion describes adapting RollPacker to other policy-optimization algorithms and asynchronous systems, and identifies limitations in GPU sharing and parallelism planning.
Extend to Other Policy Optimization Algorithms
- For DAPO, tail batching can retain its oversampling approach while queueing unfinished prompts and excluding zero-reward-variance prompts.The integration caps active requests per LLM instance and follows DAPO’s termination criteria.Source: §7 · p. 12
Extend to Asynchronous Systems
- In asynchronous off-policy training, queued prompts can be processed with one-off pipelines, partial rollouts, or fully asynchronous training.Rollouts for unfinished prompts can continue in the next training step; the reward scheduler and stream trainer can also reduce reward and training overhead, respectively.Source: §7 · p. 12
Potential Limitations
- MPS spatial GPU sharing does not guarantee error isolation, and the parallelism planner currently optimizes TP but not expert parallelism.The paper identifies Green Contexts as future work for fault tolerance and notes that enabling expert parallelism could expand the optimization space.Source: §7 · p. 12
8 Related Works
Related work covers systems for RL post-training, LLM training and inference, and rollout optimization. RollPacker is distinguished by addressing long-tail rollouts in synchronous RL.
RL Post-training Frameworks
- Prior RL frameworks improve workflow orchestration, resource use, scalability, parallelism, or stage overlap; RollPacker targets long-tail rollouts while retaining synchronous training.The passage contrasts RollPacker with approaches including veRL, DistFlow, RLHFuse, Realhf, AReal, StreamRL, and RhymeRL.Source: §8 · p. 12
LLM Training and Inference
- LLM training and inference systems use DP, TP, PP, and CP strategies, alongside attention and KV-cache optimizations.The passage describes these approaches as targeting resource utilization, latency, or memory consumption.Source: §8 · p. 12
Rollout Optimization
- Prompt-quality methods prioritize prompts by reward variance, difficulty, or predicted success, whereas RollPacker focuses on rollout speed and end-to-end latency.The cited examples include DAPO, SPEED-RL, GRESO, and MoPPS.Source: §8 · p. 13
9 Conclusion
The conclusion presents RollPacker as a system for expediting synchronous RL post-training through tail batching and optimizations across the rollout, reward, and training stages.
Conclusion
- RollPacker is designed to expedite synchronous RL post-training.Source: §9 · p. 13
- Tail batching is proposed to alleviate long-tail rollouts and improve resource utilization.Source: §9 · p. 13
- Experiments demonstrate RollPacker’s training-efficiency effectiveness against baselines.The system combines a parallelism planner, reward scheduler, and stream trainer to optimize the rollout, reward, and training stages, respectively.Source: §9 · p. 13