Source-linked AI summary
StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation
Yinmin Zhong, Zili Zhang, Xiaoniu Song, Hanpeng Hu, Chao Jin, Bingyang Wu, Nuo Chen, Yukun Chen, Yu Zhou, Changyi Wan, Hongyu Zhou, Yimin Jiang, Yibo Zhu, Daxin Jiang
TL;DR
Existing RL systems must coordinate online generation and training despite resource coupling in colocated designs and bubbles in disaggregated designs. StreamRL separates the stages into streaming services, uses ranker-guided scheduling, and supports flexible heterogeneous deployment. It reports up to 2.66× throughput and up to 1.33× cost-effectiveness in heterogeneous, cross-datacenter settings.
Problem
Colocated RL couples memory-bandwidth-bound generation and compute-bound training to identical resources, while disaggregated frameworks suffer pipeline and long-tail skewness bubbles.
Method
StreamRL separates generation and training into SGS and Trainer, streams completed samples, ranks long-tail outputs, and schedules resources and hardware independently.
Results
Up to 2.66× throughput and up to 1.33× cost-effectiveness are reported versus existing state-of-the-art systems in heterogeneous, cross-datacenter settings.
Takeaways & Limitations
Disaggregation can support flexible resource allocation, heterogeneous hardware, and cross-datacenter RL training when pipeline and skewness inefficiencies are addressed.
Takeaways & Limitations
The asynchronous-training case study verifies feasibility for specific LLM tasks, while generality and theoretical guarantees remain beyond the paper’s scope.
Abstract
from arXiv · showhide
Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs). RL for LLMs involves two stages: generation and training. The LLM first generates samples online, which are then used to derive rewards for training. The conventional view holds that the colocated architecture, where the two stages share resources via temporal multiplexing, outperforms the disaggregated architecture, in which dedicated resources are assigned to each stage. However, in real-world deployments, we observe that the colocated architecture suffers from resource coupling, where the two stages are constrained to use the same resources. This coupling compromises the scalability and cost-efficiency of colocated RL in large-scale training. In contrast, the disaggregated architecture allows for flexible resource allocation, supports heterogeneous training setups, and facilitates cross-datacenter deployment. StreamRL is designed with disaggregation from first principles and fully unlocks its potential by addressing two types of performance bottlenecks in existing disaggregated RL frameworks: pipeline bubbles, caused by stage dependencies, and skewness bubbles, resulting from long-tail output length distributions. To address pipeline bubbles, StreamRL breaks the traditional stage boundary in synchronous RL algorithms through stream generation and achieves full overlapping in asynchronous RL. To address skewness bubbles, StreamRL employs an output-length ranker model to identify long-tail samples and reduces generation time via skewness-aware dispatching and scheduling. Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems, and improves cost-effectiveness by up to 1.33x in a heterogeneous, cross-datacenter setting.
1 Introduction
RL for LLMs alternates online generation and reward-based training, but colocated execution couples distinct workloads. StreamRL uses streaming and skewness-aware scheduling to improve disaggregated execution.
- RL workflow: RL for LLMs typically alternates sample generation with reward-based model training.The generation stage produces samples from prompts, and training updates the model using rewards derived from those samples.
- Motivation: Colocation couples memory-bandwidth-bound generation and compute-bound training to identical resource quantities and hardware types.This coupling limits workload-specific scaling and hardware selection as training expands.
- Motivation: Disaggregation permits separate resource quantities, workload-suitable hardware, and deployment across heterogeneous datacenters.Point-to-point transfer between generation and training keeps communication manageable across datacenters.
- Challenges: Naive disaggregation creates pipeline bubbles because each stage waits for the other at generation, training, and weight-update boundaries.These dependencies leave the allocated resources of one stage idle while the other stage operates.
- Challenges: Long-tail output lengths leave only a few samples active late in generation, severely underutilizing GPUs.The issue is amplified by long chain-of-thought generation in reasoning-model training.
- StreamRL: StreamRL abstracts generation and training into a stream generation service and Trainer, returning completed samples incrementally for concurrent execution.It combines streaming with an output-length ranker and skewness-aware scheduling to reduce generation latency.
- Results: 2.66× throughput and 1.33× cost-effectiveness are reported for StreamRL against existing systems in heterogeneous, cross-datacenter settings.The throughput comparison is against state-of-the-art systems, while the cost-effectiveness result is reported for heterogeneous cross-datacenter deployment.
2 Background and Motivation
RL training combines generation and training workloads whose scaling and hardware needs differ. Disaggregation offers flexibility but introduces pipeline and skewness bubbles that must be addressed for high utilization.
- Background: Each RL iteration generates samples, scores them with rewards, and updates the model during training.Rewards may come from a trained Reward Model or rule-based functions, depending on the algorithm.
- Problems with Colocation: Colocated generation and training share devices despite generation being memory-bandwidth-bound and training being compute-bound.This workload mismatch creates resource coupling.
- Problems with Colocation: Generation latency quickly plateaus as resources increase, whereas compute-bound training benefits substantially more from resource scaling.The generation plateau is linked to memory bandwidth and the communication cost limiting tensor parallelism.
- Problems with Colocation: H20 GPUs provide higher HBM bandwidth and capacity than H800 GPUs while costing about 35% as much, illustrating hardware trade-offs.The passage presents H20 as suitable for memory-bandwidth-bound inference workloads.
- Disaggregation: Disaggregation removes resource coupling and enables workload-specific resources, heterogeneous hardware, and improved cost-effectiveness.These benefits follow from assigning resources and hardware separately to generation and training.
- Challenges for Disaggregation: Naive disaggregation produces pipeline bubbles that leave one stage idle while the other generates samples or waits for updated weights.The serialized stage dependency causes resource waste on both sides of the pipeline.
- Challenges for Disaggregation: Increasing sequence lengths affect generation more strongly than training, partly because larger key-value caches force smaller generation batches.RL workloads also evolve dynamically as output lengths increase during training.
- Challenges for Disaggregation: Skewness bubbles arise when only a small subset of much longer outputs remains, making large decoding batches difficult to sustain.Because decoding needs large batches for high throughput, late-stage GPU utilization declines.
3 StreamRL Overview
StreamRL separates generation and training into services that can use independent resources and hardware, including across datacenters. Streaming, ranker-guided dispatch, and elastic adjustment target pipeline and skewness inefficiencies.
- Architecture: StreamRL deploys Stream Generation Service and Trainer on physically separate resources, potentially across datacenters connected by a point-to-point link.This design supports flexible allocation, heterogeneous hardware selection, and cross-datacenter training.
- Workflow: The workflow first selects resource allocations and parallelization strategies for SGS and Trainer.These decisions are made from the available clusters, models, and algorithm configuration.
- Workflow: Trainer uses weight-update and generation APIs, while streamed samples allow algorithm-specific handling of early results to address pipeline bubbles.SGS returns completed samples incrementally rather than waiting for a whole batch.
- Workflow: An output-length ranker identifies long-tail samples so SGS can dispatch prompts to selected instances and choose their scheduling order.The mechanism targets skewness bubbles by assigning scheduling decisions based on predicted output lengths.
- Elastic adjustment: SGS monitors Trainer execution time and can elastically adjust resources as sequence lengths and workload demands evolve.The goal is to maintain close execution times between generation and training throughout training.
4 Tackle Pipeline Bubbles
StreamRL addresses pipeline bubbles by overlapping generation and training through streaming, while balancing stage resources using profiler-based allocation across single- or cross-datacenter deployments. Its synchronous design uses dynamic batching, and its asynchronous design overlaps weight transmission and generation with training.
- Pipeline overlap: Streaming is used to improve overlap between generation and training in both synchronous and asynchronous RL.The framework presents streaming-supported solutions for each RL type.
- Synchronous RL: Dynamic-batch pipelining sends completed samples immediately to training, enabling training to start when enough samples saturate GPUs and reducing pipeline bubbles.It avoids waiting for fixed mini-batches and reduces idle time except for the first mini-batch.
- Asynchronous RL: Fully asynchronous pipelining overlaps weight transmission and current-iteration generation with next-iteration training, removing weight transmission from the critical path.The design retains one-step asynchronous training semantics while tolerating limited timing fluctuations when average stage speeds are matched.
- Stage balancing: Profiler-based parallel configuration models deterministic generation and training time under a workload and GPU budget to choose execution strategies.For generation, the model incorporates skewness-aware scheduling; workloads can come from recent samples or bootstrapping.
- Stage balancing: Resource allocation supports homogeneous single-datacenter and heterogeneous cross-datacenter deployments for separately placed SGS and Trainer stages.The single-datacenter case searches feasible GPU allocations, while cross-datacenter allocation treats stage GPU counts independently.
- Stage balancing: Dynamic adjustment adds a data-parallel unit to SGS when monitored imbalance reaches a threshold, without interrupting training.The added unit's initialization overhead is described as negligible relative to overall RL training time.
5 Tackle Skewness Bubbles
StreamRL addresses skewness bubbles caused by long-tail output lengths by ranking prompts, separating long-tail and regular samples, and scheduling them to reduce generation time. Its ranker supports relative identification of long-tail samples, while dispatching and LPT scheduling manage heterogeneous workloads under GPU memory constraints.
- Problem: Random dispatching balances samples by output length without considering batch-dependent per-token latency, slowing long-tail decoding.
- Opportunity 1: StreamRL assigns long-tail samples to dedicated instances with smaller batches while grouping regular samples into large batches.
- Opportunity 2: The output-length ranker estimates relative prompt difficulty because exact generation lengths are unavailable before inference.The top 20% of long-tail samples are recalled with nearly 90% accuracy.
- Output Length Ranker: The ranker is trained by supervised fine-tuning on prompt-length pairs and periodically updated online as target-model output lengths evolve.Relative rankings remain reasonably accurate despite absolute-prediction drift, reducing the need for frequent fine-tuning.
- Overhead: Ranker preprocessing is performed offline, so it introduces no online overhead during RL training.
- Skewness-aware Dispatching: Skewness-aware dispatching sorts prompts by estimated length, reserves instances for long-tail samples, and searches allocations using P50/P90 workload estimates.The selected allocation minimizes estimated generation time, subject to batch-size and key-value-cache memory constraints.
- Scheduling Order: Longest-processing-time-first scheduling fills batches in descending estimated length order and provides a 4/3-approximation to optimal completion time.
6 Implementation
StreamRL implements separate inference and training components with optimized communication and memory-management mechanisms. Its infrastructure supports continuous batching, 3D training parallelism, CPU offloading, and zero-copy GPU-direct tensor transfers.
- RL Training Framework: SGS uses a C++ inference engine with optimized CUDA kernels, continuous batching, and prefix sharing, while Trainer uses 3D parallelism and dynamic CPU offloading.
- Tensor-native RPC Library: RL-RPC uses GPU-Direct RDMA for zero-copy tensor transfers between SGS and Trainer, avoiding CPU involvement and serialization overhead.
- Weights transmission: StreamRL includes a network-aware engine that builds topology-optimized broadcast trees for weight transmission across single- and cross-datacenter deployments.
7 Evaluation
StreamRL is evaluated across model sizes, sequence lengths, hardware settings, and RL configurations, showing throughput gains from disaggregated streaming, skewness-aware scheduling, asynchronous training, and heterogeneous deployment. The asynchronous approach achieves the largest throughput gains, while cross-datacenter deployment improves cost-normalized throughput without compromising reported convergence behavior.
- Evaluation Setup: Experiments evaluate StreamRL on Qwen2.5 models from 7B to 72B using real-world datasets and single- and cross-datacenter settings.The testbed includes H800 and H20 clusters, with PPO as the primary RL algorithm and sample throughput measured after warm-up.
- End-to-end Throughput: Compared with verl, StreamRL-Sync achieves a 1.12×–2.12× speedup across maximum sequence lengths and model sizes.Compared with ColocationRL, StreamRL-Sync achieves a 1.06×–1.41× speedup through disaggregated stream generation and skewness-aware scheduling.
- End-to-end Throughput: StreamRL-Async achieves a 1.30×–2.66× throughput improvement by addressing residual long-tail and stage-dependency limitations through one-step asynchronous training.Streaming overlaps part of the pipeline bubbles, while asynchronous training achieves nearly full overlapping.
- Ablation Studies: Skewness-aware scheduling improves throughput by 8% over ColocationRL on the 72B model with the 20K dataset.The ranker identifies long-tail samples and assigns them dedicated compute resources and smaller batch sizes; recall reaches up to 87% for the longest 20% of samples.
- Ablation Studies: Disaggregated streaming adds 15% throughput and asynchronous training adds another 25% in the improvement breakdown.The gains arise from better generation-stage resource allocation and progressively more complete overlap of stage execution.
- Cross-Datacenter and Heterogeneity: Heterogeneous cross-datacenter deployment achieves a 1.23×–1.31× higher throughput normalized by hardware cost.The SGS runs on H20 GPUs while Trainer remains on H800 GPUs; even for 72B models, weight-transfer overhead is under 2% of iteration time.
8 Related Work
Related work covers RL training architectures, LLM inference optimizations, and LLM training optimizations. Prior systems use disaggregation, colocation, or specialized parallelism and scheduling techniques to improve resource utilization and execution efficiency.
- RL training frameworks: Disaggregated RL frameworks partition GPU resources between generation and training, while colocated systems share resources through temporal multiplexing.The cited RL frameworks include NeMo, OpenRLHF, and verl.
- LLM inference optimizations: LLM inference research contributes selective batching, memory-fragmentation reduction, preemptive scheduling, and prefilling-decoding disaggregation techniques.Examples include ORCA, vLLM, FastServe, Splitwise, and DistServe.
- LLM training optimizations: LLM training research develops tensor, data, and pipeline parallelism alongside automated and elastic parallel-strategy optimization.Examples include Alpa, CoDDL, Pollux, ElasticFlow, and MegaScale.
9 Conclusion
The paper revisits disaggregation for RL training and presents StreamRL to address pipeline bubbles and skewness-induced inefficiencies. Experiments report up to a 2.66× speedup over the current state-of-the-art RL framework.
- Conclusion: Disaggregation offers flexible resource allocation, heterogeneous hardware support, and cross-datacenter scalability for RL training.These advantages motivate revisiting the architecture against widely adopted colocation.
- Conclusion: StreamRL addresses pipeline bubbles and skewness-induced inefficiencies in existing disaggregated RL frameworks.The framework is designed to unlock the performance potential of disaggregation.
- Conclusion: 2.66× speedup is achieved by StreamRL compared with the current state-of-the-art RL framework.The authors position this result as encouragement for renewed study of disaggregated RL training.