Source-linked AI summary
Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library
Weixun Wang, Shaopan Xiong, Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, Zichen Liu, Haizhou Zhao, Dakai An, Lunxi Cao, Qiyang Cao, Wanxi Deng, Feilei Du, Yiliang Gu, Jiahe Li, Xiang Li, Mingjie Liu, Yijia Luo, Zihe Liu, Yadao Wang, Pei Wang, Tianyuan Wu, Yanan Wu, Yuheng Zhao, Shuaibing Zhao, Jin Yang, Siran Yang, Yingshui Tan, Huimin Yi, Yuchi Xu, Yujin Yuan, Xingyao Zhang, Lin Qu, Wenbo Su, Wei Wang, Jiamang Wang, Bo Zheng
TL;DR
ROLL is designed to address the scalability, flexibility, and orchestration challenges of reinforcement-learning optimization for large-scale learning, including multi-turn agentic interaction. It combines modular worker abstractions, scalable execution strategies, sample-level rollout control, and flexible resource allocation, with empirical evaluations showing improved performance in FrozenLake and WebShop.
Problem
Multi-turn agentic RL for LLMs is difficult because environment execution is slow, reward feedback is hard to obtain, and environment–LLM interactions are complex.
Method
ROLL combines a single-controller architecture, Parallel Worker abstraction, optimized parallel strategy and data transfer, sample-level Rollout Scheduler, specialized workers, and AutoDeviceMapping.
Results
ROLL evaluations report success-ratio gains from 16.8% to 26.0% in FrozenLake and task-success gains from 37% to over 85% in WebShop.
Takeaways & Limitations
ROLL provides an efficient, scalable, and user-friendly framework for accelerating and scaling RL training for LLMs across large-scale and agentic settings.
Abstract
from arXiv · showhide
We introduce ROLL, an efficient, scalable, and user-friendly library designed for Reinforcement Learning Optimization for Large-scale Learning. ROLL caters to three primary user groups: tech pioneers aiming for cost-effective, fault-tolerant large-scale training, developers requiring flexible control over training workflows, and researchers seeking agile experimentation. ROLL is built upon several key modules to serve these user groups effectively. First, a single-controller architecture combined with an abstraction of the parallel worker simplifies the development of the training pipeline. Second, the parallel strategy and data transfer modules enable efficient and scalable training. Third, the rollout scheduler offers fine-grained management of each sample's lifecycle during the rollout stage. Fourth, the environment worker and reward worker support rapid and flexible experimentation with agentic RL algorithms and reward designs. Finally, AutoDeviceMapping allows users to assign resources to different models flexibly across various stages.
ROLL Team
Figure 1 presents ROLL as an efficient, scalable, and user-friendly library for large-scale RL optimization, organized around three primary user groups.
- ROLL provides specific key features for three primary user groups in large-scale RL optimization.
1 Introduction
ROLL addresses the complexity of large-scale and agentic RL training with a modular library spanning distributed execution, resource management, rollout control, and specialized workers. It targets tech pioneers, product developers, and algorithm researchers with distinct scalability, flexibility, and experimentation needs.
- Motivation: RL for LLMs supports alignment, reasoning enhancement, and agentic tool use across diverse AI tasks.
- User groups: ROLL serves tech pioneers with scalable, cost-effective, and fault-tolerant training on heterogeneous large-scale GPU clusters.
- User groups: ROLL gives product developers fine-grained control to route samples to agent environments, reward workers, and devices with minimal engineering effort.
- User groups: ROLL enables algorithm researchers to train on constrained GPU setups and experiment through modular abstractions of RL training pipelines.
- Key contributions: ROLL introduces Parallel Worker, Parallel Strategy, Data Transfer, Rollout Scheduler, Environment Worker, Reward Worker, Resource Pool, and AutoDeviceMapping modules.
- Evaluation: 200B+ MoE models were trained on thousands of GPUs for two weeks without interruption, while benchmarks covered multi-domain RLVR and three agentic RL tasks.
2 Background
RL training for LLMs uses multi-stage, multi-model workflows spanning generation, inference, and training, with distinct computational and resource demands. Multi-turn agentic interaction further introduces slow environments, difficult reward acquisition, and complex environment–LLM interactions.
- Optimization Workflow: Each RL optimization iteration contains generation, inference, and training stages.
- Generation Stage: During generation, the Actor produces responses while interacting with environments, combining compute-bound prefill, memory-bound decoding, and CPU-intensive environment execution.
- Generation Stage: Single-turn tasks mainly use stateless prefill and decoding, whereas multi-turn tasks add repeated environment interaction that can become a performance bottleneck.
- Inference Stage: Inference evaluates generated sequences with Reference, Critic, and Reward models to compute KL penalties, value scores, and quality scores.
- Inference Stage: Verifiable reward computation can require substantial CPU resources for rule-based mathematical and sandbox verification.
- Training Stage: LLM training uses parallelism and memory techniques including DP, TP, PP, CP, EP, ZeRO, activation recomputation, and offloading.
- RL Algorithms: Multi-turn agentic RL is challenged by slow environments, difficult reward feedback, and complex interactions between environments and LLMs.
3 Key Features in ROLL
ROLL’s key features target distinct user needs through scalable infrastructure, configurable workflows, modular experimentation, and agentic RL support. Its architecture combines distributed execution, scheduling, AutoDeviceMapping, resource pools, and sample-wise environment and reward processing.
- Tech pioneers: Tech pioneers receive fast, cost-effective, scalable, and fault-tolerant RL training for large GPU clusters.
- Tech pioneers: ROLL supports training a 200B-parameter model across thousands of GPUs without interruption for about two weeks.
- Product developers: Product developers can customize reward and environment implementations through dedicated Reward Workers and Environment Workers.
- Algorithm researchers: Algorithm researchers can run multiple experiments on constrained GPU resources using memory optimization techniques, including single-GPU setups.
- Algorithm researchers: ROLL abstracts each RL training stage at an appropriate granularity, enabling experimentation with new ideas through a pluggable reasoning pipeline.
- Agentic RL: ROLL supports multi-turn agent-environment interaction, sample-wise environment scaling, and asynchronous parallelized interaction for agentic RL.
- Architecture: The architecture combines a user input layer, distributed executor and scheduler, AutoDeviceMapping, and a resource pool.
4 Framework Design
ROLL combines modular worker abstractions, flexible resource mapping, and sample-level rollout control to support scalable, customizable RL training workflows.
- Architecture: Parallel Workers organize shared resources by RL role, while Reward and Environment Workers support reward computation and multi-turn environment interaction.Reward computation methods include rule-based verification, sandbox execution, and LLM-as-a-Judge.
- Parallel Strategy: ROLL integrates MegatronCore, DeepSpeed, vLLM, and SGLang to accelerate training, inference, and generation with multiple parallelism strategies.Training supports DP, PP, TP, CP, and EP, while DeepSpeed also provides ZeRO2, ZeRO3, and ZeRO-offload.
- Rollout Scheduler: The Rollout Scheduler manages each prompt request individually, dynamically adding or aborting requests according to resource availability and generation progress.This sample-level control addresses worker imbalance caused by long-tail generation times.
- Architecture: ROLL uses a single-controller pipeline and Parallel Worker abstractions to simplify development and experimentation across RLHF, RLVR, and agentic RL workflows.Actor, Critic, Reward, and Environment Workers encapsulate distinct RL roles, allowing component-level customization.
- Resource and Data Management: AutoDeviceMapping enables user-defined sharing and reallocation of devices across model stages, while data-transfer modules reshard data and synchronize parameters.A portion of GPUs assigned to Actor generation can be reallocated to Actor training to improve resource utilization.
- Rollout Scheduler: Sample-level lifecycle control accelerates dynamic sampling through asynchronous reward computation, demand-based request dispatch, and early termination of unnecessary generations.Requests can be aborted after enough prompts produce effective gradients.
5 Experiments
ROLL is evaluated on multi-domain RLVR tasks and three agentic environments, where training improves accuracy, success rates, action quality, and efficiency across models and settings.
- 5.1 RLVR Pipeline: 5,000 math examples, 2,000 code records, and curated general-domain data form the multi-domain RLVR evaluation set.Math data are sampled proportionally by difficulty, while code and general-domain data are filtered for quality.
- 5.1 RLVR Pipeline: Qwen2.5-7B-Base accuracy rises from 0.18 to 0.52, a 2.89× improvement, with gains in math reasoning and code generation.Math accuracy increases from 0.20 to 0.53, while code accuracy increases from 0.13 to 0.41.
- 5.1 RLVR Pipeline: Qwen3-30B-A3B-Base accuracy improves from 0.27 to 0.62, yielding a 2.30 × increase despite greater training fluctuations.Both evaluated models show upward accuracy trends without model collapse.
- 5.2 Agentic RL Tasks: FrozenLake training success rises from 16.8% to a peak of 26.0%, effective actions from 69.1% to 88.8%, and validation success from 12.9% to 23.8%.The model also reaches 23.8% SimpleSokoban validation success after training exclusively on FrozenLake.
- 5.2 Agentic RL Tasks: WebShop success increases from 37% to over 85% on training and validation environments, while average actions decrease from over 7 to around 4.The environment involves iterative searches, product inspection, and purchase decisions within trajectories limited to 50 steps.
6 Conclusion
The conclusion presents ROLL as a framework for optimizing large-scale RL training for LLMs, supported by empirical evaluation of its effectiveness in accelerating and scaling training.
- 6 Conclusion: ROLL combines Parallel Worker, Rollout Scheduler, Parallel Strategy, and AutoDeviceMapping as core modules for large-scale LLM RL training.The framework is designed for tech pioneers, product developers, and RL researchers.
- 6 Conclusion: Extensive empirical evaluation demonstrates ROLL's effectiveness in accelerating and scaling RL training for LLMs.
7 Authors
The listed authors are presented alphabetically within each role.
- 7 Authors: Authors are listed alphabetically within each role.
- 7 Authors: The author list includes Weixun Wang and Shaopan Xiong.
- 7 Authors: The author list also includes Gengru Chen, Wei Gao, Sheng Guo, Yancheng He, Ju Huang, Jiaheng Liu, Zhendong Li, Xiaoyang Li, Zichen Liu, and Haizhou Zhao.