PaperlayerView exact revision on arXiv

Source-linked AI summary

ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents

Hao Zhang, Mingjie Liu, Shaokun Zhang, Songyang Han, Jian Hu, Zhenghui Jin, Yuchi Zhang, Shizhe Diao, Ximing Lu, Binfeng Xu, Zhiding Yu, Jan Kautz, Yi Dong

arXiv:2603.18815v1cs.AI

TL;DR

Multi-turn agentic RL requires scalable sandboxed rollouts, but existing systems often couple rollout orchestration to training, hindering migration and maintenance. ProRL Agent decouples the full rollout lifecycle behind an HTTP service with extensible, rootless sandbox support, and demonstrates strong performance across software engineering, math, STEM, and coding tasks. The paper’s scope includes the reported experimental setup and leaves richer environments and improved cluster-scale robustness to future work.

  • Problem

    Multi-turn RL requires repeated rollout generation in heterogeneous environments, while existing frameworks tightly couple rollout orchestration with the training stack.

  • Method

    ProRL Agent provides a rollout-as-a-service HTTP infrastructure with extensible sandbox environments, token-in/token-out trajectories, and rootless deployment support.

  • Results

    Experiments show strong performance gains across software-engineering model scales and effective end-to-end RL training on software engineering, STEM, math, and coding tasks.

  • Takeaways & Limitations

    Separating rollout from training provides a modular, scalable, and deployable infrastructure for multi-turn agent RL.

  • Takeaways & Limitations

    The paper leaves richer environments and improved cluster-scale robustness to future work.

Abstract

from arXiv · show

Multi-turn LLM agents are increasingly important for solving complex, interactive tasks, and reinforcement learning (RL) is a key ingredient for improving their long-horizon behavior. However, RL training requires generating large numbers of sandboxed rollout trajectories, and existing infrastructures often couple rollout orchestration with the training loop, making systems hard to migrate and maintain. Under the rollout-as-a-service philosophy, we present ProRL Agent , a scalable infrastructure that serves the full agentic rollout lifecycle through an API service. ProRL Agent also provides standardized and extensible sandbox environments that support diverse agentic tasks in rootless HPC settings. We validate ProRL Agent through RL training on software engineering, math, STEM, and coding tasks. ProRL Agent is open-sourced and integrated as part of NVIDIA NeMo Gym.

1. Introduction

Multi-turn agentic RL makes rollout generation a major systems challenge because trajectories require heterogeneous environments, repeated tool interactions, and variable-latency feedback. ProRL Agent addresses tight trainer–rollout coupling with an open-source rollout-as-a-service infrastructure and validates it across several agentic domains.

  • Motivation: Multi-turn rollouts require repeated interaction with heterogeneous environments, making rollout generation a major bottleneck as task scale and complexity grow.Software-engineering rollouts may include many sequential environment interactions with highly variable latency.
  • Motivation: Tightly coupling agentic rollout with the RL training stack creates conflicting resource requirements and makes systems difficult to migrate and maintain.Rollout is I/O-intensive and asynchronous, whereas training is GPU-intensive and centered on gradient computation.
  • ProRL Agent: ProRL Agent treats the full rollout lifecycle as an independent HTTP service, allowing trainers to submit tasks and retrieve completed trajectories without managing rollout execution.The service covers environment initialization through outcome evaluation and can run separately from the trainer.
  • ProRL Agent: ProRL Agent supports token-in/token-out trajectories, extensible sandboxes for heterogeneous tools and tasks, and rootless deployment on shared HPC clusters.These features target tokenization fidelity, task flexibility, and permission or isolation constraints in HPC settings.
  • Validation: Across 4B, 8B, and 14B models, ProRL Agent achieves strong gains on SWE-Bench Verified and performs well on math, STEM, and coding tasks.The infrastructure was validated through end-to-end RL training and integrated into NVIDIA NeMo Gym.

2. Related Work

Related work addresses scalable agent RL through tool integration, environment abstractions, and rollout scheduling, but rollout infrastructure remains a central bottleneck. ProRL Agent targets this gap with lifecycle decoupling and HPC-oriented sandbox support.

  • Multi-turn RL for LLM Agents: Multi-turn RL extends reinforcement learning to agents that interact with external environments over long horizons.These settings build on earlier RL progress in mathematics, logic, and coding.
  • Multi-turn RL for LLM Agents: Scalable infrastructure for generating, managing, and evaluating agentic rollouts has become a major bottleneck for RL training and deployment.ProRL Agent addresses this by separating the rollout lifecycle from the training stack.
  • Agent RL Infrastructures: Existing agent RL frameworks support diverse tools, flexible environments, and rollout scheduling, yet rollout orchestration remains embedded across these systems.The orchestration includes environment lifecycle management, tool execution, trajectory collection, and evaluation.
  • Agentic Sandbox Environments: Agentic sandbox platforms provide isolation, reproducibility, and security but often rely on Docker, whose daemon and privilege assumptions are problematic on shared HPC clusters.This creates a deployment trade-off between separate evaluation infrastructure and broader shared-cluster compatibility.

3. System Design: Training–Rollout Decoupling

ProRL Agent organizes rollout execution into sandbox, server, and trainer components connected through an HTTP interface. Its server uses asynchronous lifecycle stages and manages interchangeable LLM backends.

  • Sandbox Environment: The sandbox executes each rollout in a SingularityRuntime container through AgentHandler lifecycle methods for setup, agent execution, and reward scoring.The methods are init(), run(), and eval().
  • ProRL Agent Server: The ProRL Agent Server manages rollouts through an asynchronous INIT → RUN → EVAL pipeline with independent worker pools.It also maintains an LLM backend pool supporting dynamic registration and checkpoint swapping.
  • RL Trainer: RL trainers interact with the server solely through HTTP, submitting jobs and receiving completed trajectories and rewards for policy updates.The interface supports job submission, backend management, and cancellation.

3.1. Overview

ProRL Agent separates rollout orchestration from policy training through a standalone HTTP service that executes tasks and returns trajectories with rewards. This separation supports independent development, deployment, scaling, and task integration.

  • Overview: Agentic RL samples require sandbox setup, tool execution, and outcome scoring, making rollout logic substantially more complex than single-step generation.Prior systems typically embed this task loop, execution environment, and RL algorithm within the training loop.
  • Overview: The standalone ProRL Agent Server accepts a task instance, executes the full rollout, and returns a completed trajectory with a reward signal.The training framework remains agnostic to the underlying RL infrastructure through this interface.
  • Overview: Rollout and training nodes can be developed, deployed, and optimized independently for greater throughput.The architecture separates rollout-side and training-side optimization.
  • Overview: Adding a new task requires implementing a handler plugin on the rollout server without changing training code.Agent scaffolds can likewise be modified or replaced without affecting the training infrastructure.
  • Overview: The overall architecture combines extensible sandbox environments, a rollout-scheduling server, and an RL training backend.These are the three main components illustrated in Figure 2.

3.2. Extensible Sandbox Environments

ProRL Agent provides extensible, rootless-HPC sandbox environments by separating task-specific lifecycle logic from the server core. It also reduces tool-execution overhead across multi-turn rollouts through optimized shell, Python, and interprocess communication backends.

  • Sandbox architecture: The sandbox layer combines a pluggable task abstraction with an HPC-compatible, rootless container runtime for isolated agent execution at scale.This supports heterogeneous task environments without requiring privileged Docker daemons.
  • Task abstraction: AgentHandler encapsulates task-specific logic through init, run, and eval stages that provision environments, execute agent loops, and compute rewards.Error callbacks and final-result serialization preserve well-formed outputs when rollouts fail partway through.
  • Task abstraction: Each task domain subclasses AgentHandler and registers under a unique name, allowing the server to dispatch jobs by matching their task registry entry.The server invokes the registered lifecycle methods in order after reading the task instance.
  • Efficient tool backends: Efficient Bash replaces tmux-mediated shell execution with a direct pseudo-terminal to reduce shell-command round-trip latency.Shell execution is frequent across code-centric agentic tasks, making this optimization relevant to rollout throughput.
  • Efficient tool backends: Persistent IPython kernels retain variables, imports, and helper functions across agent steps, avoiding repeated setup code during multi-step Python execution.This preserves state across successive tool calls.
  • Efficient tool backends: Unix domain sockets replace TCP loopback for tool communication, reducing networking overhead on the repeated action-observation path.The execution server runs inside the container and returns observations after carrying out shell, file, or Python actions.
  • Efficient tool backends: Together, the three tool-backend optimizations prevent tool execution from becoming the throughput bottleneck as concurrency reaches hundreds of parallel agents.The optimized backends target shell execution, persistent Python interaction, and local process communication.

3.3. ProRL Agent Server

ProRL Agent Server decouples rollout phases into asynchronous worker pools and exposes runtime control through an HTTP management API. This design overlaps heterogeneous stages, supports dynamic LLM backend management, and provides lifecycle controls for jobs and resources.

  • Asynchronous pipeline: The server separates initialization, agent execution, and evaluation into independent worker pools so phases with different resource demands can overlap across jobs.Initialization is I/O-bound, execution is inference-bound, and evaluation can range from milliseconds to minutes.
  • Management API: A lightweight HTTP management API supports job submission, cancellation, LLM-backend registration, and server lifecycle control without coupling trainers to server internals.The API includes endpoints for adding or clearing backends, processing and canceling jobs, starting or stopping the server, and querying status.
  • Asynchronous pipeline: Three thread pools drain stage-specific queues concurrently, with worker loops timing active phases, invoking handlers, storing results, and cleaning up runtimes before evaluation.Completed non-evaluation stages enqueue jobs for the next stage, while evaluation signals the waiting HTTP handler.
  • LLM backend management: The server manages pools of LLM backends, allowing trainers to register or deregister servers and swap outdated policy checkpoints during training.This supports changing inference capacity and model weights while rollouts are running.
  • LLM backend management: A min-heap routes each task to the backend with the lowest assignment count, while keeping all calls from that task on one backend to maximize prefix-cache reuse.The assignment counter is incremented once per task, and heap updates provide round-robin-like balancing under a lock.
  • Trajectory representation: Token IDs and per-token log-probabilities remain canonical throughout the pipeline, preventing re-tokenization drift between rollout generation and training.Messages propagate input_ids, output_ids, and logprobs unchanged across multi-turn trajectories.
  • Job lifecycle and cancellation: Phase-aware timers exclude inter-stage queue waiting, while cancellation discards queued jobs, stops active tasks, closes containers, and unblocks waiting handlers.These controls ensure timeout budgets reflect execution time and release resources promptly for stale or incomplete rollouts.

3.4. Connecting to RL Trainers

ProRL Agent connects rollout infrastructure to multiple RL trainers and improves agentic DAPO execution through hierarchical backend assignment and asynchronous sampling. Its optimized implementation reduces worker idle time and improves hardware utilization.

  • Connecting to RL trainers: Rollout-level decoupling enables ProRL Agent to interface with multiple RL trainers, including VeRL and NeMo RL.The server exposes rollout functionality independently from the training stack.
  • Efficient asynchronous task scheduling: The RL client assigns nearby LLM servers preferentially to same-node ProRL Agent servers, then distributes remaining servers round-robin for global balance.The first phase reduces network latency through physical-node locality.
  • Efficient DAPO: DAPO filters Zero-Variance Prompts, but naive batch-by-batch sampling is inefficient for long, asynchronous agent rollouts because it creates idle time, redundant rollouts, and discarded partial work.The trainer repeatedly requests batches until enough informative prompts are collected.
  • Efficient DAPO: The optimized DAPO implementation replenishes queues continuously, terminates excess jobs after reaching the informative-prompt target, and carries unfinished jobs into the next iteration.These mechanisms preserve partial progress while limiting redundant rollout generation.
  • Efficient DAPO: The optimized implementation significantly reduces worker idle time and improves overall hardware utilization compared with the baseline.Figure 3 compares the DAPO implementations at n=4.

4. Experiments

Experiments evaluate ProRL Agent across software engineering, STEM, math, and coding tasks, finding consistent performance improvements and scalable rollout throughput. Ablations examine system components that improve throughput through GPU utilization or action-time reductions.

  • Main Results on Software Engineering: ProRL Agent consistently improves SWE-Bench Verified performance across Qwen3 4B, 8B, and 14B models, with nearly 2× improvement for the 8B model versus SkyRL-v0.The evaluation uses a 293-instance SWE-Gym subset, with thinking mode enabled for the 8B and 14B models.
  • Generality Across Agent Domains: STEM-agent mean reward rises from approximately 0.2 to around 0.65 after 60 training steps, showing steady improvement during RL training.The upward-smoothed curve shows no signs of saturation in the reported interval.
  • Generality Across Agent Domains: Math-agent Pass@1 on AMC increases from 0.4 to approximately 0.9 during RL training.The reported improvement accompanies learning to use IPython and other tools for mathematical reasoning.
  • Generality Across Agent Domains: Code-agent Pass@1 on Codeforces increases from 0.23 to approximately 0.42 during RL training.The agent uses file editing, Bash execution, and IPython tools for program synthesis and verification.
  • Ablation Study: Table 3 evaluates Load Balancing, Efficient Bash, and Stale job Cleanup by measuring DAPO rollout throughput after removing each component.The table caption states that each component improves throughput through higher GPU utilization or reduced shell-command action time.
  • Scalability: Rollout throughput increases nearly linearly with the number of compute nodes, indicating effective use of additional resources with minimal scaling overhead.The measurement is conducted on software engineering tasks, where rollout generation is often the main system bottleneck.

5. Conclusion

ProRL Agent is an open-source, scalable infrastructure for HPC-native multi-turn agent training that separates rollout from policy training. Experiments across four agent domains show effective end-to-end RL training and strong gains across multiple model scales.

  • Conclusion: ProRL Agent separates the entire rollout lifecycle from policy training to improve modularity, scalability, and deployability for agent RL.The infrastructure is open source and integrated into NVIDIA NeMo Gym.
  • Conclusion: Experiments across software engineering, STEM, math, and code agents demonstrate effective end-to-end RL training with strong performance gains across multiple model scales.The authors leave richer environments and improved cluster-scale robustness to future work.

A. Appendix

The appendix contrasts existing agent-RL infrastructures with ProRL Agent’s rollout-level separation. Existing systems keep rollout control or lifecycle management within the training stack, whereas ProRL Agent separates the lifecycle from training.

  • Architectural Comparison: ProRL Agent separates environment management through reward computation from GPU-intensive training, decoupling I/O-intensive rollout from training.This architecture is presented as rollout-level separation of the full agentic rollout lifecycle.
  • Architectural Comparison: SkyRL-Agent keeps multi-turn rollout control inside a single CPU training driver despite using remote inference and environment containers.The driver runs concurrent trajectory-generation coroutines and coordinates the agent loop.
  • Architectural Comparison: Agent Lightning keeps its store and rollout workers within the trainer’s process tree, so rollout lacks an independent service lifecycle.Terminating the training process also stops the store and disrupts rollout workers.
  • Architectural Comparison: VeRL-Tool offloads tool execution to a CPU environment service but retains agent-loop and trajectory-collection control inside the trainer.The separate environment service does not decouple rollout orchestration from training.
  • Architectural Comparison: rLLM places the agent loop, environment management, and trajectory orchestration in one modified veRL driver without an independent rollout service or persistent trajectory buffer.Its rollout lifecycle cannot survive independently of the training driver.
  • Architectural Comparison: GEM embeds environment execution in the training process through in-memory Python objects, direct env.step() calls, and same-address-space threads.A single driver orchestrates both rollout and training while accessing GPU workers through Ray RPC.
Loading 2603.18815v1…