PaperlayerView exact revision on arXiv

Source-linked AI summary

VenusRL: A Fully Disaggregated Agentic RL System with Priority Scheduling and Scalable Interaction

Mingjun Zhang, Yucheng Li, Menghao Zhang, Shuyong Zhu, Ping Zhang, Xiaohe Hu, Jun Chen, Zhixin Wang, Xutong Wang, He Liu, Yanmin Jia, Shengrong Zhu, Peng Sun, Mingjie Zhang, Liming Liu, Jinlong Hou, Yuan Cheng, Yujun Zhang

arXiv:2610.03286v1cs.DC

TL;DR

Agentic RL faces rollout-scheduling and sandbox-memory bottlenecks in multi-turn tool interactions. VenusRL combines priority-aware scheduling with scalable environment management, achieving up to 4.24× training speedup over RollFlash and reducing environmental costs by up to 89% compared with E2B.

  • Problem

    Multi-turn agentic RL needs scheduling that completes enough trajectory groups for training while supporting high environment concurrency without stranding sandbox memory or duplicating state.

    Source: §1 · p. 1, §2.2 · p. 3

  • Method

    VenusRL disaggregates training, rollout, and environment interaction, using predicted completion priority for scheduling and a resource manager for sandbox deployment.

    Source: §1 · p. 2

  • Results

    VenusRL achieves 1.06–4.24× speedup over RollFlash and reduces environmental costs by up to 89% compared with E2B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Agentic RL requires both throughput-oriented rollout scheduling and scalable environment management.

    Source: §1 · p. 2

    Length-prediction reliability improves as trajectory turns accumulate.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

Agentic Reinforcement Learning (RL) trains LLM agents through multi-turn interactions with external tool environments. Its multi-turn nature exposes two system-level bottlenecks unaddressed by existing agentic RL frameworks. First, end-to-end training throughput is constrained by the slowest trajectories to complete, yet optimizing per-GPU utilization alone scatters rollout progress across many groups, delaying the completion of enough groups to unblock the next training step. Second, tool sandboxes are statically over-provisioned by their declared memory ceilings, leaving most physical memory stranded while replicating near-identical state across sandboxes launched from the same prompt. We present VenusRL, a fully disaggregated agentic RL system that addresses both bottlenecks. VenusRL's priority-aware action scheduler uses length-prediction heuristics to identify sample groups whose completion is most likely to unblock the next training step, and pushes them ahead of others across batch admission, KV Cache residency, and cross-worker request orchestration. VenusRL's environment resource manager combines a memory-aware admission threshold with a template-keyed page-sharing pool, packing more sandboxes per node while preserving strict memory isolation via write-protected page table entry aliasing and copy-on-write. Across representative agentic RL workloads, VenusRL achieves up to 4.24x end-to-end training speedup over state-of-the-art baselines and reduces environment cost by up to 89%.

1 Introduction

Agentic RL uses multi-turn tool interactions, creating coupled challenges in rollout scheduling and environment management. VenusRL addresses them with priority-aware scheduling and scalable sandbox management, reporting faster training and lower environment costs.

1 Introduction

  • Agentic RL needs both effective rollout scheduling and scalable environment interaction to support high-throughput multi-turn training.The paper identifies rollout scheduling and environment management as complementary system challenges.Source: §1 · pp. 1–2
  • The priority-aware action scheduler predicts remaining generation length and prioritizes critical groups across GPU execution, KV Cache residency, and request orchestration.Critical groups are unfinished groups most likely to unblock the next trainer batch.Source: §1 · p. 2
  • The environment resource manager uses page-level placement and dynamic admission control, while sharing write-protected snapshot pages across trajectories with copy-on-write.Shared pages are reused among trajectories from the same sample group; modified pages receive private copies lazily.Source: §1 · p. 2
  • VenusRL achieves 1.07–3.26× end-to-end speedup over Slime, 1.06–4.24× over RollFlash, up to 2.67× over ThunderAgent, and up to 89% lower environmental costs than E2B.The system is evaluated on representative agentic RL workloads.Source: §1 · p. 2

2 Background and Motivation

Agentic RL alternates model generation and tool interaction before completed trajectory groups are used for training. Its multi-turn execution creates rollout scheduling and sandbox-memory challenges.

2.1 Agentic RL

  • A prompt is replicated into trajectories that alternate generation and environment interaction; completed groups are rewarded and used by the trainer for policy optimization.Training may be synchronous, or asynchronous with rollout disaggregated from training.Source: §2.1 · pp. 2–3

2.2 Challenges of agentic RL

  • With a configured generation batch size of 64, the active batch averages 76.5% on an online SWE-agent workload, illustrating turn-level GPU underutilization.Trajectory-level scheduling leaves a generation slot occupied while a trajectory pauses for tool execution.Source: §2.2 · p. 3
  • Action-level scheduling overlaps tool execution and generation but can scatter progress across sample groups, delaying the next training step; multi-turn KV Cache eviction also causes recomputation.Cache usage can remain near saturation while cache hit rate drops sharply.Source: §2.2 · p. 3
  • Sandbox memory ceilings strand capacity, while independent instances duplicate state; production profiling reports about 8% average node memory use and over 60% of non-GPU costs from sandboxes.Strong isolation and hard memory limits constrain sandbox density; the cited cost share covers the past three months.Source: §2.2 · p. 4

3 VenusRL Design Overview

VenusRL separates training, rollout generation, and environment interaction into coordinated components. Its scheduler prioritizes progress toward the next training batch, while its resource manager supports denser sandbox deployment.

3 VenusRL Design Overview

  • A data coordinator schedules samples, rollout workers generate asynchronously, and actor workers compute gradients and synchronize updates once a global batch is ready.Updated actor weights are synchronized back to rollout workers before the next iteration.Source: §3 · p. 4
  • A lightweight fine-tuned predictor estimates remaining generation length at turn boundaries, guiding priority across GPU slots, KV Cache residency, and request orchestration.The scheduler identifies high-priority sample groups and prioritizes their samples across these mechanisms.Source: §3 · p. 4
  • A global controller allocates sandboxes from a shared pool, while page-level placement, dynamic admission control, and group-level page sharing target dense deployment without memory overcommitment.The manager hides sandbox deployment details from the RL framework and reuses snapshot pages across trajectories.Source: §3 · p. 4

4 Priority-aware Action Scheduler

VenusRL schedules rollout groups to minimize the time until the next training batch is ready, prioritizing groups most likely to unblock it. It applies priority across GPU execution, KV Cache residency, and worker orchestration.

4.1 Problem Formulation

  • A group becomes training-ready only after all its trajectories finish, so the objective is to minimize the time until the next B groups complete.Action-level scheduling can scatter progress across groups and increase KV Cache contention, delaying that batch.Source: §4.1 · pp. 4–5

4.2 Critical Groups Identification

  • Stochastic response lengths and environment interactions make arrival order a poor predictor of group completion, limiting FCFS scheduling.VenusRL uses a lightweight length-prediction heuristic to identify critical groups whose completion is most likely to unblock training.Source: §4.2 · p. 5
  • VenusRL fine-tunes a lightweight predictor on online trajectory traces and updates group priorities asynchronously, keeping prediction off the training critical path.A trajectory at its last valid weight version is promoted to highest priority to prevent it from being repeatedly demoted until stale.Source: §4.2 · p. 5

4.3 Priority-aware Scheduling

  • GPU scheduling admits low-priority samples opportunistically and preempts them for high-priority samples, while rollout-engine batching prioritizes high-priority requests.Preempted samples retain partial rollout state; the engine also retracts low-priority samples first when decode token space is insufficient.Source: §4.3 · p. 6
  • Trajectory-aware KV Cache references protect pages used by high-priority trajectories, with unused pages evicted before low-reference pages and high-reference pages.When high-priority demand exceeds cache capacity, selected trajectories are demoted to preserve residency for the others.Source: §4.3 · pp. 6–7
  • Runtime request re-dispatch moves pending requests from workers under high-priority KV pressure to workers with lower pressure, mitigating cross-worker priority inversion.The design also migrates prefix KV Cache with requests; the overall scheduling design is described as improving training throughput.Source: §4.3 · p. 7

5 Environment Resource Manager

VenusRL combines node-level sandbox placement and dynamic admission control with group-level page sharing. These mechanisms target denser deployment while managing memory pressure and preserving sandbox isolation.

5 Environment Resource Manager

  • The environment resource manager combines a cluster-level shared sandbox pool with fine-grained per-node placement to increase deployment density.Its two designs are dynamic page-level placement and admission control, plus group-level page sharing.Source: §5.1 · p. 7, §5 · p. 7

5.1 Sandbox Placement

  • Declared sandbox memory ceilings can strand nearly 90% of physical memory, while admitting too many sandboxes risks later page-allocation failures and OOM.Lazy snapshot-backed page allocation creates a gap between physical use and the virtual memory bound, making safe runtime overcommitment the central admission challenge.Source: §5.1 · p. 7
  • A dynamic admission threshold reserves headroom for predicted near-term memory growth, adjusting admission as sandbox memory growth changes.VenusRL predicts cold-start growth from offline profiling of the same template and running growth from a recent time window; on OOM, it migrates the largest-resident sandbox first until memory returns to a safe threshold.Source: §5.1 · p. 8

5.2 Page Sharing

  • A template-keyed, per-group page pool shares read-only snapshot pages across sandboxes, while write-protected mappings and copy-on-write keep modifications private.Sharing is installed at sandbox creation and pages are populated lazily; each sandbox retains its own virtual memory area and page table.Source: §5.2 · p. 9

6 Evaluation

VenusRL is evaluated on representative SWE-agent tasks using Qwen3-4B and Qwen3-32B, against three agentic RL systems. Results cover end-to-end training, scheduler components, and environment resource management.

6.1 Experimental Setup

  • Experiments use Qwen3-4B and Qwen3-32B on OpenSWE tasks, with eight samples per prompt, up to 300 turns, and a 128K-token output limit.The testbed has four servers and 32 GPUs; 16 GPUs are allocated each to generation and training.Source: §6.1 · pp. 9–10

6.2 End-to-End Performance

  • VenusRL achieves 1.07–3.26× speedup over Slime, 1.06-4.24× over RollFlash, and 1.05–2.67× over ThunderAgent across both models and all settings.Even at async_factor=1.0, throughput improves in every setting; all three systems reach nearly identical rewards within comparable steps.Source: Figure 10 · p. 10, §6.2 · p. 10, Figure 11 · p. 10
  • Across the evaluated settings, VenusRL incurs no higher long-tail discard rate than Slime or RollFlash, while RollFlash shows a higher rate.The heatmaps cover trajectory-length percentiles over 100 Qwen3-4B training steps.Source: Figure 12 · p. 10, §6.2 · p. 10
  • On a 400GB node, VenusRL keeps online memory occupancy around 80%–95%, versus native E2B's roughly 8%, and migrates sandboxes in under 3 seconds under residual OOM pressure.At 320 concurrent sandboxes throughput is 0.81×, and at 256 it is 0.51×, showing resource limits can back-pressure rollout.Source: Figure 13 · p. 11, §6.2 · p. 11, Figure 14 · p. 11

6.3 Priority-Aware Action Scheduler

  • Priority-aware scheduling reaches 1.81× and 1.42× speedup at batch sizes 32 and 64; length prediction beats action-level scheduling by up to 95% and FCFS-based priority by up to 58%.The breakdown attributes further gains to KV residency and request migration, including up to 0.48× additional speedup from migration at batch size 64.Source: Figure 15 · p. 11, §6.3 · p. 11
  • The length predictor reaches a Spearman correlation of 0.98, and its latency is much smaller than tool-interaction latency, allowing the overhead to be overlapped.It uses problem specifications and accumulated trajectory history, and can be updated at training checkpoints to mitigate policy drift.Source: Figure 16 · p. 11, §6.3 · pp. 11–12
  • At batch size 64, VenusRL maintains over 95% cache hit rate and achieves up to 2.09× speedup over HiCache and 1.58× over VenusRL without priority demotion.Without priority demotion, the trajectory-aware radix cache reaches only 79% hit rate under batch-size-64 pressure.Source: Figure 17 · p. 12, §6.3 · p. 12

6.4 Environment Resource Manager

  • On a 400GB node, sandbox placement raises density from native E2B's 100 to 743 sandboxes, and page sharing raises it to 905.VenusRL's RDMA migration recovers a sandbox in 2.70 s versus 3.68 s for native E2B; environment-management overhead is reported as negligible.Source: Figure 18 · p. 12, Figure 19 · p. 12, Table 2 · p. 12

7 Related Work

VenusRL's KV cache residency complements existing cache-management techniques. The paper also distinguishes its environment-management scope from prior autoscaling work.

7 Related Work

  • PagedAttention [29] and vAttention [38] address KV allocation, while Mooncake and HiCache extend KV storage across memory and storage tiers.VenusRL's priority-aware KV cache residency is described as complementary to these techniques and integrable with them.Source: §7 · pp. 12–13
  • Prior RL autoscaling work includes DistRS, which uses history-based scaling and timeout-aware mechanisms to adjust resources at batch boundaries.For LLM inference, cited systems [58] [61] optimize resource scaling for large-scale serving workloads.Source: §7 · p. 13
  • VenusRL does not explore dynamic scaling because MicroVM-based environments provide strong isolation; the paper identifies it as future work.Source: §7 · p. 13

8 Conclusion

VenusRL combines priority scheduling with scalable environment interaction for agentic RL. The paper reports faster training and lower environment costs.

8 Conclusion

  • VenusRL is a fully disaggregated agentic RL system with priority scheduling and scalable interaction.Source: §8 · p. 13
  • Its priority-aware action scheduler targets multi-turn rollout scheduling, while its environment resource manager targets scalable and cost-efficient environment interaction.Source: §8 · p. 13
  • VenusRL accelerates training by up to 4.24× and reduces environment costs by nearly 89%.Source: §8 · p. 13

A Analysis on sandbox density bottleneck

The analysis identifies memory as the deployment bottleneck for sandboxes, with CPU and network presenting trade-offs rather than the same capacity limit.

A Analysis on sandbox density bottleneck

B NP-Hardness of Critical-Group identification

Under a stateful execution model, finding an optimal group-completion schedule is NP-hard. The section also explains why priority scheduling does not add off-policy behavior beyond policy staleness and why bounded staleness and promotion protect long trajectories from persistent delay.

B NP-Hardness of Critical-Group identification

  • The decision version—whether a schedule completes at least B unfinished groups by deadline D—is NP-hard, as shown by a polynomial-size reduction from Partition.The reduction makes the two sums equal T, yielding a valid Partition solution.Source: B NP-Hardness of Critical-Group… · p. 16
  • Optimal group-completion scheduling is NP-hard even with two identical unit-capacity GPU slots, single-request groups, known processing times, no environment delays, and sufficient KV-cache capacity.The proof considers state transfer with a positive handoff delay or state reconstruction requiring positive GPU work.Source: B NP-Hardness of Critical-Group… · p. 16

C Effect of priority scheduling on model learning

  • Priority scheduling changes trajectory completion order, not the behavior or optimization policies, so any effect on off-policy degree is indirect through policy staleness.It introduces no new source of off-policy behavior beyond staleness already present in fully asynchronous training.Source: C Effect of priority scheduling on… · pp. 16–17
  • Bounded staleness limits the competing set to O((η + 1)B) trajectories, so long trajectories can continue as that set shrinks.At the last admissible weight version, VenusRL promotes a trajectory to highest priority until completion, preventing further delay by newer short trajectories.Source: C Effect of priority scheduling on… · p. 17

D Impact of OOM Recovered Behaviour

VenusRL controls out-of-memory risk with a workload-specific admission threshold and treats OOM as recoverable. In the production trace, memory bursts were rare and small, and observed OOM frequency was near zero.

D Impact of OOM Recovered Behaviour

  • Admission is allowed only when projected resident-sandbox memory stays below a workload-specific safety threshold.Reserved headroom absorbs memory growth after admission and between control intervals.Source: D Impact of OOM Recovered Behaviour · p. 17
  • With a workload-tuned safety threshold, observed OOM frequency is near zero because aggregate unaccounted memory growth is usually within reserved headroom.Fewer than 0.5% of sandboxes on a node had concurrent bursts within a control interval, each adding at most roughly 100 MiB.Source: D Impact of OOM Recovered Behaviour · p. 17
  • When OOM occurs, VenusRL recovers the sandbox on another node by transferring its full memory state over P2P RDMA and resuming execution there.Recovery takes approximately 2.7 s, substantially less than one RL step; inspected recoveries preserved filesystem contents and running process state.Source: D Impact of OOM Recovered Behaviour · p. 17

E Robustness of the predictor

VenusRL predicts remaining generation lengths to rank trajectories for scheduling, using task specifications and evolving trajectory history. Its predictor is trained at the target sampling temperature, which can be changed through temperature-matched fine-tuning.

E Robustness of the predictor

  • The predictor uses relative ordering of remaining-length estimates to separate long and short samples, and updates estimates at turn boundaries using task specifications and trajectory history.The history includes intermediate reasoning, tool calls, and environment observations; the reported Spearman correlation is 0.98.Source: E Robustness of the predictor · p. 17
  • Training the predictor on trajectories collected at the target sampling temperature avoids distribution shift between training and rollout-time sampling behavior.A different rollout temperature requires fine-tuning on trajectories collected under that temperature.Source: E Robustness of the predictor · pp. 17–18
  • The reported temperature 0.7 reflects the SWE-agent configuration used in the evaluation, not a predictor limitation.The prediction mechanism is not tied to a particular temperature setting.Source: E Robustness of the predictor · p. 18
Loading 2610.03286v1…