PaperlayerView exact revision on arXiv

Source-linked AI summary

PEARL: Adaptive Prefill-Decode Execution with Elasticity for Agentic Reinforcement Learning

Jiaan Zhu, Wei Gao, Youhui Bai, Zewen Jin, Ju Huang, Siran Yang, Jiamang Wang, Lin Qu, Cheng Li

arXiv:2609.35158v1cs.AI

TL;DR

Agentic RL rollout is costly, and elastic resources alone do not resolve the workload-dependent choice of PD mode and ratio. PEARL jointly adapts resources and PD execution with cost-aware reconfiguration. It achieves 2.17–2.79× the throughput of fixed-resource ROLL and improves over RLBoost+ by up to approximately 26.9% for Qwen3-8B and 36.3% for Qwen3-30B-A3B.

  • Problem

    Rollout remains costly despite asynchronous execution and elastic GPUs because scaling, PD configuration, and transition costs depend on workload and available resources.

    Source: §1 · pp. 1–2

  • Method

    PEARL predicts rollout batch completion time from runtime profiles and jointly selects resource allocation and PD execution, applying changes through cost-aware incremental transitions.

    Source: §1 · p. 2, §3.4 · p. 5, §4.2 · p. 6

  • Results

    2.17–2.79× throughput over fixed-resource ROLL, with up to 26.9% and 36.3% improvement over RLBoost+ for Qwen3-8B and Qwen3-30B-A3B, respectively.

    Source: §1 · p. 2, §8 · p. 13

  • Takeaways & Limitations

    Under the evaluated SWE-bench preemptible-GPU traces, jointly adapting resource elasticity and PD execution improves throughput over fixed-resource and elastic baselines.

    Source: §1 · p. 2, §8 · p. 13

    Representative existing systems support either resource elasticity or PD-disaggregated rollout, but the latter use static modes and P/D ratios that need not remain efficient as workload and resources change.

    Source: §3.4 · p. 4

Abstract

from arXiv · show

Multi-turn rollout dominates the cost of agentic reinforcement learning (RL). Asynchronous execution and elastic GPU resources can accelerate this stage, but adding rollout replicas yields diminishing returns while training GPUs remain idle between updates. We observe that effective resource use also depends on the prefill--decode (PD) configuration. Both the choice between colocation and disaggregation and the optimal PD ratio vary with the workload, making resource scaling and PD configuration interdependent. Exploiting this opportunity requires selecting effective configurations and realizing their benefits within transient resource-availability windows despite reconfiguration costs. We present PEARL, an asynchronous agentic RL system that coordinates external resource elasticity, temporary reuse of idle training GPUs, and adaptive PD execution. PEARL maintains a unified GPU--worker--role state and uses runtime profiles to predict rollout batch completion time, accounting for environment-induced reductions in decode concurrency. It selects the PD mode and ratio under the current GPU budget and translates each decision into an incremental transition plan that minimizes worker and role changes. Cost-aware switching and borrowing policies suppress transitions with insufficient expected benefit while ensuring timely return of training GPUs. Our evaluation show that PEARL achieves $2.17$--$2.79\times$ the throughput of fixed-resource ROLL across different LLMs. Compared with RLBoost+, throughput improves by up to approximately 26.9\% for Qwen3-8B and 36.3\% for Qwen3-30B-A3B.

1 Introduction

Agentic RL rollout is costly, and asynchronous execution plus elastic GPUs do not by themselves ensure efficient use of resources. PEARL jointly adapts resource elasticity and prefill–decode execution, with cost-aware transitions, and improves throughput over fixed-resource ROLL and RLBoost+.

1 Introduction

  • Asynchronous overlap and elastic GPU capacity address rollout bottlenecks, but additional rollout resources can yield diminishing returns while training GPUs wait between updates.The paper identifies external resource changes as inter-scale elasticity and temporary reuse of idle training GPUs as intra-scale elasticity.Source: §1 · pp. 1–2
  • The preferred PD mode and ratio vary with workload, while resource arrivals and reclamations change feasible allocations and workload per worker.Colocation can incur recurrent prefill interference; disaggregation removes that interference but adds KVCache transfers and partitions capacity.Source: §1 · p. 2
  • Configuration changes can incur worker initialization, weight synchronization, and unfinished-trajectory handling costs, while borrowed training GPUs must return before the next update.These transition costs can outweigh expected acceleration, so selection must account for both execution performance and switching costs.Source: §1 · p. 2
  • PEARL jointly manages external GPU scaling, temporary reuse of idle training GPUs, and adaptive PD execution.It predicts batch completion time from runtime profiles, selects a mode and ratio under the current GPU budget, and plans incremental transitions while suppressing changes with insufficient expected benefit.Source: §1 · p. 2
  • 2.17–2.37× and 2.43–2.79× throughput over fixed-resource ROLL for Qwen3-8B and Qwen3-30B-A3B, respectively.Against RLBoost+ under the same resource-availability traces, throughput improves by up to approximately 26.9% and 36.3%, respectively.Source: §1 · p. 2

2 Background

Agentic RL alternates rollout and training, with multi-turn environment interaction making rollout costly. Asynchronous execution overlaps the stages, while elastic resources can expand the rollout GPU pool when capacity becomes available.

2.1 Agentic RL and Asynchronous Training

  • Rollout executes the policy to generate samples, and training consumes them to update model weights for the next workflow step.During multi-turn rollout, each response and environment observation contributes to the next prompt until the task is complete.Source: §2.1 · p. 2
  • Multi-turn interaction makes rollout the dominant stage in the evaluated workload because contexts grow and generation and environment execution are sequential within each trajectory.For Qwen3-8B on SWE-bench, rollout accounts for 81.8% of end-to-end execution time even with four times as many GPUs as training.Source: §2.1 · p. 2
  • Asynchronous RL overlaps rollout and training on separate GPU pools, but training GPUs can still idle while awaiting the next trajectory batch.The approach permits rollout to use weights with bounded staleness.Source: §2.1 · pp. 2–3

2.2 Resource Elasticity in Agentic RL

  • Elasticity can add rollout capacity through preemptible cloud instances or spare capacity in shared clusters, which may be reclaimed or reassigned.The rollout cluster must resize as these resources come and go.Source: §2.2 · p. 3

3 Motivation

The motivation section shows that elastic GPU scale-out alone leaves rollout bottlenecks, while PD execution choices and resource allocations depend on workload and available capacity. Existing approaches do not jointly adapt these choices and transitions.

3.1 Underutilization of Elastic Resources

  • 50% more rollout GPUs reduce latency by only 10.3%, and adding 8 more GPUs reduces it by a further 8.9%.With an idealized unlimited rollout pool, rollout still takes 216.3 s versus 163.8 s for training; estimated idle training-GPU time is 24.3%.Source: §3.1 · p. 3

3.2 Efficiency Degradation under PD Colocation

  • Prefill repeatedly interrupts decode despite chunked prefill and prefix caching, limiting effective use of added GPUs under colocation.For Qwen3-30B-A3B, prefill processes 96.68% of tokens in 26.48% of elapsed time, while decode produces 3.32% of tokens in 73.52% of the time.Source: §3.2 · pp. 3–4

3.3 Opportunity of PD Disaggregation

  • At BS32, the best disaggregated configuration beats colocation by 9.7%, but at BS96 colocation wins by 6.1%.The mode trade-off is between reduced interference and KV-transfer overhead plus a partitioned GPU pool.Source: §3.3 · p. 4
  • The best disaggregated PD ratio changes from 4P4D at BS48 to 5P3D at BS64, so no single ratio is best across batch sizes.Allocating more GPUs to prefill leaves fewer decode GPUs under a fixed budget and can make decode the bottleneck.Source: §3.3 · p. 4

3.4 Limitations of Existing Solutions

  • Existing asynchronous systems use fixed resources and colocated PD; elastic systems add resources or reuse training GPUs but retain colocation.Systems supporting PD-disaggregated rollout use static execution modes and P/D ratios, which need not remain efficient as resources or workload change.Source: §3.4 · pp. 4–5
  • PEARL couples workload-aware configuration selection with cost-aware reconfiguration across inter-scale and intra-scale resources to reduce rollout-batch makespan.Transitions must handle in-flight requests, weight synchronization, worker-role changes, resource reclamation, and training resumption; their costs can outweigh optional expansion or borrowing gains.Source: §3.4 · p. 5

4 Design

PEARL uses a shared scheduling and orchestration path to select rollout configurations and enact resource changes. It chooses feasible colocation or disaggregation allocations to minimize predicted trajectory-batch completion time as the GPU budget changes.

4.1.1 Architecture.

  • The Pipeline Scheduler handles rollout requests, trajectories, and weight synchronization, while the Elastic Scheduler responds to training notifications and preemptible-GPU changes.A Runtime Tracker builds a workload profile from request telemetry for the PD Decision Engine.Source: §4.1.1 · p. 5

4.1.2 Workflow.

  • Each elastic event updates the rollout budget, selects a mode and role allocation, plans incremental reconfiguration, and executes that plan.Training-GPU shrink returns borrowed GPUs before training; after training, borrowing is admitted only when the benefit gate and worker readiness conditions permit.Source: §4.1.2 · pp. 5–6
  • Preemptible-GPU expansion initializes workers in the background and admits them after weight synchronization; only affected workers change while others keep serving.Borrowing and expansion are optional when predicted gain exceeds handoff cost, but returning borrowed training GPUs is mandatory.Source: §4.1.2 · pp. 5–6

4.2 Problem Formalization

  • PEARL selects the feasible configuration with minimum predicted batch makespan, the completion time of the last of the B trajectories.The allocation chooses colocation or disaggregation and, when disaggregated, the prefill and decode worker counts under the current worker budget.Source: §4.2 · p. 6
  • A feasible disaggregated assignment must provide enough prefill capacity for the token workload and enough decode-worker memory for the batch KV cache.The stated KV-cache constraint is B·K_seq ≤ N_D·H_max.Source: §4.2 · p. 6

4.3 PD Decision Engine

PEARL predicts rollout batch makespan for colocated and disaggregated prefill–decode execution, then selects a feasible configuration under the current GPU budget. Its switching control accounts for online profile updates and reconfiguration costs.

4.3 PD Decision Engine

  • The PD Decision Engine compares feasible disaggregated role allocations with colocation and selects the configuration with the lowest predicted batch makespan.After each rollout step, the Runtime Tracker refits model parameters from request telemetry; on an elastic signal, the engine enumerates feasible prefill/decode worker splits.Source: §4.3 · p. 6, §4.3.2 · p. 7

4.3.1 Bubble-Aware Makespan Models

  • Colocation models serial prefill and decode on shared GPUs, while disaggregation models concurrent paths and environment-induced reductions in decode concurrency.The disaggregated batch makespan is the longer of the prefill and decode paths plus environment interaction time; its prefill path includes KV-transfer time.Source: §4.3.1 · pp. 6–7

4.3.2 Online Decision and Switching Control

  • Mode-dependent hysteresis admits a switch only when predicted makespan savings exceed a threshold, larger for mode changes than for ratio adjustments.Forced switches remain possible when the current topology becomes infeasible.Source: §4.3.2 · p. 7

4.4 GPU–Worker–Role Orchestration

GPU–Worker–Role Orchestration maps target configurations onto physical workers and turns them into incremental plans. A shared state and minimal-change planning reduce conflicts and reconfiguration overhead across elastic events.

4.4 GPU–Worker–Role Orchestration

  • The orchestration layer maintains a consistent GPU–worker–role mapping and minimizes reconfiguration overhead when placing workers and transitioning to a target configuration.Preemptible GPUs join or leave the dedicated rollout zone, while training GPUs remain in the overlap zone and can be borrowed during idle windows.Source: §4.4 · p. 7

4.4.1 A Unified GPU–Worker–Role State

  • A unified state shared across components records each worker’s fixed GPU binding, role, and activity status, linking target allocations to concrete execution.Active workers serve rollout requests; inactive workers remain alive but serve none, while transitioning workers must drain or finish initialization before serving.Source: §4.4.1 · p. 7

4.4.2 Minimal-Change Plan Generation

  • Minimal-change planning preserves existing bindings and roles where possible, decomposing reconfiguration into worker membership changes and role changes among survivors.Keeping roles up to their target counts and filling deficits with added or reassigned workers achieves the minimum change count; both execution modules consume the same plan format.Source: §4.4.2 · pp. 7–8

4.5 Inter-Scale Cluster Rebuilder

The Inter-Scale Cluster Rebuilder handles preemptible-GPU arrivals and reclamations while rollout continues on unaffected workers. It treats shrinking as urgent and expansion as deferrable until new workers are ready.

4.5 Inter-Scale Cluster Rebuilder

  • The Rebuilder executes add/remove changes for preemptible-GPU events while rollout continues on unaffected workers.Reclaimed GPUs are returned synchronously after dispatch stops; interrupted requests retain partial trajectories for later resumption.Source: §4.5 · p. 8
  • Expansion initializes workers in the background and synchronizes the latest weights after initialization, then admits the group atomically once runtime preparation is complete.If any worker fails to initialize, the all-or-nothing policy cancels the expansion rather than exposing a partial topology.Source: §4.5 · p. 8

4.6 Intra-Scale Toggle Manager

The Intra-Scale Toggle Manager borrows training GPUs during idle windows by time-sharing them with rollout. It gates borrowing on whether predicted benefit exceeds activation and return costs.

4.6 Intra-Scale Toggle Manager

  • The Manager borrows training GPUs during idle windows, time-sharing existing GPUs with rollout rather than adding physical GPUs.Training workers and rollout processes are retained across handoffs while device-state residency changes for the active phase.Source: §4.6 · p. 8
  • Training must release GPU memory when borrowing begins, and overlap workers must drain before borrowed GPUs are returned.Returning the GPUs is mandatory because training cannot proceed until they are released; rollout on dedicated GPUs continues uninterrupted.Source: §4.6 · p. 8
  • Borrowing is activated only when its predicted benefit exceeds the incremental cost of activation and return.The cost-aware gate avoids handoff overhead for windows too short to amortize it.Source: §4.6 · p. 8

5 Implementation

PEARL extends ROLL with elastic rollout management and runtime tracking while retaining worker processes across training and rollout handoffs. It tracks GPU bindings, worker roles, and service states to coordinate those transitions.

5 Implementation

  • Training and rollout workers remain as Ray actors across intra-scale handoffs, with device state released or restored as GPUs move between roles.For disaggregated execution, restoration also refreshes KV-buffer communication metadata; interrupted requests preserve their generated prefix and environment progress when resuming.Source: §5 · pp. 8–9
  • The experimental rollout pool counts 4 reserved GPUs and up to 12 preemptible GPUs, excluding 16 reserved training GPUs.Source: Figure 4 · p. 9
  • New rollout workers asynchronously fetch complete weight snapshots that reuse weights aggregated during normal updates, avoiding additional checkpoints.The controller stores only version, metadata, and references; workers load the snapshot before handing it to the local inference process.Source: §5 · p. 9
  • Runtime profiles aggregate interaction, token, and environment-processing statistics across steps to update workload estimates without assuming stationarity.Exponential moving averages estimate switching costs for toggle-enabled and toggle-disabled steps and update the intra-scale gate.Source: §5 · p. 9

6 Evaluation

Across the evaluated trace segments and workloads, PEARL improves throughput over RLBoost+ and fixed-resource ROLL, with gains varying by resource availability and workload. The evaluation also tests its component contributions, ratio selection, and reconfiguration costs.

6.1 Experimental Setup

  • The experiments train Qwen3-8B and Qwen3-30B-A3B on SWE-bench using GRPO, with 16 training GPUs and 4 reserved plus up to 12 preemptible rollout GPUs.Three two-hour availability-trace segments vary in resource abundance and frequency of changes; ROLL-Large uses fixed rollout pools of 14, 10, or 12 GPUs.Source: §6.1 · p. 9

6.2 End-to-End Performance

  • PEARL improves average throughput over RLBoost+ by 5.8–26.9% for Qwen3-8B and 7.4–36.3% for Qwen3-30B-A3B across trace segments.Against fixed-resource ROLL, it achieves 2.17–2.37× and 2.43–2.79× throughput for the respective models; Figure 5 compares model and segment averages, normalized to ROLL.Source: §6.2 · p. 10, Figure 5 · p. 10

6.3 Ablation Study

  • At 4 rollout GPUs, forced PD disaggregation reduces throughput 20.6% for Qwen3-8B and 30.9% for Qwen3-30B-A3B, while ungated toggle raises it 93.6% and 118.3%, respectively.For Qwen3-30B-A3B, adding the gate raises the gain to 129.5%; for Qwen3-8B, the gated gain is 92.3%, slightly below ungated toggle at 93.6%.Source: §6.3 · p. 10, Figure 7 · p. 11
  • Across 8–16 rollout GPUs, PD disaggregation improves throughput 12.8–15.3% for Qwen3-8B and 7.8–8.9% for Qwen3-30B-A3B; gated gains span 17.9–26.5% and 9.2–27.5%.Figure 7 compares the PD-disaggregation ratio, ungated toggle, and benefit-aware gate against ROLL; gated configurations remain faster than baseline at all tested allocations.Source: §6.3 · pp. 10–11, Figure 7 · p. 11

6.4 Workload Sensitivity

  • PEARL improves throughput over RLBoost+ by 5.3–63.7% across tested rollout batch sizes and by 9.8–26.5% across tested trajectory-length limits.With trajectory length fixed at 32K tokens, RLBoost+ peaks at batch size 64 and PEARL at 96; throughput declines beyond each peak. With batch size fixed at 64, both systems slow as the length limit increases, but PEARL remains faster.Source: §6.4 · p. 11, Figure 8 · p. 11

6.5.1 PD Ratio Selection

  • The PD Decision Engine matches the empirical best ratio at all three dedicated-only resource points and two of three points with overlap enabled.The sole mismatch, with 10 dedicated GPUs and overlap, is approximately 0.4% below the empirical optimum; the comparison may reflect measurement variation.Source: §6.5.1 · p. 12, Table 2 · p. 12

6.5.2 Reconfiguration Cost

  • Inter-scale expansion takes 56–67 s for background preparation and 11–18 s for finalization; intra-scale activation takes approximately 6 s and deactivation 3.2–4.4 s.The benefit-aware gate borrows only when predicted rollout-time savings exceed activation and return costs; the extra full-prefill pass for an interrupted request takes less than 1% of a complete trajectory’s generation time.Source: §6.5.2 · p. 12, Table 3 · p. 12

7 Related Work

Prior work accelerates RL training through asynchronous execution, rollout optimizations, and resource adaptation, while serving systems plan or adjust PD execution. PEARL combines resource elasticity with workload-dependent PD planning for rollout-batch makespan.

Agentic RL training systems

  • Agentic RL frameworks support multi-turn environment interactions, and RollArt, NeMo RL, and Slime also support PD-disaggregated rollout [8].PEARL extends ROLL [35] to changing resource availability while keeping the training allocation fixed.Source: §7 · p. 12

RL with resource adaptation

  • RLBoost and ROSE [7] provide inter-scale elasticity, while SeamlessFlow [34] and BiDiRL [29] reuse training resources for rollout.PEARL considers both forms of elasticity together with PD execution planning and transition costs.Source: §7 · pp. 12–13

LLM serving techniques for RL rollouts

  • Serving systems including Splitwise [23], DOPD [18], AMPD [14], and PPD [17] optimize serving metrics, whereas PEARL plans execution to reduce rollout-batch makespan.PEARL selects execution mode and P/D ratio as external resources scale and training GPUs are borrowed or returned.Source: §7 · p. 13

8 Conclusion

PEARL jointly adapts rollout resources, idle training-GPU reuse, and PD execution roles to minimize batch makespan. On SWE-bench under preemptible-GPU traces, it outperforms fixed-resource ROLL and RLBoost+ in throughput.

8 Conclusion

  • 2.17–2.79× end-to-end throughput of fixed-resource ROLL; throughput over RLBoost+ improves by up to 26.9% for Qwen3-8B and 36.3% for Qwen3-30B-A3B.Evaluated on SWE-bench under preemptible-GPU traces.Source: §8 · p. 13
  • PEARL jointly adapts rollout resources, idle training-GPU reuse, and PD execution roles to minimize batch makespan.Source: §8 · p. 13
  • Its orchestration layer uses the Inter-Scale Cluster Rebuilder for preemptible-GPU expansion and the Intra-Scale Toggle Manager to borrow idle training GPUs.The former uses lazy weight propagation and atomic admission; the latter uses process retention and device-state handoff.Source: §8 · p. 13
Loading 2609.35158v1…