Source-linked AI summary
Reshaping Rollout Workloads for Asynchronous RL Post-Training on Heterogeneous Accelerators
Jiahui Li, Hao Nie, Yibo Zhu, Pengjin Xie, Yu Zhou, Xiaolong Zheng, Liang Liu, Huadong Ma
TL;DR
Heterogeneous rollout pools must balance aggregate decode throughput with fast trajectory completion as workload mixes evolve, and migration-benefit estimation depends on future placements. CadenceRL reshapes workloads to specialize workers, improving throughput and reducing P95 trajectory latency compared with the strongest baseline.
Problem
Large decode batches improve aggregate throughput but slow individual trajectories, and evolving workloads make specialization difficult to sustain while migration-benefit estimates depend on future placements.
Source: §1 · p. 2, §3.2 · p. 5Method
CadenceRL reshapes workloads through pacing and concentration, and stages accumulated KV prefixes before selecting a destination.
Source: §1 · p. 2, §3.3 · p. 5, §4.3 · p. 8Results
Up to 48% higher aggregate decode throughput and up to 64% lower P95 trajectory latency than the strongest baseline on heterogeneous configurations.
Source: §1 · p. 3Takeaways & Limitations
Heterogeneous scaling is composable: adding high-bandwidth accelerators reduces tail latency, while adding cost-efficient accelerators increases throughput, without manual routing configuration.
Source: §1 · p. 3, §8 · p. 121DORA∗’s reallocation model assumes homogeneous hardware, so it is excluded from heterogeneous comparisons.
Source: §6.2 · p. 10
Abstract
from arXiv · showhide
Reinforcement learning (RL) post-training increasingly relies on long-horizon, multi-turn rollouts. As post-training jobs outgrow a single cluster, rollout pools assembled across clusters introduce hardware heterogeneity. Rollout scheduling must serve two stakeholders: the hardware needs high aggregate decode throughput, while each trajectory needs to finish quickly. The tension arises from the memory-bandwidth-bound nature of autoregressive decoding. A large active batch amortizes weight reads for high throughput but leaves each trajectory a smaller bandwidth share and a longer completion time. The scheduling objective is therefore specialization, letting different workers serve different roles. Heterogeneous hardware further enables this specialization. High-bandwidth accelerators favor long-context work, while cost-efficient accelerators sustain large batches. Workload evolution makes this specialization difficult to sustain, and dynamic reassignment faces a circular dependency because a move's benefit depends on subsequent placement decisions. We present CadenceRL, which bypasses this dependency through structural workload reshaping rather than per-move benefit estimation. Pacing replaces long-context trajectories with shorter ones, providing a structurally positive transformation that sustains large active batches for high throughput. When accumulated staleness demands faster completion, concentration directs the residual long-context tail onto high-affinity workers. Late-bound KV preparation stages accumulated prefixes before a destination is selected. On heterogeneous rollout pools, CadenceRL improves decode throughput by up to 48% and reduces P95 trajectory latency by up to 64%. Adding high-bandwidth accelerators reduces tail latency, while adding cost-efficient accelerators increases throughput, without manual routing configuration.
1 Introduction
Heterogeneous rollout scheduling must balance aggregate decode throughput against fast trajectory completion as workload mixes evolve. CadenceRL reshapes workloads to specialize workers without estimating each migration’s benefit.
1 Introduction
- Large active batches amortize model-weight reads for high throughput but give each trajectory a smaller bandwidth share and slower decoding.Because decoding is memory-bandwidth-bound, different workers must serve throughput and fast-completion objectives rather than compromise on each worker.Source: §1 · pp. 1–2
- High-bandwidth accelerators suit long-context work, while cheaper, lower-bandwidth devices can sustain large batches of shorter contexts.These complementary strengths make hardware diversity a basis for specialization.Source: §1 · p. 2, Figure 1 · p. 2
- Workload evolution undermines fixed specialization, while migration-benefit estimates depend on unknown remaining lengths and future destination batch compositions.Existing approaches simplify the problem through homogeneous-pool assumptions, stage-level capacity changes, or static device specifications, foregoing within-pool specialization.Source: §1 · p. 2
- The decode-throughput landscape rewards reshaping toward specialization, letting CadenceRL act without per-migration benefit estimation.Throughput workers sustain large active batches, while tail workers dedicate concentrated decode resources to long-context trajectories.Source: §1 · p. 2
- CadenceRL uses pacing while fresh work remains and concentrates the residual long-context tail onto high-affinity workers when fresh work is exhausted.Pacing replaces long-context work with shorter work; late-bound KV preparation stages prefixes before a destination is selected.Source: §1 · p. 2
- Up to 48% higher aggregate decode throughput and up to 64% lower P95 trajectory latency than the strongest baseline; adding different accelerator types improves different performance axes.Adding high-bandwidth accelerators reduces tail latency, while adding cost-efficient accelerators increases throughput, without manual routing configuration.Source: §1 · p. 3
2 Background
Asynchronous RL post-training separates rollout from training, while policy versioning ties worker progress to trajectory completion. Workload variability and hardware-dependent throughput make static placement increasingly mismatched to runtime conditions.
2.1 RL Post-Training Workflow
- Asynchronous post-training separates trajectory collection from policy updates, with rollout and training running in independently provisioned pools.Workers adopt newer policy weights only after resident trajectories complete or depart, so long-running trajectories can hold back advancement and accumulate staleness.Source: §2.1 · p. 3
2.2 Workload Evolution
- Trajectories from the same prompt under the same policy can differ in response length by an order of magnitude, contributing to long-tailed response-length and interaction-latency distributions.Runtime model–environment interactions determine response length, turn count, and wall-clock duration, so initially balanced assignments may diverge.Source: §2.2 · p. 3, Figure 3 · p. 3
2.3 Hardware-Workload Interaction
- H800 sustains the highest throughput across context lengths, while one-time placement often leaves workers far from their controlled-profile throughput as workloads evolve.Effective throughput depends on batch composition as well as mean context length, motivating dynamic workload redistribution.Source: §2.3 · pp. 3–4, Figure 5 · p. 4
3 Design Principles
CadenceRL seeks throughput and fast trajectory completion through specialization: different workload compositions, and complementary hardware, serve those goals. It replaces per-migration benefit estimates with workload transformations that have structurally positive gains.
3.1 Specialization in Heterogeneous Pools
- Throughput workers sustain large batches while tail workers concentrate resources on long-context trajectories; heterogeneous hardware aligns high-bandwidth devices with long contexts and cost-efficient devices with throughput.The roles emerge from workload composition rather than requiring every worker to optimize the same metric.Source: §3.1 · p. 4
3.2 Circular Dependency in Benefit Estimation
- A migration’s predicted gain depends on unknown remaining trajectory lengths, future destination co-residents, and migration costs, creating a circular dependency between evaluation and placement.The paper proposes acting on transformations whose gains follow from workload structure instead of estimating each move’s benefit.Source: §3.2 · p. 5
3.3 Guaranteed Gain from Workload Reshaping
- Replacing long-context residents with shorter trajectories keeps active batches larger and slows context growth, sustaining higher-throughput states for longer.The gain comes from the combined long-out, short-in transformation, not from either departure or placement alone.Source: §3.3 · p. 5, Figure 6 · p. 5
- The directional long-out, short-in transformation differentiates workers into throughput and tail roles without explicitly assigning those roles.Workers making replacements become throughput workers; those absorbing displaced long-context trajectories become tail workers.Source: §3.3 · p. 5
3.4 The Throughput–Latency Trade-off
- Pacing can raise pool throughput while speeding each resume window, but it may lengthen the wait before a displaced trajectory resumes.A generous staleness budget extends pacing, increasing throughput but prolonging tail displacement; a strict budget exhausts fresh short work sooner and accelerates tail completion.Source: §3.4 · pp. 5–6
- After fresh short work is exhausted, concentration directs the residual long-context tail to workers suited to it, shifting the guaranteed gain from sustaining throughput to accelerating tail drain.Neither pacing nor concentration requires per-migration benefit estimation.Source: §3.4 · p. 6
4 CadenceRL Design
CadenceRL reshapes a shared rollout pool by separating trajectory departures from placements and using headroom to sustain specialization. When fresh work runs out, it concentrates the remaining tail, with late-bound KV preparation supporting placement across workers.
4 CadenceRL Design
- CadenceRL continuously reshapes workload distribution so throughput and tail roles emerge; pacing operates while fresh work remains, and concentration takes over when it is exhausted.Both mechanisms move trajectories through an intermediate pending state.Source: §4 · p. 6
4.1 Separating Departure from Placement
- A trajectory departs to a policy-version-indexed pending set before its destination is chosen, separating the direction of eviction from later placement.The pending set also provides backpressure: its growth signals that evictions are outpacing placements.Source: §4.1 · p. 6
4.2 Sustaining Specialization
- During pacing, workers below the headroom floor evict their longest resident, then pending trajectories are placed shortest-first on feasible workers with the smallest post-placement headroom.Tight fit directs short work toward nearly full workers and reserves larger headroom gaps for long trajectories, amplifying workload-driven role differentiation; in heterogeneous pools, faster hardware consequently receives relatively longer contexts.Source: §4.2 · p. 7
4.3 Concentrating the Residual Tail
- When the admission budget is exhausted, concentration drains the residual tail by selecting low-affinity sources and placing long trajectories on high-affinity targets.Pending-set growth throttles displacement when placements cannot absorb it; concentration uses the headroom floor without the pacing decode reserve.Source: §4.3 · p. 8
4.4 KV Preparation for Late-Bound Placement
- Late-bound KV preparation snapshots departed state to host storage, stages selected prefixes to relay nodes before destination selection, then loads them through parallel fan-in.Prepared transfer applies between compatible workers in the same cluster; cross-cluster or cross-type moves use token transfer and destination-side re-prefill.Source: §4.4 · pp. 8–9
5 Implementation
The prototype separates cross-worker scheduling control from local inference scheduling and uses two parameters to govern scheduling.
5 Implementation
- The prototype uses vLLM for inference and Steptron for training; CadenceRL manages resident membership and pending trajectories while vLLM handles local batching and memory management.Cross-worker placement decisions belong to CadenceRL’s control plane.Source: §5 · p. 9
- The decode reserve δ controls pacing aggressiveness, while admission budget parameter η controls how far new work may run ahead of training consumption.The credit model bounds steady-state in-flight samples to approximately (η+1) iteration-batches; η=1 corresponds to one-step off-policy execution.Source: §5 · p. 9
6 Evaluation
The evaluation compares CadenceRL with rollout schedulers on homogeneous and heterogeneous accelerator pools, then tests workload dynamics, parameter sensitivity, and KV preparation overhead.
6.1 Experimental Setup
- Tests use Qwen3-8B, Qwen3-14B, and Qwen3-32B across A910X, BI-V150, and H800 pools, with rollout as the throughput bottleneck and training on H800.The comparison includes Sync, One-off, DORA∗, Laminar, and AReaL.Source: §6.1 · p. 9
6.2 End-to-End Performance
- On homogeneous H800 pools, CadenceRL has throughput comparable to Laminar and AReaL at Qwen3-8B, with a wider gap at 14B and 32B; it has the lowest latency among asynchronous strategies.Sync and One-off have the lowest latency overall, at the cost of rollout–training overlap.Source: §6.2 · pp. 9–10, Figure 10 · p. 10, Figure 11 · p. 10
- Across heterogeneous configurations, CadenceRL matches or exceeds baseline throughput while substantially reducing trajectory latency; high-bandwidth additions reduce tail latency, while cost-efficient additions increase throughput.The comparison excludes DORA∗ because its reallocation model assumes homogeneous hardware.Source: §6.2 · p. 10, Figure 12 · p. 11
- On heterogeneous pools, H800 workers sustain higher mean context lengths than A910X and BI-V150 workers; specialization re-emerges as load subsides.Under high load, all accelerator types carry elevated context lengths.Source: §6.2 · p. 10, Figure 13 · p. 11
- A roughly 16K-token trajectory under CadenceRL repeatedly yields and returns at high KV share, completing first despite being resident for less than half its lifecycle.Laminar keeps the trajectory continuously resident at persistently low KV share.Source: §6.2 · p. 11, Figure 14 · p. 11
6.3 Sensitivity Analysis
- Raising η from 1 to 3 improves throughput, but further increases reduce it while latency rises monotonically; reward curves overlap across six strategies at η=1.The throughput decline beyond η=3 is attributed to additional overlap prolonging concentration without increasing throughput.Source: §6.3 · pp. 11–12, Figure 15 · p. 11, Figure 17 · p. 12
- On Qwen3-14B, increasing decode reserve δ from 512 to 4096 tokens leaves aggregate throughput stable but substantially raises P95 latency.A larger reserve reduces placement headroom and scheduling freedom.Source: §6.3 · p. 11, Figure 16 · p. 12
6.4 KV Cache Transfer Overhead
- KV preparation on Qwen3-8B modestly improves throughput during pacing but slightly reduces it during concentration, with both effects within a few percent.Transfer-time tails continue to shrink as relay nodes are added, with diminishing returns.Source: §6.4 · p. 12, Figure 18 · p. 12
7 Related Work
Related work addresses RL orchestration, trajectory execution and tail scheduling, and serving-side KV management; CadenceRL focuses on reshaping unfinished generations within a rollout pool.
RL execution and resource orchestration
Trajectory execution and tail scheduling
- Trajectory methods handle chunks, iterations, or routing pools: Seer [24], Heddle [48], APRIL [52], and TailSieve [42]; CadenceRL instead controls worker resident sets using pending work and KV headroom.Its pacing and concentration avoid fixed chunks and per-migration benefit prediction.Source: §7 · p. 12
Serving and KV-cache management
- Serving systems offer batching and KV-management mechanisms, while CadenceRL changes rollout-pool composition, stages KV state for late-bound placement, and falls back to re-prefill when state cannot be reused.The paper describes CadenceRL as complementary to these serving techniques.Source: §7 · p. 12
8 Conclusion
CadenceRL reshapes heterogeneous rollout workloads to specialize workers for throughput or fast tail completion, bypassing per-migration benefit estimation. Its conclusion is that complementary accelerator strengths enable scaling across different performance axes without manual routing.
8 Conclusion
- CadenceRL uses pacing and concentration within a shared control structure to reshape workloads toward specialization, bypassing per-migration benefit estimation; heterogeneous accelerator strengths improve different performance axes without manual routing.The conclusion characterizes this heterogeneous scaling as composable.Source: §8 · p. 12
A Rollout Coordination Mechanisms
Rollout coordination mechanisms govern when unfinished work runs, where it resumes, and when workers advance policy versions, but not how workload shapes are matched to heterogeneous accelerators.
A Rollout Coordination Mechanisms
- Partial rollout, redundant rollout, independent weight advancement, and migration coordinate unfinished work, but do not decide which workload shapes go to which heterogeneous accelerators.That matching depends on how accelerator performance changes with the workload shape that emerges during execution.Source: A Rollout Coordination Mechanisms · p. 16, Figure 19 · p. 16
B Scheduling Algorithms
CadenceRL couples pacing, which reshapes fresh work, with concentration, which places the residual tail according to hardware affinity. Observable workload and hardware properties guide these decisions without per-move benefit estimation.
Pacing
- Pacing restores KV headroom by evicting the longest residents, then places pending trajectories shortest-first using prompt affinity or a tight fit.A worker is feasible when post-placement headroom clears the floor plus a per-trajectory decode reserve.Source: B Scheduling Algorithms · p. 16
Concentration
- Concentration drains a source and places the longest residual trajectories first on workers ordered by descending affinity and headroom.The highest-affinity worker is the preferred tail target; the lowest-affinity worker drains first and advances to fresh work.Source: B Scheduling Algorithms · pp. 16–17
Affinity rank
- The ordinal affinity rank avoids per-move benefit prediction because hardware types’ relative strengths across workload shapes remain stable during execution.The reported order is H800 > BI-V150 > A910X, matching long-context tail work to H800 and fresh large-batch work to A910X.Source: B Scheduling Algorithms · p. 17
Concentration
- A source eviction waits until the pending set is empty, so each displacement cascade resolves before another begins.This observable gate ties evacuation to demonstrated downstream absorption rather than predicted completion times.Source: B Scheduling Algorithms · p. 17
Structural guarantees
- Pacing, affinity ranking, and the pending-set gate use structural properties of observable quantities to avoid the benefit-estimation loop.Pacing relies on throughput monotonicity, ranking on stable comparative advantages, and the gate on whether the pending set is empty.Source: B Scheduling Algorithms · pp. 17–18
Across policy versions
- The mechanisms form a cycle: workers pace fresh work, concentrate the tail when fresh work is exhausted, then re-enter pacing after version advancement.An advancing worker joins an intermediate version first when weaker-affinity workers there still hold residual tail work.Source: B Scheduling Algorithms · pp. 16, 18
C Experimental Configuration
The homogeneous experiments provision spare trainer capacity so rollout remains the throughput bottleneck, with device allocations and scheduling parameters varying by model size.
Capacity provisioning
- All configurations reserve spare trainer capacity and make rollout the throughput bottleneck; Qwen3-8B and Qwen3-14B do so with a larger trainer device share.For Qwen3-32B, the minimum trainer allocation is TP×PP=8×2=16 devices.Source: C Experimental Configuration · p. 18
Qwen3-32B
- Qwen3-32B requires a 1:1 rollout-to-trainer device ratio at both pool sizes because its minimum trainer allocation is 16 devices.The device split alone does not ensure rollout is the actual bottleneck at that ratio.Source: C Experimental Configuration · p. 18
Scheduling parameters
- At the Qwen3-32B 1:1 ratio, a higher staleness parameter η increases concurrent policy versions, supplying fresh work to keep rollout fully utilized.The staleness parameter η and decode reserve δ vary by model size.Source: C Experimental Configuration · p. 18
D Baseline Descriptions
The evaluation compares CadenceRL with five rollout-coordination strategies in a shared underlying rollout and training framework, varying the coordination strategy.
Comparison setup
- CadenceRL and all five baselines reuse the same rollout and training framework, isolating rollout coordination as the comparison dimension.The strategies span synchronous, one-step off-policy, resource allocation, asynchronous repacking, and partial-rollout approaches.Source: D Baseline Descriptions · p. 18
Sync and One-off
- Sync runs rollout and training sequentially without policy-version staleness, while One-off overlaps them using trajectories from the previous policy version.One-off bounds staleness to one policy step [50].Source: D Baseline Descriptions · p. 18
DORA* and Laminar
- DORA* reallocates rollout workers across policy versions at weight synchronization points, while Laminar independently advances workers and repacks unfinished trajectories within a version group.DORA* is a partial implementation of its workload-aware resource-allocation component [11]; Laminar's repack consolidates work from underutilized workers [31].Source: D Baseline Descriptions · p. 18
AReaL
- AReaL interrupts generation when new policy weights arrive, then resumes unfinished trajectories with the updated weights and recomputes their interrupted prefixes.A trajectory may therefore contain segments generated by multiple policy versions [5].Source: D Baseline Descriptions · p. 18
E Heterogeneous Pool Analysis
CadenceRL improves throughput and latency over baselines in both heterogeneous configurations. Phase-level analysis relates their throughput differences to accelerator behavior and long-context trajectories’ time awaiting active decoding.
Overall comparison
- Both 32A and 32B improve throughput and latency over baselines, eliminating the throughput–latency trade-off visible in the H800 comparison.The analysis connects the results to workload-composition sensitivity and phase-level throughput.Source: E Heterogeneous Pool Analysis · p. 18
Pacing phase
- During pacing, both accelerator types operate in a higher, more concentrated throughput range, with A910X sustaining higher throughput than BI-V150.At TP=2, A910X has twice the aggregate HBM capacity of BI-V150—128 GB versus 64 GB—supporting larger active batches; pacing sustains A910X batch size and reduces BI-V150 context-length skew.Source: E Heterogeneous Pool Analysis · p. 19, Figure 20 · p. 19
Concentration phase
- Under the same workload distribution, fresh-work supply, and admission budget, 32A reaches concentration sooner, and its throughput advantage persists while both pools drain the long-context tail.The earlier transition allows more decode resource per remaining trajectory.Source: E Heterogeneous Pool Analysis · p. 19, Figure 20 · p. 19
Non-resident time
- The 32A non-resident fraction is lower than 32B’s at every reported percentile, with the gap widening toward the upper tail.Higher aggregate decode throughput lets workers free KV headroom sooner, returning pending trajectories to active decoding and reducing non-resident time.Source: Table 3 · p. 19, E Heterogeneous Pool Analysis · p. 19