StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training

1 Introduction

• Asynchronous execution improves throughput but can feed training stale trajectories, while variable trajectory lengths create workload skewness that degrades resource utilization and performance.Rollout and training overlap and jointly dominate end-to-end execution time.Source: §1 · p. 1, Figure 1 · p. 1
• Existing systems trade off convergence and performance because strict staleness guarantees can constrain skewness mitigation, while flexible coordination often relaxes staleness control.The paper identifies trajectory-level control and globally coordinated, staleness-aware rollout coordination as unresolved challenges.Source: §1 · p. 2
• StaleFlow tracks each trajectory’s lifecycle with a global consistency protocol and adds trajectory and parameter servers to enable flexible rollout coordination.A centralized coordinator analyzes system snapshots and issues rollout commands.Source: §1 · p. 2
• 1.42-2.68× (1.18-1.91× on average) higher throughput than state-of-the-art RL systems, while preserving convergence, was achieved by StaleFlow.Evaluations were conducted on a 128-GPU cluster.Source: §1 · p. 2

2.1 RL Workflow and Trend

• Asynchronous overlap across rollout, reward, and training supports independent resource use, but older rollout parameters can impair convergence and variable-length trajectories degrade system performance.A prompt alone is considered an initial trajectory in this paper.Source: §2.1 · pp. 2–3

2.2 Data Staleness in RL

• A larger staleness bound can reduce training idle waiting and improve throughput, but generally degrades convergence; existing systems range from user-defined bounds to one-step asynchrony or no explicit guarantee.The paper identifies strict staleness control as offering performance-optimization opportunities while maintaining a bounded guarantee.Source: §2.2 · p. 3

2.3 Data Skewness in RL

• Trajectory-length skewness leaves instances underutilized by long-tail trajectories and makes faster instances wait for slower ones before global parameter synchronization.Partial rollout, redundant rollout, multi-version instances, and migration address different aspects of the imbalance, but are often used separately and ad hoc.Source: §2.3 · pp. 3–4

2.4 Challenges and Our Solutions

• Existing systems lack a fine-grained control mechanism compatible with flexible coordination and a unified, adaptive way to optimize diverse rollout techniques together.Coarse controls based on one rollout model version or the amount of in-flight data fall short when coordination uses multiple versions or redundant rollouts.Source: §2.4 · p. 4
• StaleFlow tracks every trajectory’s lifecycle for fine-grained staleness control, then uses trajectory and parameter servers with snapshot-command strategies to coordinate rollout flexibly.The strategies unify coordination techniques and invoke them at appropriate times while preserving strict staleness control.Source: §2.4 · p. 4

• A staleness manager uses a global consistency protocol to verify trajectory admission, register in-progress and rewarded trajectories, and retire consumed batches under the staleness bound.Source: §3 · pp. 4–5
• Trajectories move from the trajectory server through rollout, reward, and training; updated parameters are pushed to the parameter server for selective rollout-instance pulls.Source: Figure 6 · p. 5
• The rollout service adds a centralized coordinator, trajectory server, and parameter server to support flexible coordination of trajectories and model parameters.Source: §3 · pp. 4–5

4.1 Trajectory Version Identifier

• Each trajectory’s V_traj identifies the model version used to generate it, or the oldest tolerated version when partial rollout uses different versions.Source: §4.1 · p. 5
• The staleness manager checks proposed versions against η and tracks each assigned trajectory throughout its lifecycle.Source: §4.1 · p. 5

4.2 Global Consistency Protocol

• Buffers enforce V_traj + η ≥ V_buf through Reserve placeholders for unfinished trajectories and greedy Occupy placement after completion.The protocol simulates Reserve to test whether a proposed assignment is admissible; it tracks metadata rather than trajectory payloads, and its measured end-to-end latency overhead is under 1%.Source: §4.2 · pp. 5–6

4.3 Support for Flexible Rollout Coordination

• The buffer protocol supports partial rollout and migration, group-level sampling, redundant rollout, and filtering of excess or delayed trajectories.For group sampling, entries are reserved and occupied per group; when a buffer is stuck beyond a configurable threshold, remaining reserved entries can be aborted and later occupied entries moved forward.Source: §4.3 · pp. 6–7

5.1 Rollout Service Architecture

• Trajectory and parameter servers decouple data movement from rollout execution, enabling the coordinator to control trajectory delivery and parameter synchronization.The servers add movement overhead, but TS Route/Interrupt and PS Pull account for less than 3% of total overhead.Source: §5.1 · pp. 7–8

5.2 Snapshot-Command Cycle

• Snapshots record five per-instance fields, while speculative state validates that prior commands took effect before the coordinator uses a new snapshot.Commands across data planes run concurrently, with dependencies handled by waiting or keeping data pending.Source: §5.2 · p. 8

5.3 Rollout Coordination Strategies

• Routing prioritizes staler trajectories in a multi-level queue and selects admissible instances using estimated marginal throughput gain.The cost model estimates throughput from running trajectories and KV Cache usage; candidate instances are grouped by model version, with smaller versions prioritized.Source: §5.3 · p. 9
• Synchronization occurs when a stale instance cannot accept admissible trajectories and a tentative update enables routing; migration responds to waiting-queue limits or throughput gaps.The coordinator interrupts and returns excess or selected trajectories to the TS for possible rerouting.Source: §5.3 · p. 10

6.1 Experimental Setup

• The experiments use a 16-machine H20-GPU testbed, DAPO training, and four Qwen-family models, measuring throughput and convergence with tokens/s, reward curves, and pass@1.The smallest setting uses Qwen2.5-Math-7B with GSM8K and MATH; larger settings train on DAPO-Math-17k and evaluate on AIME24.Source: §6.1 · pp. 10–11

6.2 End-to-End Throughput

• 1.42× (1.18× on average) higher throughput than VeRL-Async, the best-performing strict-staleness baseline, with StaleFlow consistently leading across models and bounds.It improves throughput over VeRL by up to 2.68× (1.91× on average) and over VeRL-Pipeline by up to 1.95× (1.51× on average).Source: §6.2 · p. 11

6.3 Convergence

• With no staleness control or an excessively large bound such as η=10, training collapses; at properly configured bounds, convergence is comparable to synchronous VeRL.At η=3, StaleFlow and VeRL-Async reach similar AIME24 scores after approximately 150 steps, while StaleFlow delivers higher throughput.Source: §6.3 · p. 12

6.4 Scalability

• StaleFlow maintains the highest absolute throughput as response length and batch size vary, and scales at least comparably as GPU count increases.Its relative throughput scales better than other systems as response lengths increase, and its advantage becomes more pronounced at larger staleness bounds.Source: §6.4 · p. 12, Figure 16 · p. 12

6.5 Performance Interpretation

• Replacing vanilla rollout strategies progressively improves throughput, with the largest gain when routing, synchronization, and migration strategies are all enabled.The all-vanilla configuration performs similarly to VeRL-Async; StaleFlow’s routing balances throughput rather than trajectory counts, its synchronization avoids unnecessary interruptions, and its migration proactively redistributes work.Source: §6.5 · p. 12
• StaleFlow routes according to marginal throughput gain, synchronizes at each instance’s pace, and continuously migrates trajectories to prevent growing queues and load imbalance.In contrast, vanilla routing distributes assignable trajectories evenly, greedy global synchronization can interrupt running trajectories, and passive migration does not proactively rebalance work.Source: §6.5 · pp. 12–13, Figure 18 · p. 13
• With η=3, no trajectory exceeds staleness 3, and most buffers contain trajectories at that maximum as training proceeds.This reflects strict staleness control alongside use of the configured bound to extract throughput.Source: §6.5 · p. 13, Figure 19 · p. 13
• Per-token decoding during trajectory generation accounts for 89.2% of average per-step rollout time, while KV Cache prefill or re-prefill after routing or migration takes 7.9%; global consistency protocol overhead remains below 1%.Rollout coordination commands add below 3% overhead; Route and Interrupt consume 0.5%, while Pull takes longer because it transfers larger model parameters.Source: §6.5 · p. 13

RL post-training systems

• Earlier RL post-training systems largely used synchronous shared resources, while newer systems increasingly use asynchronous, disaggregated designs and rollout coordination to mitigate skewness.The paper characterizes most existing coordination techniques as ad hoc and says existing systems lack comprehensive staleness control for global optimization under a bound.Source: §7 · p. 13

Optimizing training and generation

• Training- and generation-efficiency optimizations, including parallelism, speculative decoding, and quantization, are described as orthogonal to StaleFlow and applicable to it.StaleFlow’s disaggregated architecture separates rollout from training so each phase can be optimized independently.Source: §7 · pp. 13–14

Optimizing data skewness

• Unlike skewness in pre-training, fine-tuning, and serving, RL post-training also has data staleness, which the paper says must be optimized jointly with skewness.StaleFlow incorporates staleness constraints into its data-management design while prioritizing convergence and system performance.Source: §7 · p. 14

8 Conclusion and Future Work

• StaleFlow jointly addresses data staleness and skewness with a global consistency protocol and staleness-aware, throughput-oriented rollout coordination, achieving up to 1.42–2.68× higher throughput than state-of-the-art systems while preserving convergence.The reported average throughput gains are 1.18–1.91×; future work plans broader RL tasks, including coding and agent-based tasks, and multimodal models.Source: §8 · p. 14

A Implementation

• StaleFlow comprises approximately 22k lines of Python, including about 2k lines for the staleness manager and nearly 2k for its training side.The rollout service is roughly 10k lines, including vLLM generation, the rollout coordinator, trajectory server, and parameter server.Source: A Implementation · p. 17
• The parameter server uses NIXL, built on UCX, to abstract point-to-point communication across GPU, CPU, and storage hardware.This abstraction enables CPU deployment of the parameter server while communicating with GPUs used for training and rollout.Source: A Implementation · p. 17

A.1 Parameter Server Deployment

• Parameter-server workers mediate between training and rollout, and load-balancing plans govern parameter-slice transfers during Push and Pull.The deployment uses workers configured through a unified interface.Source: §A.1 · p. 17, §A.2 · p. 17

A.2 Load-Balancing Communication Plan

• For Push, the planner greedily assigns each required slice to the sender with the lowest accumulated estimated communication latency, then reuses the mappings.The latency estimate divides slice size by sender-receiver bandwidth and adds a profiled constant latency term.Source: §A.2 · pp. 17–18, Figure 23 · p. 17

A.3 Scalability

• Increasing the cluster from 16 to 128 GPUs does not increase parameter-synchronization overhead under proportional scaling of workers and replicas.The communication plans let replicas communicate concurrently without interference.Source: §A.3 · p. 18

B Cost Model

• The cost-model section derives estimated instance generation throughput and compares estimates with measured throughput.The derivation and accuracy check are described in Appendix B.1 and B.2.Source: B Cost Model · p. 18

B.1 Derivation

• Matrix-multiplication latency is the maximum of a memory-cost term proportional to parameter-matrix size and a computation-cost term proportional to FLOPs.The modeled FLOPs are 2 × n × h × h'.Source: §B.1 · p. 18
• The model’s max term switches from memory-bound to compute-bound when k3 × n exceeds k2.For bf16 on an H20 GPU, the threshold is approximately 37 running trajectories.Source: §B.1 · pp. 18–19
• Instance throughput is estimated as the number of running trajectories divided by per-decoding-step latency.The model treats pure decoding throughput as overall generation throughput because decoding occupies most rollout time.Source: §B.1 · pp. 18–19
• The final estimate uses running-trajectory count and KV-cache size, with coefficients profiled offline and fitted by linear regression.T_i(S) = |S[i].run_trajs| / (k1 × S[i].kv_cache + max(k2, k3 × |S[i].run_trajs|) + k4).Source: §B.1 · p. 19

B.2 Estimation Accuracy

• The accuracy comparison used Qwen3-30B-A3B on 128 GPUs, with batch size 128, group size 16, response length 40K, and staleness bound 3.The coefficients were profiled and summarized in Table 4.Source: §B.2 · p. 19
• The model’s estimates were compared with actual measured generation throughput over the reported experimental setup.Figure 25 presents the comparison.Source: §B.2 · p. 19
• 10.52% average estimation error indicates that the cost model closely estimates actual throughput, with only a few outliers.The authors report that it can reliably guide rollout coordination strategies.Source: §B.2 · p. 19

C Ablation Study on Redundant Rollout

• Batch-level redundant rollout reduces mean and maximum response lengths more than group-level redundancy at the same 1/16 ratio.Trajectories within a group tend to be either all long or all short, so batch-level redundancy can discard an entire long-sequence group.Source: C Ablation Study on Redundant Rollout · pp. 19–20
• Redundant rollout reduces post-training time per step and modestly improves throughput, although both total tokens and total time decrease.The authors attribute the throughput improvement to reduced data skewness after excessively long trajectories are eliminated.Source: C Ablation Study on Redundant Rollout · p. 20
• The main §6 experiments omit redundant rollout because it introduces a noticeable distribution shift in generated lengths.The authors discuss it separately to ensure a fair baseline comparison.Source: C Ablation Study on Redundant Rollout · p. 20

D Detailed Algorithms for Rollout Coordination Strategies

• The detailed coordination algorithms cover routing, synchronization, and migration, with separate procedures for each operation.The appendix labels Algorithms 3, 4, and 5 as routing, synchronization, and migration strategies, respectively.Source: D Detailed Algorithms for Rollout… · pp. 20–21