DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training

Motivation

• Replication-based methods [9] [45] discard in-flight long trajectories, while partial-rollout methods [36] [41] [8] resume them under updated policies, incurring overhead or algorithmic compromises.Discarding trajectories can distort GRPO [30] advantage estimates; partial rollouts can invalidate KV-Cache and stitch policy versions within a trajectory.Source: §1 · p. 2

Long-Tail Dilemma

• Because long trajectories must either finish before an update or be sacrificed under single-version rollout, that assumption leaves no path to avoid both overhead and algorithmic compromise.In mathematics and coding, the 99th-percentile output can exceed the median by over an order of magnitude, and these long trajectories carry information-dense reasoning steps.Source: §1 · p. 2

Multi-Version Streaming Training

• DORA maintains multiple policy versions concurrently, letting long trajectories finish under the policy active at dispatch while new requests use the latest version.This multi-version streaming paradigm eliminates the long-tail bubble without significant overhead or algorithmic compromise.Source: §1 · p. 2

Dynamic Orchestration

• A centralized orchestrator re-partitions data-parallel groups according to pending workloads and migrates requests to rebalance resources across versions.This addresses underutilized instances running older versions while the latest version is oversubscribed.Source: §1 · p. 2

Nearly Zero Re-prefill KV-Cache Reuse

• Same-version KV-Cache transfer avoids prefill recomputation during request migration, especially benefiting long-context reasoning and agentic scenarios.The cache states are equivalent across instances hosting the same policy version.Source: §1 · pp. 2–3

Extensive Evaluation and Real-World Deployment

• On open-source benchmarks, DORA achieves up to 2.12× end-to-end and 8.2× rollout-stage speedup over synchronous training while preserving model convergence.Production deployment with thousands of accelerators yields up to 6.2× rollout speedup.Source: §1 · p. 3

2.1 Asynchronous RL Training

• Asynchronous training replaces the batch barrier by mixing trajectories from different behavior-policy versions, with GRPO [30] using per-trajectory behavior policies in its importance ratio.The policy-version gap is bounded by configurable staleness K, and the training and rollout batch sizes are denoted TBS and RBS.Source: §2.1 · p. 3
• For GRPO [30], sampled trajectories share group-relative advantages across their tokens and update the policy through a clipped importance-weighted objective.The asynchronous objective substitutes each trajectory’s behavior policy in the importance ratio, subject to the staleness condition.Source: §2.1 · p. 3

2.2 Long-Tail Dilemma

• On an open-source benchmark and production workload, the 99th-percentile output exceeds the median by over an order of magnitude, while prefill duration grows with input length.Because decoding is memory-bound, adding compute cannot flatten the response-length tail; re-triggering prefill becomes increasingly expensive for long contexts.Source: §2.2 · p. 3
• Long-tailed rollout creates intra-node idle slots and inter-node waits for the slowest instance; in MoE, skew can stall the entire expert-parallel group.Long-tail trajectories remain indispensable for RL, so existing asynchronous approaches risk re-prefill overhead or sampling bias when mitigating rollout bubbles.Source: §2.2 · p. 4

3.1 System Overview

• DORA combines multi-version streaming, dynamic orchestration, and KV-cache reuse to eliminate rollout bubbles while preserving trajectories and bounding staleness.The design targets the long-tail dilemma without significant system overhead or algorithmic compromise.Source: §3.1 · pp. 4–5

3.2 Multi-version Streaming Training

• Multiple policy versions let completed trajectories stream into training while long-running requests continue under their dispatch-time policy within a bounded sliding window.The window advances only after trajectories from the oldest version are collected and forwarded; larger staleness bounds trade convergence impact for throughput.Source: §3.2 · pp. 5–6

3.3 Dynamic Resource Orchestration

• A centralized orchestrator reallocates DP groups in proportion to version workloads, migrates active requests, and supplements data within the staleness bound.Its update-driven, utilization-based, and temporal triggers coordinate workload balancing and request migration without abandoning trajectories.Source: §3.3 · pp. 6–7

3.4 KV-Cache Reuse

• Single-version trajectories have equivalent KV-cache states across instances hosting that version, enabling migration without re-prefill.DORA also prioritizes returning requests to original ranks when possible and can offload KV caches to host memory under VRAM pressure.Source: §3.4 · p. 7

4.1 Experimental Setup

• Open-source experiments use Qwen2.5-32B on a 16-node, 128-H800-GPU cluster, while production-scale evaluation uses a ∼500B-parameter MoE model.Baselines share the in-house framework and identical hardware; the setup also reports five-iteration steady-state averages and DAPO-style training configurations.Source: §4.1 · p. 8

4.2 Training Performance

• On 64 GPUs, DORA cuts rollout-only duration from 14.9 to 1.8 minutes, an 8.2× reduction, and achieves 1.56× end-to-end speedup over synchronous training.Against partial rollout, DORA achieves 1.18× end-to-end speedup; on 128 GPUs, it achieves 5.9× rollout and 1.93× end-to-end speedup over synchronous training.Source: §4.2 · p. 8
• 34,135 tokens/s on 128 GPUs gives DORA 2.12× the throughput of synchronous training and 1.11× that of partial rollout.At 64 GPUs, DORA reaches 23,327 tokens/s, 1.65× synchronous and 1.17× partial rollout.Source: §4.2 · p. 8
• Both DORA variants closely track synchronous training reward over 100 steps, although k=3 converges moderately more slowly than k=1.The study uses 72 GPUs and identifies bounded staleness as important to convergence behavior.Source: §4.2 · p. 8

4.4 Overhead Analysis

• At 64 and 128 GPUs, load balancing uses 0.4% and 1.5% of execution time, request transfer stays below 4%, and free-cache operations stay below 0.03%.Request-transfer overhead falls from 3.6% to 2.1% as throughput increases; aggregate overhead does not grow with cluster size.Source: Figure 12 · p. 9, §4.4 · p. 9

4.3 Ablation Study

• KV-cache reuse reduces average rollout time on Dense-32B from 183 to approximately 166 seconds, a 9% speedup.The dynamic resource orchestration module is not ablated because it is indispensable for the system to function.Source: Figure 11 · p. 9, §4.3 · p. 9

4.5 Production Deployment

• On a ∼500B-parameter MoE model, DORA achieves 3.6× rollout speedup on mathematical and tool-integrated reasoning and up to 6.2× on agentic training.The production comparison uses a synchronous baseline and 4,096 accelerators; the authors report 2–4× end-to-end speedup with no quality degradation at scale.Source: Figure 13 · p. 9, §4.5 · p. 9

5 Conclusion and Limitations

• DORA’s single-policy-per-trajectory design enables KV-cache equivalence across instances, supporting zero-re-prefill migration alongside bounded-staleness asynchronous training.The conclusion reports up to 8.2× rollout speedup and 2.12× end-to-end acceleration over synchronous training.Source: §5 · p. 9
• Evaluations use a controlled in-house framework, and MoE scaling is validated primarily on production data and cluster rather than broader public-system and open-source MoE comparisons.The authors identify benchmarking against veRL and AReaL and testing more open-source MoE models as future work.Source: §5 · p. 10

A Related Work

• Oversampling methods [9] [45] discard unfinished trajectories, while one-step off-policy methods [47] [22] [14] [12] leave intra- and inter-node rollout bubbles.The discussion says discarded long trajectories can distort advantage estimation in group-relative methods such as GRPO [30].Source: A Related Work · p. 13
• Partial-rollout methods [36] [41] [6] [8] splice policy versions within trajectories, requiring algorithmic corrections and re-prefill after weight updates; concurrent systems [29] [32] use different orchestration and staleness mechanisms.DORA contrasts these approaches with end-to-end single-version trajectories and zero-re-prefill migration.Source: A Related Work · p. 13

B Experimental Setup

• Open-source experiments run on 16 nodes with eight H800 GPUs each, while the production cluster uses non-CUDA accelerators with approximately 60 GB of device memory.The H800 cluster uses NVLink within nodes and network interfaces between nodes.Source: B Experimental Setup · p. 13
• The evaluations use Qwen2.5-32B for dense models and an approximately 500B open-source MoE model, measuring tokens per second and wall-clock minutes per RL iteration.Reported metrics average five post-warm-up RL iterations.Source: B Experimental Setup · p. 14
• Training uses DAPO-Math-17k with input and output limits of 2K and 30K tokens, respectively, and compares synchronous, one-step off-policy, and partial-rollout baselines.The DAPO-style configuration samples 16 responses per prompt and uses a global batch of 8,192; the selected checkpoint averages 2.4K-token responses with a 30K maximum.Source: B Experimental Setup · p. 14

C Case Study: Empirical Validation of Multi-Version Streaming

• The case study analyzes 22,820 rollout records containing 365,120 responses from steps 20–100 at staleness bound K=3.It compares DORA, partial rollout, and synchronous training; each record contains 16 responses scored by a binary reward model.Source: C Case Study: Empirical Validation of… · p. 14

• Aggregated pass rate is 58.9% for staleness-0 versus 55.6% for staleness-1, an apparent 3.3 percentage-point gap.Source: §C.1 · p. 14
• After controlling for difficulty, ∆<0.6% across bins; Table 1 stratifies pass rates by problem difficulty.Source: Table 1 · p. 14, §C.1 · p. 14, Figure 14 · p. 15
• Harder problems produce longer responses and thus higher staleness labels; staleness-1 responses have a median of 1,495 tokens versus 1,015 for staleness-0.Source: §C.1 · p. 14
• In 141,632 Partial Rollout responses, staleness-0 and staleness-1 perform identically within each difficulty bin.Source: §C.1 · p. 14, Table 2 · p. 15
• The on-policy improvement rate is 0.113% per step, making the theoretical maximum penalty for K=3 only 0.34%.Source: §C.1 · p. 15

C.2 Observation 2: Long-Tail Trajectories Carry the Highest Training Signal

• Medium-difficulty records (pass rate 0.25–0.75) have reward variance ∼1.0, the strongest GRPO signal, while also being the longest and most stale.Source: §C.2 · p. 15, Table 3 · p. 15
• Replication-based systems that stop after collecting TBS short trajectories discard these high-signal samples, whereas DORA preserves every sampled trajectory via multi-version streaming.Source: §C.2 · p. 15

C.3 Observation 3: Quantifying the Long-Tail Bubble

• Synchronous training’s inter-node bubble exceeds 74% at step 20 and reaches 88.5% at step 100; DORA’s streaming could theoretically eliminate it.The bubble is measured as the fraction of decode time wasted waiting for the longest trajectory.Source: §C.3 · p. 16, Figure 15 · p. 16

C.4 Summary

• DORA’s design is supported by near-zero difficulty-controlled degradation at staleness ≤3, preservation of the highest-signal long trajectories, and a 75–89% synchronous bubble.Source: §C.4 · p. 16

D A Policy-Improvement View of Single-Version Rollout in DORA vs. Partial Rollout

• The comparison is a stylized mixed-behavior formulation, not a faithful model of full system dynamics or partial-rollout correction mechanisms.Source: D A Policy-Improvement View of… · p. 17
• DORA keeps the policy version fixed throughout each trajectory, while partial rollout may switch to a newer version at a weight update.Source: D A Policy-Improvement View of… · p. 17

D.1 Setup and Notation

• The setup uses a token-level MDP in which a response is a length-H trajectory and the active policy version may evolve along it.Source: §D.1 · p. 17

D.2 Monotonic Improvement under Trajectory-Level Mixing

• Under trajectory-level mixing, the augmented-state advantage reduces to each version’s own advantage, so the standard sample-reuse bound applies without cross-version correction.Source: §D.2 · pp. 17–18

D.3 Advantage-Substitution Bias under Step/Segment-Level Mixing

• Under step/segment mixing (σ > 0), substituting a single-version advantage creates nonzero bias when the switched-to version differs, while DORA’s σ = 0 removes it.The bias vanishes as σ approaches zero; the penalty enters the improvement guarantee directly.Source: §D.3 · p. 18

D.4 Discussion: Consequences for DORA

• Because DORA’s per-trajectory version makes the standard GRPO surrogate exact, it needs no algorithmic correction; partial rollout uses decoupled objectives [8] or gradient masking [36].The passage attributes DORA’s freedom from corrections to trajectory-level mixing, not an engineering simplification.Source: §D.3 · p. 18, §D.4 · p. 19

D.3 Advantage-Substitution Bias under Step/Segment-Level Mixing

• Even this idealized proxy understates partial rollout’s substitution error: its critics or shared group-relative advantages may not recover Aπwi exactly.The passage characterizes the bias estimate as a lower bound on true substitution error.Source: §D.3 · p. 18

D.4 Discussion: Consequences for DORA

• A single behavior version gives DORA a well-defined sequence-level importance ratio, unlike stitched trajectories at segment boundaries.The stitched trajectory corresponds to no single behavior policy, so its sequence-level ratio loses that interpretation.Source: §D.4 · p. 19
• The sliding-window bound K limits staleness but requires manual configuration and clipping; segment switching can refresh deviation faster, so neither mixing regime uniformly dominates.DORA trades a higher per-trajectory staleness ceiling bounded by K for exact advantage reduction, a well-defined ratio, and KV-Cache equivalence enabling zero-re-prefill migration.Source: §D.4 · p. 19

E Proofs for the Policy-Improvement Analysis

• The policy-improvement proofs are carried out on an augmented MDP that treats mixed behavior as one Markov policy, with DORA’s per-version form recovered through Lemma 1.The appendix proceeds from regularity conditions and the performance-difference identity to proofs of the lemma, theorem, and corollary.Source: E Proofs for the Policy-Improvement… · p. 19, §E.1 · p. 19

E.1 Regularity Conditions and the Performance-Difference Identity

• The analysis uses discounted infinite-horizon notation for finite LLM responses; the finite-horizon decomposition replaces 1/(1 − γ) with a horizon-dependent constant.The augmented-MDP argument also assumes bounded advantages, common support, and a bounded importance ratio.Source: §E.1 · pp. 19–20

E.2 Proof of Lemma 1 (Advantage Reduction)

• With trajectory-level mixing, the version index stays fixed, so the augmented advantage equals the per-version advantage and visitation factorizes by version fraction.The proof uses the fixed-index behavior policy to identify the augmented value with the base-MDP value for that version.Source: §E.2 · p. 20

E.3 Proof of Theorem 1 (Monotonic Improvement)

• The theorem proof applies the augmented trust-region bound for any index kernel, then uses trajectory-level factorization and advantage reduction to recover the per-version bound.The augmented expectation separates into version-weighted expectations under each version’s visitation distribution.Source: §E.3 · pp. 20–21

E.4 Proof of Corollary 1 (Advantage-Substitution Bias)

• When σ = 0, the substitution penalty vanishes; when versions switch, the augmented advantage can differ from the current-version advantage, producing a penalty in the improvement bound.The proof ties the deviation to the σ-weighted continuation under another version, except where exact cancellation occurs.Source: §E.4 · p. 21

F A Control-Variate View of Baseline Choice in DORA vs. Partial Rollout

• The control-variate analysis compares constant-baseline choices for DORA and partial rollout, using the framework to describe their variance gap.The comparison is presented as a stylized analytical lens, not a faithful model of both systems’ sampling processes.Source: F A Control-Variate View of Baseline… · p. 21

F.1 Setup and Notation

• F.1 models each response as generated under one version—a faithful DORA abstraction but a stylized stand-in for partial rollout—and isolates baseline choice from stitched-segment corrections.The estimators share a sampling distribution and importance-weighted score; under standard regularity, both are unbiased, and batch-averaged variance scales as 1/G.Source: F A Control-Variate View of Baseline… · pp. 21–22, §F.1 · p. 22

F.2 Optimal Constant Baseline

• The variance-minimizing constant baseline adds Ck/Bk, a within-version reward–score coupling correction, to mean reward; that correction is generally impractical in sequence generation.The version-specific baseline retains version-level reward information, while the schedule-averaged baseline aggregates across versions; the latter is optimal only when Ck/Bk is zero for every version and µk is constant.Source: §F.2 · p. 23

F.3 Exact Variance Decomposition

• The variance gap decomposes into non-negative drift and sign-indeterminate coupling terms, so the gap itself can have either sign.The drift reflects cross-version pass-rate variance weighted by gradient energy; the coupling reflects covariance between version pass rates and reward–gradient covariance.Source: §F.3 · p. 23

F.4 Structural Observations

• Under the single-version sampling abstraction, the coupling term is bounded by cross-version variability in pass rates and reward–gradient covariance, but the analysis makes no claim of uniformly lower variance.It does not extrapolate to full partial-rollout systems, whose additional mechanisms lie outside the abstraction.Source: §F.4 · p. 24

F.5 Summary

• The framework compares a version-specific baseline that retains version-level reward information with a schedule-averaged baseline that aggregates across versions.Both are tractable approximations to the optimal constant baseline, and the framework is descriptive rather than a claim about full-system variance.Source: §F.5 · p. 24

G Proofs for the Control-Variate Analysis

• The proof appendix collects regularity assumptions and unbiasedness identities, then proves Lemma 2 and Theorem 2.Its stated assumptions include bounded rewards and scores, non-degenerate gradient energy, and importance-sampling regularity.Source: G Proofs for the Control-Variate… · p. 24, §G.1 · p. 24

G.1 Regularity Assumptions and Unbiasedness

• Under the stated importance-sampling regularity, both baseline-shifted estimators are unbiased for the target policy gradient, including conditional on version k.The conditional expectation of (R − b)S_k equals the target gradient for every constant baseline b.Source: §G.1 · pp. 24–25

G.2 Proof of Lemma 2

• For a fixed version, minimizing estimator variance over a constant baseline reduces to minimizing a strictly convex quadratic, yielding a unique minimizer.The proof uses the baseline-independent conditional mean and B_k > 0.Source: §G.2 · p. 25

G.3 Proof of Theorem 2

• Theorem 2 follows by conditioning on version, decomposing the second moment around the version-specific mean, and aggregating the resulting terms across versions.The proof uses the law of total covariance and identifies the cross term as a covariance over the schedule distribution.Source: §G.3 · pp. 25–26

H Social impacts

• DORA’s training-efficiency gains may reduce LLM training’s environmental footprint, and validation on non-CUDA mid-range accelerators broadens hardware applicability.The paper frames improved access on less advanced accelerators as a contribution to democratizing LLM training.Source: H Social impacts · p. 26