Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony

1 Introduction

• Synchronous barriers between generation, environment interaction, and reward assessment leave GPUs idle when long-tail responses hold up a batch.Rollout can account for over 70% of training time, and the longest responses can exceed median length by more than 20×.Source: §1 · p. 2
• ROLL Flash extends ROLL with fine-grained parallelism and rollout–train decoupling, enabling asynchronous execution and more efficient rollouts.Its components support queue scheduling, prompt replication, environment-level asynchronous rollout, and redundant environment rollout; an asynchronous ratio bounds each sample’s policy-version gap.Source: §1 · pp. 2–3
• Asynchronous training uses a producer–consumer model in which continuous rollout generation avoids stalling for training and mitigates waste from long-tail rollouts.The coarse-grained allocation of training and rollout resources motivates empirical investigation.Source: §1 · p. 2
• Up to 2.24× speedup on RLVR tasks and 2.72× on agentic tasks; agentic gains were 2.72× on ALFWorld and 1.81× on SWE.A small asynchronous ratio often suffices for near-maximal acceleration while preserving sample freshness, and off-policy algorithms match synchronous final performance.Source: §1 · p. 3

2.1 Synchronous RL Post-Training

• Synchronous post-training repeats rollout, reward assignment, and weight updates, with strict synchronization barriers that create resource bubbles.In agentic tasks, rollout produces state-action trajectories before a reward worker scores them.Source: §2.1 · p. 3
• PPO uses an actor-critic clipped surrogate objective to limit policy changes at each update.Its objective compares the current and previous policies; the clipping range is controlled by ϵ.Source: §2.1 · pp. 3–4
• GRPO avoids a critic by sampling multiple responses per prompt and normalizing their rewards to construct advantage signals.Its objective also includes KL-divergence regularization between the current and reference policies.Source: §2.1 · p. 4

2.2 Asynchronous LLM Post-Training

• Asynchronous training overlaps rollout and training, but stale-policy samples can degrade accuracy and make synchronous algorithms unstable.The section describes gradient truncation and importance-sampling optimization as off-policy approaches to stabilize training.Source: §2.2 · p. 4
• Off-policy methods address the old-versus-current policy mismatch through gradient truncation or clipped importance-sampling weights.TOPR truncates only low-return trajectories, preserving learning signals from high-return trajectories while suppressing noise from poor ones.Source: §2.2 · pp. 4–5

3.1 Theoretical Analysis

• Under queue scheduling, per-sample completion time approaches µgen/K as α → ∞; when K = N, theoretical speedup is capped at (Lgen + µgen)/µgen.The generation-time bound assumes K workers and sample generation times in [0, Lgen].Source: §3.1 · p. 5
• With K workers split between generation and training, Async's optimally balanced upper bound is tighter than Sync's and strictly improves when α > 0.The analysis defines α as the permitted lag, in model updates, of the rollout policy behind the current training model.Source: §3.1 · pp. 5–6

3.2 Experimental Validation

• At 128 GPUs, Async achieves 2.24× higher throughput than Sync-Naive for Qwen3-8B-Base; for Qwen3-8B-Think, 8× GPUs yield 7.6× speedup, 2.13× above the traditional synchronous (Sync-Naive) baseline.Source: §3.2 · pp. 6–7
• As GPU count rises, shrinking per-GPU workloads amplify long-tail effects in synchronous training, while Async decoupling removes the straggler bottleneck.Sync-ROLL partially alleviates the problem using queue scheduling and prompt replication.Source: §3.2 · p. 7
• With 16 training and 24 inference GPUs, Async achieves nearly 2× speedup over baseline1; modest waiting for fresh samples can also help stabilize training.The 24Infer configuration performs best overall; 32Infer avoids waiting for data but leaves resources underutilized during generation.Source: §3.2 · p. 7, Figure 3 · p. 7
• In 24Train16Infer, Async Ratio 1 gives 28% speedup, while ratio 2 gives 64% throughput gain; higher ratios add no improvement in that setting.The optimal ratio varies with generation throughput; ratio 2 suffices in most practical scenarios and is largely insensitive to model size.Source: §3.2 · p. 8
• At Async Ratios 2 and 8, evaluated off-policy methods and GRPO deliver gains on par with synchronous training, with comparable Pass@1 accuracy across benchmarks.Async variants slightly outperform Sync on Math500 and OlympiadBench and lag marginally on Minerva Math.Source: §3.2 · p. 8

4.1 Design Principles

• Users can run rollout and training workers as a pipeline on specified resources, choosing blocking or non-blocking model updates and adjusting asynchrony frequency.Source: §4.1 · pp. 8–9
• Fine-grained parallelism overlaps generation, environment interaction, and reward computation across samples, while prompt replication distributes generation across GPUs.Source: §4.1 · p. 9

4.2 Asynchronous Execution Workflow

• Within rollout, generation, environment work, and reward computation overlap; across stages, rollout and training run in parallel.Source: §4.2 · pp. 9–10
• LLMProxy coordinates inference workers, while each EnvManager mediates between the LLM and its environment until a trajectory terminates.Source: §4.2 · pp. 9–10
• EnvManagers enqueue trajectories in a shared SampleBuffer; AsyncController retrieves minibatches for training while rollout workers collect the next batch.Source: §4.2 · p. 10

4.3 Asynchronous Ratio

• The asynchronous ratio α limits how many policy versions old a sample may be, bounding the buffer at (1 + α) × batchsize samples without generating samples that violate freshness.Source: §4.3 · p. 10

5.1.1 Queue Scheduling

• At 8 × 8, queue scheduling with 16 redundant prompts reduces average per-step generation time from 125 seconds to 37 seconds (3.4× speedup).Source: Figure 7 · p. 11, §5.1.1 · p. 11

5.1.2 Prompt Replication

• At 64 × 16, prompt replication reduces latency from 149 seconds to 81 seconds (1.84× speedup).Source: Figure 8 · p. 12, §5.1.2 · p. 12

5.2 Agentic Pipeline

• Agentic trajectories face long-tail environment latency, motivating environment-level asynchronous rollout and redundant rollout to improve training efficiency.Source: §5.2 · p. 12

5.2.1 Environment-Level Asynchronous Rollout

• Environment-level asynchrony dispatches pending trajectories to available serving workers during environment interaction; with (10, 10) latency, step time falls from 892 seconds to 362 seconds (2.46×).Source: §5.2.1 · pp. 12–13, Figure 9 · p. 13

5.2.2 Redundant Environment Rollout

• On SWE, redundant rollout cuts training time from 6.09h to 5.65h (−7.2%) atop asynchronous rollout; on ALFWorld, it cuts 5.87h to 4.91h (−16.4%).Source: Figure 11 · p. 13, §5.2.2 · pp. 13–14
• At fixed rollout batch size 256, increasing environment groups is more effective than enlarging group size; 32 × 8 to 36 × 12 cuts step time from 243 seconds to 45 seconds (5.45×).Source: Figure 10 · p. 13, §5.2.2 · p. 13

6 Conclusion

• ROLL Flash extends ROLL with native support for asynchrony, grounded in fine-grained parallelism and rollout–training decoupling.Source: §6 · p. 14
• Its mechanisms include queue scheduling, prompt replication, asynchronous training, environment-level asynchronous rollout, and redundant environment rollout.Source: §6 · p. 14
• The report presents theoretical and empirical evidence for asynchronous training benefits and reports extensive experiments demonstrating ROLL Flash's efficiency.Source: §6 · p. 14

A.1 RLVR Pipeline

• DAPO-MATH-18K is the training dataset; evaluation uses MATH-500, OLYMPIADBENCH, MINERVAMATH, AMC 2023, AIME 2024, and AIME 2025.Source: §A.1 · p. 18
• The example selects PPO and GRPO-related reward settings, with queue scheduling, prompt replication, and asynchronous generation controlled by configuration flags.Source: §A.1 · p. 18
• The example uses Qwen/Qwen3-8B-Base, with SGLANG and VLLM generation backends and Megatron distributed training.Source: §A.1 · p. 18
• The configuration sets 256 prompts per rollout batch, 16 responses per prompt, one PPO epoch, 2,048-token prompts, and 30,720-token responses.Source: §A.1 · p. 18
• Sampling is fixed at temperature = 1 and top-p = 1 to preserve original token probabilities, limiting sampling-hyperparameter flexibility pending interface support.Source: §A.1 · p. 19

• Training uses R2E-Gym-Lite for SWE, ALFWorld-Train for ALFWorld, and ShopSimulator-SingleTurn for ShopSimulator; evaluation uses SWE-Bench-Verified, ALFWorld, and ShopSimulator-SingleTurn.Source: §A.2 · p. 19
• Default train and validation environment managers use group_size 16 with 8 groups and group_size 1 with 128 groups, respectively.Source: §A.2 · p. 19
• Redundant Env relaxes the group_size × num_env_groups = rollout_batch_size condition, using train settings 17 × 9 and validation settings 1 × 144.Source: §A.2 · p. 19
• The example configuration uses rollout_batch_size 128 and sequence_length 32768; async_generation_ratio 0 denotes Sync and values above 0 denote Async.Source: §A.2 · p. 19
• Environment limits are max_steps 50 and max_new_tokens 8192 for AlfworldEnv, and max_steps 30 and max_new_tokens 4096 for ShopSimulator.Source: §A.2 · p. 19