PaperlayerView exact revision on arXiv

Source-linked AI summary

ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learning

Seil Kang, Hangoo Kang, Tarun Suresh, Youngeun Kim, Shreyas Pimpalgaonkar, Seong Jae Hwang, Azalia Mirhoseini

arXiv:2610.05935v1cs.LGcs.AI

TL;DR

Agentic RL needs long, variable-duration trajectories, but synchronous learners wait for rollout and verification, while asynchronous overlap can introduce policy staleness. ThunderSyncRL streams gradients as objective inputs become ready while retaining synchronous updates. It reaches common pass@3 targets up to 1.9× faster than Sync and scores 1.27 to 2.47 points above Async at 128 GPU-hours.

  • Problem

    Agentic RL uses costly, heterogeneous trajectories, and synchronous learners wait for rollout and verification while asynchronous overlap can introduce policy staleness.

    Source: §1 · p. 2

  • Method

    ThunderSyncRL streams GRPO trajectory gradients after rewards arrive and OPD turn gradients after teacher scoring, then combines them for one synchronous optimizer step.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    On SWE-bench Verified and Terminal Bench 4.0, ThunderSyncRL reaches common pass@3 targets up to 1.9× faster than Sync and scores 1.27 to 2.47 points above Async at 128 GPU-hours.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ThunderSyncRL recovers idle learner time without changing the objective, shortening the time needed to turn the same compute into research evidence.

    Source: §5 · p. 11

    The benefit depends on workload and objective readiness, and starting work early requires retaining partial gradient state, exchanging trainer memory for wall time.

    Source: A Limitation and further discussion · p. 18

Abstract

from arXiv · show

Language models are moving beyond generating answers to pursuing long-horizon goals in interactive environments. Post-training these agents requires long, heterogeneous trajectories, and synchronous systems leave learner engines idle until rollout and verification finish. To squeeze out these pipeline bubbles, asynchronous training overlaps rollout and learning across updates, but comes at the cost of policy staleness. We introduce ThunderSyncRL, which starts gradient computation as soon as all required inputs are fixed, without policy staleness. For group relative policy optimization (GRPO), ThunderSyncRL computes each trajectory's score gradient as soon as the reward for that trajectory arrives, without waiting for the group. For on-policy distillation (OPD), it computes gradients for each completed agentic turn's teacher-scored actions while tool calls run in the sandbox. We prove that gradient streaming produces the same GRPO and OPD updates as batch-synchronous training, without changing either objective. On SWE-bench Verified and Terminal Bench 4.0, we train models to the same performance up to $1.9 \times$ faster than synchronous training. With zero policy staleness, ThunderSyncRL also outperforms asynchronous training at a fixed budget by up to $2.47$ percentage points.

A Blog post § GitHub

Figure 1 compares ThunderSyncRL and Sync on SWE-bench Verified pass@3, including training-cost and fixed-budget comparisons. The front matter defines three contributor marks.

A Blog post § GitHub

  • Figure 1 reports that ThunderSyncRL matches Sync’s SWE-bench Verified pass@3 at up to 1.9× lower training cost and improves pass@3 by up to 2.47 percentage points over asynchronous baselines at 128 GPU-hours.The GRPO results are shown in the left pair and OPD in the right pair.Source: Figure 1 · p. 1
  • The contributor marks denote core contributor (*), co-corresponding author (†), and main corresponding author (‡).Source: A Blog post § GitHub · p. 1

1 Introduction

Synchronous agentic RL leaves learners waiting for rollout and verification, while asynchronous overlap can introduce policy staleness. ThunderSyncRL streams ready gradients and retains synchronous updates, reporting faster training to common pass@3 targets and higher fixed-budget pass@3 than Async.

1 Introduction

  • Synchronous learners wait for rollout and verification, while asynchronous execution overlaps updates but can use trajectories from older policies.The introduction reports learner GPUs idle for 49.5% of synchronous step time and notes prior reports of idle fractions as high as 81%.Source: §1 · p. 2
  • ThunderSyncRL starts GRPO backward passes when each trajectory’s reward arrives and releases OPD turn-level passes after teacher scoring, then combines gradients for one optimizer step.The schedule streams ready gradients while preserving synchronous policy updates.Source: §1 · p. 2, Figure 2 · p. 3
  • The schedule analysis predicts step-wall-time speedup approaching 2× when rollout and learner work are balanced and residual tail and publication costs vanish.Source: §1 · p. 2
  • ThunderSyncRL derives exact gradient reductions for GRPO and turn-level OPD, enabling early backward computation while preserving batch-synchronous updates.Source: §1 · p. 2
  • On SWE-bench Verified and Terminal Bench 4.0, ThunderSyncRL reaches common pass@3 targets up to 1.9× faster than Sync and scores 1.27 to 2.47 points above Async at 128 GPU-hours.The reported comparisons retain zero policy staleness.Source: §1 · p. 2

2 Related Work

Related work spans agentic post-training methods and systems that reduce idle time through overlapping or scheduling computation. ThunderSyncRL schedules learner work as objective inputs become ready while preserving a fixed policy snapshot and synchronous optimizer step.

2.1 Post-training with Reinforcement Learning Towards Agentic Tasks

  • Agentic post-training work includes hierarchical multi-turn RL in ArCHer and combined offline-online RL for mobile-device control in DigiRL [ZZP+24] [BZC+24].Source: §2.1 · p. 3

2.2 Optimization of Reinforcement Learning

  • PipelineRL and RolloutPipe pipeline rollout and learner execution for heterogeneous agentic rollouts [PKP+25] [CHY+26].Source: §2.2 · p. 3
  • In SWE-smith GRPO training, synchronous learners wait for the last reward for 49.5% of each step; ThunderSyncRL completes 15 optimizer steps in the GPU-hours Sync needs for 8.Figure 3 plots four rollout GPUs above the axis and four learner GPUs below, with hatched bands marking idle time.Source: Figure 3 · p. 4
  • ThunderSyncRL releases learner work as each objective source becomes ready, unlike systems that pipeline rollout and learning or control asynchronous weight synchronization and policy staleness.The agent, reward, and optimization objective remain unchanged.Source: §2.2 · pp. 3–4

3 Methods

ThunderSyncRL releases learner work as soon as its objective dependencies are fixed, while keeping rollouts and gradients bound to the same policy and preserving a single synchronous optimizer step. Its GRPO and OPD procedures stream eligible gradient work before the full group or batch is ready, then reduce it into the batch update.

3 Methods

  • ThunderSyncRL uses exact gradient reduction and a synchronous optimizer barrier to preserve GRPO and OPD update semantics while overlapping rollout and learning.All sources in update k use θk; parameters are published only after the batch update and optimizer step.Source: §3 · p. 4, §3.3 · p. 5

3.1 Background

  • Synchronous training waits for all trajectories and rewards before learner work, while asynchronous training overlaps updates but can use rollouts from older policies.ThunderSyncRL instead releases work when its mathematical dependencies are satisfied.Source: §3.1 · p. 4

3.2 Problem Formulation

  • An objective source is an independently releasable learner unit, ready once its inputs are fixed; GRPO sources are trajectories and OPD sources are turns.For GRPO, group statistics still await all rewards; for OPD, teacher scores release the student backward pass.Source: §3.2 · p. 5

3.3 ThunderSyncRL Framework

  • Ready sources enter a work-conserving queue, and ThunderSyncRL moves eligible learner work into rollout intervals while retaining a final batch-update barrier.It waits for all registered work and deferred gradient propagation before reduction, clipping, and one optimizer step.Source: §3.3 · p. 5

3.4 Objective-Specific Gradient Streaming

  • For GRPO, each trajectory’s backward pass starts when its reward arrives, and exact group-gradient recovery waits only for group closure.The trajectory score-gradient contributions are combined with group-normalized advantages once the required group statistics are available.Source: §3.4 · p. 6
  • For OPD, turn-level student backward passes start once frozen-teacher scores are available, overlapping learner computation with tool execution.ThunderSyncRL defers total scored-token normalization until the logical batch closes, then produces the same mathematical update as batch-synchronous OPD.Source: §3.4 · p. 6

4 Experiments

Experiments use Qwen and GLM models, SWE-smith and SETA-Env post-training, and SWE-bench Verified and Terminal Bench 4.0 evaluation; Terminal Bench tasks are excluded from post-training. Across these settings, ThunderSyncRL improves training efficiency, with scaling dependent on workload and stage balance.

4.2 Learning efficiency and agent performance

  • ThunderSyncRL reaches matched pass@3 targets up to 1.9× faster than Sync on SWE-bench Verified and 1.8× faster under GRPO and 1.9× faster under OPD on Terminal Bench 4.0.For GRPO on SWE-bench Verified, the 74.01% target is reached at 121.3 GPU-hours versus 233.5 GPU-hours for Sync.Source: §4.2 · p. 8, Figure 5 · p. 8
  • At 128 GPU-hours, ThunderSyncRL exceeds Async pass@3 by 2.47 and 1.27 percentage points on SWE-bench Verified, and 1.52 and 2.02 points on Terminal Bench 4.0, under GRPO and OPD respectively.Scores use the latest completed checkpoint available within the budget.Source: §4.2 · p. 8, Figure 5 · p. 8
  • ThunderSyncRL steps take 687.7 s under GRPO and 685.7 s under OPD, 1.91× and 1.94× faster than Sync while retaining zero policy lag.After rollout and verification, Sync adds 646.4 s and 656.2 s of learner work, whereas ThunderSyncRL adds 20.5 s and 11.6 s.Source: §4.2 · p. 8

4.3 Efficiency across models and hardware

  • On B200, speedups over Sync are 1.91× and 1.94× for Qwen under GRPO and OPD, versus 1.64× and 1.63× for GLM; on H200, Qwen reaches 1.63× and 1.67×.The comparisons hold workload and GPU allocation fixed against Sync on the same hardware.Source: §4.3 · p. 9, Table 1 · p. 9, Table 2 · p. 10
  • ThunderSyncRL has the highest trainer active MFU in every configuration, from 18.7% for GLM to 51.0% for Qwen on H200, but uses more peak GPU memory than Sync.For Qwen GRPO, peak learner-GPU memory ranges from 117.0 to 122.8 GB; pinned host RAM for the FP32 accumulator and GRPO statistics is excluded.Source: §4.3 · p. 9

4.4 Workload scaling and overlap efficiency

  • With balanced rollout and learner times, complete overlap can approach 2× speedup when tail and update overhead are negligible; as trajectory-duration spread grows, GRPO rises from 1.58× to 1.95×, while OPD stays at 1.73×–1.94×.The scaling measurements cover more than 150 steps per objective at balance.Source: §4.4 · pp. 9–10
  • As batch size grows, the learner tail and publication take a smaller share and predicted speedup approaches 2×; with fixed completion-time distributions, more OPD turns shorten the tail, while GRPO speedup does not change with turn count.OPD gains diminish as deferred work limits further tail reduction.Source: §4.4 · p. 10
  • Across 588 paired scaling configurations, release-trace predictions differ from analytical predictions by 1.6% on average and at most 10.6%; measured wall-clock speedups average 6.5% below trace predictions.These differences represent release-model error and remaining runtime overhead, respectively.Source: §4.4 · p. 10, Figure 8 · p. 11

5 Conclusion

ThunderSyncRL overlaps rollout and learning without off-policy staleness while preserving synchronous GRPO and OPD updates. Its step-time speedup can approach 2× under favorable stage balance, shortening training loops at fixed compute.

5 Conclusion

  • ThunderSyncRL streams ready gradients at a fixed policy snapshot, then takes one optimizer step after the batch closes, preserving synchronous GRPO and OPD updates.GRPO uses gradient sufficient statistics to start backward passes before all group rewards arrive; OPD uses teacher-scored turns during tool execution.Source: §5 · p. 11
  • Step wall-time speedup can approach 2× when rollout and learner work are balanced and the learner tail and update overhead are negligible.Source: §5 · p. 11
  • Recovering idle learner time without changing the objective shortens the research loop, allowing the same compute to answer more research questions.Source: §5 · p. 11

A Limitation and further discussion

The discussion frames early learner work as dependent on when objective inputs become ready, with memory costs from retaining partial gradient state. The appendices define efficiency metrics, their accounting windows, hardware allocation, and memory measurements.

A Limitation and further discussion

  • Earlier readiness can increase overlap, but partial gradient state must remain until group or batch closure, trading trainer memory for wall time.The authors identify extending exact gradient streaming to other objectives, multi-node training, and broader agentic workloads as future work.Source: A Limitation and further discussion · p. 18

B.1 Trainer active MFU

  • Trainer active MFU is learner FLOPs divided by complete update wall time and trainer-GPU peak capacity, with elapsed-time accounting spanning rollout, scoring, learning, and publication.The FLOP ledger counts executed model operations and contexts, while rollout generation and teacher-scoring FLOPs are excluded from the trainer-side numerator.Source: §B.1 · p. 18

B.2 Average trainer active MFU

  • Average trainer active MFU uses each update's learner FLOPs and complete wall time; shorter updates raise the metric without requiring higher kernel throughput.The reported window profiles updates 5–40 after discarding four startup updates, across three seeds.Source: §B.2 · p. 19

B.3 Step wall time speedup

  • Step wall time covers a complete update through rollout-policy acknowledgement, and speedup is the ratio of mean step times under matched workload and hardware.Async measurements use complete steady-state cycles with the declared one-update policy lag.Source: §B.3 · p. 19

B.4 Hardware allocation and memory

  • Each eight-GPU setup assigns four GPUs to the learner; GRPO uses four rollout GPUs, while OPD uses two rollout and two teacher GPUs.Peak GPU memory is measured on learner GPUs after persistent state is resident; pinned host-memory accumulators are excluded from that GPU counter.Source: §B.4 · pp. 19–20

C.1 Data sources and versions

The experiments use pinned SWE-smith and SETA-Env workloads, with explicit filtering, validation, and benchmark-separation checks. The schedules share objective-specific training controls, agent interfaces, and hardware while differing in execution schedule.

C.1 Data sources and versions

  • SETA-Env supplies 4,567 synthesized and evolved terminal-task directories; the selected maintained Python SWE-smith release contains 50,908 instances from 131 repositories.These are inspected source-pool sizes before filtering and development splitting.Source: §C.1 · p. 20, Table 3 · p. 20

C.2 Filtering and development splits

  • Filtering retains 39,471 SWE-smith instances and all 4,567 SETA tasks, then assigns 5% of repository or source-family groups to development.Executable validation checks SWE-smith before and after applying its bug-inducing patch, and checks SETA no-op and reference solutions.Source: §C.2 · p. 21, Table 4 · p. 21

C.3 Scope of benchmark separation

  • Excluding benchmark task identifiers and checking repository disjointness do not establish semantic decontamination or that every task was unseen during model development.The protocol excludes identified derivatives, but upstream generation or model pretraining may still expose benchmark content.Source: §C.3 · p. 22

D.1 Logical batches and optimization

  • Each logical update uses one optimizer step, with group-normalized advantages held fixed during GRPO differentiation and OPD reduced using a global scored-token denominator.The comparison fixes shared optimization settings across schedules within each objective.Source: §D.1 · pp. 22–23, Table 5 · p. 22

D.2 Agent interaction and generation

  • Agent rollouts alternate model generation with shell-tool execution, while independent trajectories run concurrently and shared vLLM serving settings apply across schedules.ThunderSyncRL changes when ready learner work is admitted; shared serving optimizations include chunked prefill and automatic prefix caching.Source: §D.2 · pp. 23–24, Table 6 · p. 23

D.3 Teacher and hardware allocation

  • The reference setup uses eight B200 GPUs; learner work is distributed across four GPUs, with objective-specific allocations for rollout and teacher scoring.The admission cap k = 1 bounds statistics and batch-accumulator host memory across learner ranks at 1.30 TB for Qwen and 1.49 TB for GLM.Source: §D.3 · pp. 24–25

D.4 Comparison controls

  • Sync and ThunderSyncRL use one policy snapshot for rollout and learning, whereas the controlled Async baseline trains on trajectories sampled one optimizer update behind.Completion order changes admission, not batch membership; every scheduled trajectory and scored turn completes before the update.Source: §D.4 · p. 25

E.1 Benchmarks and success criteria

The evaluation defines benchmark success, pass@3 aggregation, and matched-budget comparisons across execution schedules. Its scheduling analysis relates learner-tail time to when work becomes ready and shows how earlier releases and repeated rollout waves affect overlap speedup.

E.1 Benchmarks and success criteria

  • SWE-bench Verified evaluates all 500 tasks, while Terminal Bench 4.0 evaluates all 66 release tasks; Terminal Bench tasks are excluded from post-training.Scores reflect the specified agent interface and inference budget, so they are not directly interchangeable with leaderboard results from other settings.Source: §E.1 · pp. 25–26

E.2 Attempts, aggregation, and uncertainty

  • Pass@3 is the fraction of tasks solved at least once across three independent attempts, with each task weighted equally.Three evaluation attempts from one checkpoint are not three independent training runs.Source: §E.2 · p. 26

E.3 Checkpoints and GPU-hours comparisons

  • Schedules are compared by time to a fixed benchmark-specific pass@3 target and pass@3 at 128 GPU-hours, using matched tasks and evaluation settings.Checkpoint snapshots are evaluated on a fixed cadence, and the fixed-budget result uses the latest completed checkpoint published by 128 GPU-hours.Source: §E.3 · pp. 26–27

F.1 Step time and the learner tail

  • The modeled overlap speedup is at most 2×, reached only with equal rollout and learner stage times, negligible suffix, and no learner tail.The comparison assumes fixed job service times and enough device memory to admit sources as they become ready.Source: §F.1 · pp. 27–28

F.2 How work release determines the tail

  • Learner-tail time lies between the unavoidable service excess [ρ − 1]+ and that excess plus ρD, where D is the maximum lag behind uniform service release.With OPD work released at each teacher-scored turn and no deferred work, the modeled release fraction yields the minimum tail [ρ − 1]+; finite horizons can realize less benefit.Source: §F.2 · pp. 28–29

F.3 Batch size and successive rollout waves

  • For repeated full rollout waves, the modeled speedup approaches 2 at balance as wave count grows, while away from balance it approaches (1 + ρw) / max(1, ρw).Continuous refill and partially occupied waves require their actual release trace or an explicit approximation.Source: §F.3 · p. 29

F.4 Finite batches and model limitations

The appendix analyzes finite-batch release approximations, compares model predictions with measured runtimes, and sweeps rollout-to-learner balance. The predicted speedup peaks near balance, while batch size affects the peak more than horizon.

F.4 Finite batches and model limitations

  • Midpoint-quantile finite batches have discrepancy at most 1/(2n) and tail error at most ρ/(2n); replacing random releases by their mean is optimistic for completion time.The endpoint is a maximum of affine functions of the release distribution, so its expected value is at least the endpoint computed from the mean distribution.Source: §F.4 · p. 29
  • Static learner queues must be modeled per rank and completed at the latest rank; admission delays, packing waits, unequal turn costs, and teacher delays also belong in the trace.When concurrent rollout changes kernel service or methods execute different physical work, predictions should use service times measured under those conditions.Source: §F.4 · p. 29

F.5 Figure parameters and comparison with measurements

  • The comparison covers 588 paired configurations; within each, Sync and ThunderSyncRL use the same model, logical workload, GPU roles, precision, and generation settings.The plotted ratio uses measured stage times for the selected GPU allocation.Source: §F.5 · p. 30
  • Across both objectives and rows, mean e_i is 1.5911%, maximum e_i is 10.5541%, and mean d_i is 6.5148%.e_i measures release-model agreement with trace prediction; d_i measures the remaining gap to runtime execution.Source: §F.5 · p. 30

F.6 Balance sweeps

  • 1.84× (GRPO, 256 trajectories per step) to 2.00× (OPD, 2,048 trajectories per step) are predicted near r≈1; at B=64, peaks shift near r≈0.5.At B=64, the predicted peaks are 1.71× for GRPO and 1.95× for OPD.Source: §F.6 · p. 31
  • At 256 trajectories per step, OPD speedup at r=1 changes only from 1.967× at 10 turns to 1.975× at 300 turns, while GRPO is unchanged by turn count.The horizon effect is limited to near balance; larger batches widen the near-balance speedup band by amortizing the final tail and suffix.Source: §F.6 · p. 31, Figure 9 · p. 31

G Empirical update equivalence

Replay experiments compare ThunderSyncRL and Sync on identical logical batches, finding matching updates within training precision. The appendix also describes five-attempt pass@1 and pass@5 evaluation against training cost.

G Empirical update equivalence

H.1 Pass@1 and Pass@5

  • Pass@k estimates solving a task at least once in k attempts; five independent attempts per checkpoint provide pass@1 and pass@5 scores plotted against training cost.Pass@1 is the mean attempt success rate; pass@5 credits a task solved in any of the five attempts.Source: §H.1 · p. 35, Figure 13 · p. 35

H.2 Efficiency with LoRA

LoRA preserves ThunderSyncRL’s efficiency ordering over Sync and Async, with measured step speedups and zero policy lag. Freezing base weights reduces the learner work available to overlap, while ThunderSyncRL uses less GPU memory than full fine-tuning.

H.2 Efficiency with LoRA

  • 16.7% to 33.0% trainer active MFU for ThunderSyncRL is slightly above Async’s 16.5% to 32.6% and above Sync’s 10.3% to 19.5%.Trainer active MFU is measured over the complete update wall time, including rollout waiting.Source: §H.2 · pp. 35–37
  • 1.69/1.71× for Qwen and 1.64/1.62× for GLM are ThunderSyncRL step wall-time speedups over Sync under GRPO/OPD, with zero policy lag in every configuration.For comparison, Async speedups are 1.67/1.69× for Qwen and 1.62/1.60× for GLM, respectively.Source: Table 7 · p. 36, §H.2 · p. 36
  • Freezing base weights removes parameter-gradient and optimizer work, reducing learner work available to overlap; LoRA speedups are smaller for Qwen than full fine-tuning and barely change for rollout-bound GLM.Forward and input-gradient computation through frozen weights remains necessary.Source: §H.2 · p. 36
  • LoRA lowers trainer GPU memory versus full fine-tuning; ThunderSyncRL peaks at 75.0 to 79.2 GB per learner GPU, adding 1.9 to 3.8 GB relative to Sync.Measurements include frozen weights, adapter and optimizer state, activations, and temporary buffers; separate rollout and teacher pools are excluded.Source: §H.2 · p. 37

I ThunderSyncRL algorithm and implementation

ThunderSyncRL releases learner work as sources become ready at a fixed policy snapshot, then completes all deferred work before one synchronous optimizer step. GRPO releases trajectory gradients after rewards arrive, while OPD releases turn gradients after teacher scoring.

I ThunderSyncRL algorithm and implementation

  • During one update, rollout and learner threads share fixed policy snapshot θk; eligible source work is released as events arrive, but the batch barrier precedes one optimizer step and publication.Shared-prefix computation is reused across schedules, while ThunderSyncRL admits eligible branch work earlier and retains the shared-prefix backward pass.Source: I ThunderSyncRL algorithm and… · pp. 37–38

I.1 GRPO

  • GRPO computes each reward-ready trajectory’s score gradient independently, accumulates reward-weighted and unweighted group sums, and combines them when the group closes.The summed group directions are reduced and clipped before exactly one ascent step at the logical-batch barrier.Source: §I.1 · p. 38

I.2 On-policy distillation

  • OPD releases backward computation for each teacher-scored turn using its realized causal context, then replays deferred context gradients and normalizes by the global scored-token count before one descent step.The teacher is frozen, and the student scores come from the same fixed policy snapshot.Source: §I.2 · p. 39

Related papers

Loading 2610.05935v1…