Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

1 Introduction

• Synchronous RL waits for complete rollout batches, leaving GPUs idle behind long agentic trajectories; asynchronous RL consumes arrivals continuously to improve utilization and wall-clock efficiency.Variable rollout lengths make this bottleneck particularly relevant to agentic and coding workloads.Source: §1 · p. 2
• Mixed-policy trajectories threaten stability, while GRPO [Shao et al., 2024, Wang et al., 2022] adds group-waiting delays and conflicts with single-feedback environments.Asynchronous trajectories can contain outputs from multiple rollout-model versions, worsening off-policy effects. Earlier asynchronous RL work [Fu et al., 2025, Noukhovitch et al., 2024] primarily emphasized efficiency rather than effectiveness.Source: §1 · p. 2
• SAO replaces group-wise sampling with one rollout per prompt to reduce off-policy effects, supported by targeted value-model training strategies.The critic is updated more frequently than the actor, and the value model is fine-tuned with frozen attention.Source: §1 · p. 2
• Token-level importance sampling uses rollout-engine log-probabilities with stricter double-sided clipping and masking to stabilize training under varying policy lag.SAO targets stable, effective training while preserving the efficiency of asynchronous execution.Source: §1 · p. 2
• A skip-observation token-level GAE estimator computes advantages across action-to-action boundaries, avoiding noise propagation through environment observation tokens not generated by the model.This supports multi-turn agent trajectories with interleaved environment feedback.Source: §1 · p. 2
• Around one thousand steps of stable training are achieved by SAO, alongside consistently better performance than improved GRPO on coding and math reasoning benchmarks.Evaluation includes SWE-Bench Verified [Jimenez et al., 2023], AIME2025 [Balunovi´c et al., 2025], BeyondAIME, HMMT [Balunovi´c et al., 2025], and IMOAnswerBench [Luong et al., 2025]. Simulated online learning also demonstrates adaptation to changing environments.Source: §1 · p. 2

2 Preliminaries

• Language-model RL treats response generation as a stochastic policy and maximizes a clipped surrogate objective to encourage stable updates.The objective uses a current-to-old policy probability ratio and a clipping hyperparameter.Source: §2 · pp. 2–3

Proximal Policy Optimization (PPO)

• PPO [Schulman et al., 2017] and GRPO [DeepSeek-AI, 2024b] differ in their advantage estimation and auxiliary value-network requirements.Standard PPO trains a separate critic concurrently with the policy, minimizing squared error between predicted value and cumulative reward.Source: §2 · p. 3
• Generalized Advantage Estimation balances bias and variance through an exponentially weighted sum of temporal-difference errors.Maintaining the value-function model essentially doubles training memory and increases computational overhead.Source: §2 · p. 3

3.1 Stabilizing Asynchronous RL via Direct Double-Sided Importance Sampling (DIS)

• Direct Double-Sided Importance Sampling computes policy-to-rollout probability ratios from rollout logs and excludes tokens outside its trust region from gradient computation.Multiple rollout-model updates within trajectories make exact historical-policy tracking impractical. Removing separate old-policy inference accepts controlled off-policy bias while reducing computation and improving training stability through stricter masking.Source: §3.1 · pp. 3–4

3.2 Reducing Off-Policy with Single Rollout

• Single-rollout sampling feeds each completed trajectory directly into training, avoiding group-wise sampling’s imbalanced-generation bias and waiting for the slowest sample.Single-rollout optimization introduces high-variance gradient estimates, motivating stronger value modeling.Source: §3.2 · p. 4
• 2 value-network updates per policy update let value estimates adapt faster, reducing noisy advantages and destructive policy updates.The general strategy decouples optimization frequencies, requiring K > 1 value updates for each policy update.Source: §3.2 · p. 4
• Frozen-Attention training freezes the value model’s attention parameters and optimizes only MoE projections to regularize value learning.Pilot experiments traced unusually large value-model gradient norms primarily to Full Attention layers, while MoE layers remained relatively stable. The authors hypothesize pretrained attention already supplies sufficient capability to attend to relevant tokens.Source: §3.2 · pp. 4–5
• Skip-Observation GAE links consecutive model actions directly, bypassing environment-feedback tokens when constructing token-level advantage estimates.Action-to-observation boundaries introduce noise because the model does not generate observations. The authors also report that step-level value estimation can produce suboptimal performance.Source: §3.2 · p. 5
• Scaling the value-pretraining corpus addresses a major cold-start bottleneck and strengthens single-rollout learning and faster value updates from early training.The larger corpus provides a more robust initialization for value estimation.Source: §3.2 · p. 5

4.1 Experimental Setup

• Evaluation covers four math benchmarks using Pass@1 and SWE-Bench Verified using OpenHands, with a 128k-token generation limit.SAO uses one rollout per prompt versus eight for GRPO variants, with both using batch size 128. Math evaluations permit up to 50 interaction turns, versus 300 for coding, and average performance across 4–16 runs depending on the math benchmark.Source: §4.1 · p. 6

4.2 Main Results

• SAO consistently outperforms all baselines on both agentic reasoning and coding benchmarks.Standard GRPO uses the clip-higher implementation Yue et al. [2025] and collapses around 160 training steps. DIS stabilizes GRPO, but its initially comparable performance diverges from SAO after approximately 400 steps.Source: §4.2 · pp. 6–7

4.3 Ablation Studies

• Every tested ablation underperforms SAO, supporting faster critic updates, frozen-attention value training, and a learned value baseline.Variants use one rather than two critic updates per batch, full-parameter critic training, or an eight-reward per-prompt running-mean baseline. Vanilla VAPO[Yue et al., 2025] also underperforms and quickly collapses.Source: §4.3 · p. 7

4.4 Training Dynamics

• After approximately 400 training steps, SAO achieves significantly higher explained variance than the single-critic-update baseline, indicating faster value convergence and better policy-distribution alignment.Explained variance measures alignment between predicted values and ground-truth returns.Source: §4.4 · p. 7, Figure 4 · p. 7
• Frozen-attention value training maintains lower, smoother critic gradient norms than full-parameter training, suggesting improved numerical stability.Full-parameter critic optimization exhibits substantially larger gradient norms, suggesting unstable optimization dynamics.Source: §4.4 · pp. 7–8, Figure 4 · p. 7
• Approximately 90 steps into training, VAPO collapses despite a near-zero token clip ratio, which fails to gate divergent off-policy updates.The comparison monitors SAO with DIS against standard VAPO without DIS.Source: Figure 4 · p. 7, §4.4 · p. 8

4.5 Online Learning Simulation

• SAO rapidly adapts to changing writing-style preferences, recovering rewards faster and reaching higher convergence levels than the Running Mean baseline.The simulation shifts preferences from cute to chuunibyou to classical; GLM-4.7[GLM et al., 2025] awards reward only when quality and style both pass. SAO’s state-dependent critic tracks shifts, whereas the baseline’s 128-reward window retains outdated feedback; value-based estimation permits single-trajectory updates without sample groups.Source: §4.5 · pp. 8–9, Figure 5 · p. 9

5.1 Reinforcement Learning for Language Models

• GRPO [Shao et al., 2024, DeepSeek-AI, 2024a] normalizes rewards within prompt-level groups, creating synchronization barriers that exacerbate staleness and off-policy drift under asynchrony.Standard RLHF learns a preference-based reward model and optimizes policies with PPO [Ouyang et al., 2022, Schulman et al., 2017]. Critic-free objectives, including GRPO and RLOO [Ahmadian et al., 2024], reduce value-learning overhead and instability, but GRPO updates must await every group member.Source: §5.1 · p. 9
• Recent objective refinements primarily address synchronous RL, leaving asynchronous importance sampling and clipping less explored.Examples include sequence-level importance weighting [Zheng et al., 2025], adaptive clipping [Yang et al., 2025b], and smoother alternatives to hard clipping [Yue et al., 2025]. Exact importance-sampling ratios are easier to obtain in the synchronous setting.Source: §5.1 · p. 9

5.2 Synchronous and Asynchronous RL for LLMs

• Synchronous rollout batches suffer stragglers and idle time with long-tail outputs, while concurrent actor–learner designs introduce policy lag and off-policy drift.Most large-scale implementations alternate fixed-policy batch collection with optimization [Ouyang et al., 2022]; asynchronous actor–learner designs overlap generation and learning [Mnih et al., 2016, Sutton and Barto, 2018]. Their policy lag often requires staleness-aware training or off-policy corrections [Espeholt et al., 2018].Source: §5.2 · p. 10
• The paper complements asynchronous LLM RL systems by targeting single-rollout learning, where group-based baselines are structurally mismatched, and stabilizing asynchronous optimization.Noukhovitch et al. [2024] examine online-but-off-policy RLHF robustness; AReaL [Fu et al., 2025] decouples rollout and training with staleness-aware PPO-style updates; ROLL Flash adds fine-grained parallelism and rollout–train decoupling [Lu et al., 2025]. MobileRL instead studies mobile GUI agents, introducing difficulty-adaptive GRPO variants for stability and sample efficiency in multi-turn environments [Xu et al., 2025].Source: §5.2 · p. 10

6 Conclusion

• SAO consistently outperforms GRPO baselines on agentic reasoning and coding tasks.Source: §6 · p. 10
• SAO addresses off-policy effects and training instability using token-level importance sampling and double-sided clipping/masking.Source: §6 · p. 10
• SAO improves generalization by replacing group-wise sampling with single-rollout sampling, enabled by stronger value-model training.Source: §6 · p. 10
• SAO adapts effectively in simulated online learning.Source: §6 · p. 10

A.1 RL with Agentic Step as Action

• Step Average defines each step’s value as the average of its token value predictions and trains the value model on all tokens.Source: §A.1 · p. 13
• Last-Token Prediction uses the final token’s value as the step value and masks intermediate tokens from the value-model training loss.This assumes the final token provides the most accurate step-value prediction because it contains the most comprehensive information about the step.Source: §A.1 · p. 13
• Step-wise GAE treats each conversation turn as one step and assigns its tokens a shared advantage to reduce token-level prediction noise.A single advantage per step is derived from the step-level temporal-difference error using step rewards and successive step values.Source: §A.1 · p. 13
• Length-adaptive GAE scales its decay factor λ using the total number of conversation steps rather than raw token length.Source: §A.1 · p. 13
• The authors attribute step-wise underperformance to losing fine-grained critic and policy supervision needed to capture logical transitions within complex reasoning trajectories.Source: §A.1 · p. 13
• Token-level value training outperforms both step-wise variants in benchmark performance and training reward.Table 5 compares performance after the same 400 training steps; Figure 6 shows training rewards.Source: Table 5 · p. 13, §A.1 · p. 13, Figure 6 · p. 13

• SAO outperforms SPO Xu and Ding [2025] and historical running-mean reward baselines in the reported experiments.Source: §A.2 · p. 14
• SPO Xu and Ding [2025] and running-mean baselines rely on prior information about training-data difficulty.Source: §A.2 · p. 14
• SPO Xu and Ding [2025] and using historical running-mean reward as the advantage-estimation baseline both support RL with one rollout per prompt.Source: §A.2 · p. 14

B Limitations and Broader Impact

• SAO’s conclusions may not transfer directly to smaller models, non-agentic RLHF, or environments with dense rewards and shorter rollouts.Experiments focus on large-scale agentic reasoning, coding, and simulated online writing using a Qwen3-30B-A3B backbone.Source: B Limitations and Broader Impact · p. 14
• Real user-facing online adaptation would require stronger safeguards, monitoring, and privacy review than the controlled simulated preference-shift study provides.The online learning study evaluates a controlled simulated preference shift rather than real user-facing adaptation.Source: B Limitations and Broader Impact · p. 14
• SAO requires a trained value model and infrastructure that reliably preserves token-level behavior probabilities during asynchronous generation.Deployment depends on retaining rollout log-probabilities.Source: B Limitations and Broader Impact · p. 14
• More stable and efficient LLM reinforcement learning can reduce agentic training costs but could also facilitate optimization for harmful objectives.The authors emphasize data filtering, access controls, evaluation, responsible release, and monitoring for deployed systems derived from this work.Source: B Limitations and Broader Impact · p. 14