PaperlayerView exact revision on arXiv

Source-linked AI summary

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Zhenyu Hou, Yujiang Li, Jie Tang, Yuxiao Dong

arXiv:2607.07508v1cs.LGcs.AI

TL;DR

Asynchronous RL for long-horizon agentic tasks faces policy-lag, stability, and task-effectiveness challenges, while group-wise sampling does not naturally fit asynchronous training. SAO uses single-rollout updates with value-model training and strict token-level clipping, achieving stable training for around 1,000 steps and consistently outperforming GRPO variants across coding and reasoning benchmarks.

  • Problem

    Asynchronous RL for agentic tasks lacks well-established evidence on training stability and task effectiveness under policy lag and asynchronous group-wise sampling.

  • Method

    SAO combines single-rollout sampling with value-model training and strict double-sided token-level importance-ratio clipping to stabilize asynchronous RL.

  • Results

    SAO trains stably for around 1,000 steps and consistently outperforms GRPO baselines on agentic coding and math reasoning benchmarks.

  • Takeaways & Limitations

    Single-rollout asynchronous RL is particularly effective for adapting to changing environments in simulated online learning.

  • Takeaways & Limitations

    The findings may not transfer directly to smaller models, non-agentic RLHF settings, or dense-reward environments with shorter rollouts.

Abstract

from arXiv · show

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).

1 Introduction

Asynchronous RL improves utilization for variable-length agentic rollouts but introduces policy-lag off-policy instability and mismatches group-wise sampling. SAO addresses these challenges with single-rollout updates, practical value-model training, and stricter token-level stabilization mechanisms.

  • 1 Introduction: RL is increasingly used for LLM post-training, while most pipelines remain synchronous and interleaved.These pipelines generate rollout batches before starting optimization.
  • 1 Introduction: Variable rollout lengths cause stragglers that leave substantial GPU capacity idle in agentic and coding workloads.Asynchronous RL consumes rollouts continuously as they arrive, improving utilization and wall-clock efficiency.
  • 1 Introduction: Asynchrony creates unpredictable off-policy effects from multiple rollout-model versions and makes GRPO-style group-wise methods difficult to apply.These issues threaten training stability and arise alongside prior work’s emphasis on efficiency rather than effectiveness.
  • 1 Introduction: SAO preserves asynchronous efficiency while targeting stable, effective training under policy lag through single-rollout updates and value-model training strategies.It replaces group-wise sampling, such as GRPO, with one rollout per prompt.
  • 1 Introduction: SAO stabilizes optimization with token-level importance sampling, stricter double-sided clipping, and masking using rollout-engine log-probabilities.The design directly uses log-probabilities from the rollout engine.
  • 1 Introduction: Around one thousand steps, SAO trains stably and consistently outperforms improved GRPO across agentic coding and math reasoning benchmarks.The evaluation includes SWE-Bench Verified, AIME2025, BeyondAIME, HMMT, and IMOAnswerBench.

2 Preliminaries

The preliminaries formulate language-model reinforcement learning as clipped policy optimization over response sequences generated by a stochastic policy. They distinguish PPO from GRPO through advantage estimation and auxiliary value networks, then describe PPO’s critic and GAE machinery and its memory overhead.

  • Language-model reinforcement learning: RL models responses as sequences y generated by a stochastic policy πθ(y|q) for queries q sampled from dataset D.Optimization maximizes a clipped surrogate objective intended to encourage stable policy updates.
  • Clipped policy optimization: The probability ratio rt(θ) compares current and old policies, while clipping is controlled by hyperparameter ϵ.PPO and GRPO fundamentally differ in advantage estimation and whether auxiliary value networks are required.
  • Proximal Policy Optimization (PPO): PPO uses an Actor-Critic architecture with a separately parameterized value function Vϕ that estimates expected returns and is trained alongside the policy.The critic minimizes value error against cumulative reward R.
  • Proximal Policy Optimization (PPO): PPO computes advantages with Generalized Advantage Estimation as an exponentially weighted sum of temporal-difference errors δt.The temporal-difference error is δt = rt + γVϕ(st+1) − Vϕ(st).
  • Proximal Policy Optimization (PPO): Maintaining a separate value-function parameter copy essentially doubles training memory and increases computational overhead.This overhead follows from concurrently training the critic alongside the policy.

3 Asynchronous Reinforcement Learning with Single Rollout

SAO addresses asynchronous RL instability and off-policy drift by combining single-rollout sampling, token-level clipping, and value-model training strategies. The resulting design supports stable scaling to thousands of steps and improves agentic-task performance.

  • Token-Level Off-Policy Correction: SAO uses the rollout policy as the behavior proxy and directly computes importance ratios from rollout log-probabilities, avoiding separate old-policy inference and historical checkpoint tracking.This accepts controlled off-policy bias in exchange for lower computational complexity and avoids errors from a potentially stale latest old policy.
  • Token-Level Off-Policy Correction: SAO restricts token ratios to [1 −ϵℓ, 1 +ϵh] and masks tokens outside this interval from gradient computation, preventing instability from extreme policy divergence.Unlike standard PPO clipping, the strategy applies a strict double-sided trust-region mask to off-policy tokens.
  • Single-Rollout Sampling: SAO replaces group-wise sampling with single-rollout sampling, feeding each generated sample into training immediately to reduce imbalanced-generation bias and waiting for slow samples.Group-wise sampling can worsen off-policy effects in asynchronous RL, while single-rollout optimization introduces high gradient-estimation variance.
  • Value-Model Training: Faster Value Update decouples policy and value-model optimization frequencies because inaccurate value estimates produce noisy advantages and destructive single-rollout policy updates.The design targets the interdependence between the policy and value function identified as a primary source of instability.
  • Value-Model Training: Frozen-Attention training regularizes the value model by restricting optimization to Mixture-of-Experts layers after instability was traced primarily to Full Attention layers.Pilot experiments found significantly larger value-model gradient norms than policy-model norms, while MoE layers remained relatively stable.
  • Value-Model Training: Skip-Observation GAE bridges consecutive action values across environment-feedback tokens, filtering feedback stochasticity from token-level advantage estimation in agentic trajectories.The method links the last token of one action directly to the first token of the next action and avoids discontinuous action-observation transitions.
  • Value-Model Training: Scaling value-model pretraining data alleviates cold-start bottlenecks and provides robust initialization for single-rollout and TTUR mechanisms from early training.The paper identifies expanded value-pretraining data as essential for supporting the proposed value-model mechanisms.

4 Experiments

Experiments show that SAO outperforms baselines on agentic reasoning and coding benchmarks while maintaining stable long-horizon training. Ablations and a simulated non-stationary online task support the value-training, clipping, and single-rollout designs.

  • Evaluation Setup: The evaluation covers AIME2025, BeyondAIME, HMMT Nov 2025, IMOAnswerBench, and SWE-Bench Verified using long-horizon tool-interaction settings.Math evaluations allow up to 50 turns, while SWE-Bench Verified allows up to 300 OpenHands interaction turns.
  • Benchmark Results: SAO consistently outperforms all baselines on agentic reasoning and coding benchmarks, while vanilla GRPO collapses at approximately 160 training steps.GRPO with DIS remains stable, whereas SAO and GRPO with DIS diverge after approximately 400 training steps.
  • Ablation Studies: All examined value-training and critic-update variants perform worse than SAO, supporting faster critic updates and frozen-attention value training.A single critic update insufficiently tracks rapid policy shifts, producing less reliable baseline estimates.
  • Training Dynamics: After approximately 400 training steps, SAO achieves significantly higher explained variance than the single-critic-update baseline, indicating faster value convergence.Explained variance measures alignment between predicted values and ground-truth returns.
  • Training Dynamics: Frozen-attention value training maintains lower and smoother critic gradient norms than full-parameter training, implying improved numerical stability.Full-parameter value training exhibits significantly larger critic gradient norms.
  • Online Adaptation: SAO rapidly realigns its policy after sequential reward shifts among cute, chuunibyou, and classical writing styles in a simulated online environment.The task uses single-trajectory feedback, which is incompatible with GRPO’s group-relative advantage estimation; SAO instead uses a value-based critic.

5 Related Work

Prior RLHF work developed PPO and critic-free objectives such as GRPO and RLOO, while later methods refined stability and variance reduction mainly for synchronous RL. Asynchronous LLM RL addresses synchronization inefficiency through decoupled actor–learner systems, but remains comparatively less explored for importance sampling and clipping.

  • RLHF objectives: PPO remains the standard RLHF policy-optimization method, while critic-free objectives such as GRPO and RLOO reduce value-function overhead and instability.The cited objectives include Group Relative Policy Optimization (GRPO) and REINFORCE-style baselines such as RLOO.
  • Objective refinements: Recent GRPO/PPO refinements target stability and variance reduction through sequence-level importance weighting, adaptive clipping, and smoother alternatives to hard clipping.These methods primarily study synchronous RL, where exact importance-sampling ratios are easier to obtain.
  • Open gap: Importance sampling and clipping strategies for asynchronous RL remain less explored than their synchronous counterparts.The gap follows from the greater difficulty of obtaining exact importance-sampling ratios asynchronously.
  • Asynchronous RL systems: Synchronous LLM RL collects a full rollout batch under a fixed policy snapshot before optimization, but long-tail outputs create stragglers and substantial idle time.These inefficiencies motivate asynchronous actor–learner designs that generate rollouts and learn concurrently.
  • Asynchronous RL systems: Asynchronous LLM RL systems include online-but-off-policy analysis, staleness-aware PPO updates, and rollout–training decoupling with fine-grained parallelism.Examples include Noukhovitch et al.’s analysis, AReaL, and ROLL Flash for reasoning, RLVR, and agentic training.

6 Conclusion

The paper proposes SAO, a single-rollout asynchronous RL strategy that addresses off-policy effects and training instability. It combines token-level importance sampling, double-sided clipping and masking, and stronger value-model training, yielding consistent outperformance on agentic reasoning and coding tasks.

  • Conclusion: SAO addresses off-policy effects and training instability through a single-rollout asynchronous RL strategy.It replaces group-wise sampling with single-rollout sampling.
  • Conclusion: SAO stabilizes training with token-level importance sampling and double-sided clipping and masking.
  • Conclusion: SAO improves generalization through single-rollout sampling enabled by stronger value-model training and consistently outperforms on agentic reasoning and coding tasks.

A Additional Experimental Results · A.1 RL with Agentic Step as Action

This section evaluates treating an agentic conversation turn as an RL action through step-wise value estimation and GAE. Both step-level variants underperform token-wise value training, which better preserves fine-grained supervision for complex reasoning trajectories.

  • A.1 RL with Agentic Step as Action: Step Average defines a step’s value as the mean of its constituent token-value predictions and trains the value model on all tokens.This aggregation is intended to reduce variance in token-level value predictions.
  • A.1 RL with Agentic Step as Action: Last-Token Prediction defines a step’s value using only its final token and masks intermediate tokens during value-model training.The method assumes the final token encapsulates the most comprehensive information about the step.
  • A.1 RL with Agentic Step as Action: Step-wise GAE treats each conversation turn as one action and assigns a uniform learning signal to all tokens within that step.The step-level advantage is derived from the TD error δ_i = R_i + γV(S_i+1) − V(S_i).
  • A.1 RL with Agentic Step as Action: Both step-wise approaches underperform token-wise value training in training reward and benchmark performance.Figure 6 shows better training rewards for token-level SAO training, while Table 5 reports the corresponding action-granularity ablation at 400 training steps.
  • A.1 RL with Agentic Step as Action: Token-level training provides finer-grained supervision for both the critic and policy, which is essential for capturing logical transitions in complex reasoning trajectories.This is the paper’s explanation for the failure of step-level value and policy training.
  • A.1 RL with Agentic Step as Action: The ablation compares Step-level and Token-level action granularities for value and policy model training using the same 400 training steps.Step-level views each agent step as an action, whereas Token-level views each token as an action.

A.2 Comparison to Other Baselines of Single-Rollout Strategies

SPO and historical running-mean rewards are feasible single-rollout baselines, but both rely on prior training-data difficulty information and perform worse than SAO.

  • Single-rollout baselines: SPO and historical running-mean rewards are feasible baselines for RL with one rollout per prompt.Both approaches can support single-rollout training and use different advantage-estimation baselines.
  • Single-rollout baselines: Both SPO and running-mean baselines rely on prior information about training-data difficulty.This dependence is identified as a distinguishing limitation relative to SAO.
  • Single-rollout baselines: SPO and running-mean baselines achieve worse performance than SAO in the experiment section.The passage reports the performance comparison without specifying numerical results.

B Limitations and Broader Impact

The study’s conclusions may not generalize beyond the evaluated large-scale agentic tasks and Qwen3-30B-A3B backbone, while deployment also requires specialized asynchronous-generation infrastructure. Although SAO may reduce training costs, it could facilitate harmful optimization without safeguards.

  • Limitations: The experiments use a Qwen3-30B-A3B backbone and focus on agentic reasoning, coding, and simulated online writing tasks.The conclusions may not transfer directly to smaller models, non-agentic RLHF settings, or dense-reward environments with shorter rollouts.
  • Limitations: SAO deployment requires a trained value model and infrastructure that preserves token-level rollout log-probabilities during asynchronous generation.
  • Broader Impact: Improving LLM reinforcement-learning stability and efficiency could reduce the cost of training capable agentic systems.
  • Broader Impact: The same capabilities could make it easier to optimize models for harmful objectives without appropriate data filtering, access controls, evaluation, responsible release, and monitoring.
Loading 2607.07508v1…