Source-linked AI summary
SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL
Kai Ruan, Jinghao Lin, Qianshan Wei, Ziqi Zhou, Zihe Huang
TL;DR
Single-stream RL avoids waiting for sibling rollouts but must align persistent prompt statistics with both policy event time and the token measure optimized by the actor. SPO++ makes those alignments explicit, and across ALFWorld and Math-TIR it improves reward-curve area over SPO, with action-token-measure normalization the strongest isolated contributor.
Problem
Single-stream Policy Optimization avoids sibling-rollout waits, but receipt-order tracking and trajectory whitening can mismatch the policy clock and token-mean actor loss.
Method
SPO++ freezes prompt values at dispatch, attaches outcomes to their generating policy events, and standardizes terminal advantages under the action-token measure.
Results
Across two ALFWorld model scales and Math-TIR, SPO++ improves reward-curve area over SPO, and a paired ablation identifies action-token-measure normalization as the strongest isolated contributor.
Takeaways & Limitations
Single-stream RL requires persistent statistics to align with both the policy clock and the measure optimized by the actor.
Takeaways & Limitations
Experiments use small Qwen3.5 models under limited budgets, while future evaluations should extend to longer horizons and out-of-distribution tasks.
Abstract
from arXiv · showhide
Group-relative reinforcement learning waits for sibling rollouts of the same prompt, which is costly for long and variable tool-use trajectories. Single-stream Policy Optimization (SPO) removes this dependency with a persistent prompt-level value estimate, but its recipe whitens one advantage per trajectory before optimizing a token-mean actor loss. We show that trajectory centering generally does not center the token-weighted quantity consumed by the actor, and fix the mismatch by standardizing terminal-outcome advantages under the action-token measure. We additionally organize prompt evidence by the policy event that generated it rather than learner receipt order. Across matched runs on ALFWorld at two model scales and on Math-TIR, SPO++ improves online learning efficiency over SPO. A paired ablation identifies action-token-measure normalization as the strongest tested component.
1 Introduction
SPO++ addresses two consistency problems in single-stream RL: receipt-order prompt memory and trajectory-level whitening followed by token-mean optimization. It introduces event-time tracking and action-token-measure normalization, then evaluates the combined recipe across ALFWorld and Math-TIR.
- Group-relative RL waits for the slowest sibling rollout, making long and variable tool-use trajectories costly to train.
- SPO avoids sibling-rollout dependency with persistent prompt evidence, but receipt order and variable response length can distort the baseline and actor-loss center.
- SPO++ freezes prompt values at dispatch and records returned outcomes by the policy event that generated them.
- Action-token-measure normalization aligns advantage standardization with the token-mean actor loss while preserving trajectory-level credit granularity.
- The evaluation covers two ALFWorld model scales and Math-TIR, with a matched ablation isolating the normalization measure.
2 Method
SPO++ combines event-time persistent prompt memory with action-token-measure standardization while retaining SPO’s single-rollout, terminal-outcome setup. The resulting design separates how historical evidence is selected from how completed batches are normalized for the actor loss.
- Single-stream prompt values: SPO++ retains one online rollout per prompt and uses persistent evidence to supply a prompt-level baseline.
- Event-time prompt memory: Prompt values are frozen at dispatch, while returned outcomes are attached to the policy event that generated them.
- Event-time prompt memory: Event-time tracking makes the baseline independent of receipt permutation for a fixed visible history, up to floating-point reduction error.
- Event-time prompt memory: A returned outcome updates later dispatches, but its own baseline remains the snapshot read when the request was dispatched.
- Event-time prompt memory: Under the experimental regime, retention is geometric with ρ = 0.875 across policy events, while SPO derives retention from post-update token drift.
- Action-token-measure normalization: Trajectory-wise whitening can leave a nonzero token-weighted residual when trajectory length and advantage co-vary.
- Action-token-measure normalization: SPO++ standardizes advantages under the action-token measure consumed by the loss, assigning one scalar to every valid action token in a trajectory.
- Protocol: The evaluated recipes also differ in retention and negative Dual-Clip settings, so the end-to-end comparison is not an isolated normalization test.
3 Experiments
Matched experiments compare SPO++ and SPO across ALFWorld, Math-TIR, and a normalization ablation. SPO++ improves online learning efficiency, while the paired ablation identifies action-token-measure normalization as the strongest tested component.
- Protocol: SPO++ is evaluated against SPO with matched checkpoints, prompts, initialization, decoding, budgets, optimizer settings, and asynchronous runtime.
- Protocol: Results cover Qwen3.5-0.8B and Qwen3.5-2B on ALFWorld and Qwen3.5-0.8B on Math-TIR.
- Main results: Every ALFWorld run improves in curve area, while six of seven finish with positive end-reward differences; Math-TIR improves on both metrics in four of five runs.
- Main results: The larger gains on long-horizon ALFWorld and smaller Math-TIR improvement are qualitatively consistent with measure mismatch mattering when response length covaries with advantage.
- Ablation: 10.70 ± 3.98 percentage points is the paired learning-composite gain from action-token-measure normalization over trajectory normalization.
- Optimization diagnostics: SPO++ exhibits lower PPO KL and clip fraction at a comparable gradient-norm scale, but these are recipe-level diagnostics because retention and negative Dual-Clip also differ.
4 Related Work
Related work spans group-relative and single-rollout advantage estimation, learned values, objective and supervision changes, and asynchronous policy optimization. SPO++ is positioned within these lines while addressing persistent statistics and policy-event timing.
- Single-rollout advantage estimation: GRPO uses within-prompt relative comparison, while SPO carries prompt-level evidence across visits for single-rollout estimation.
- Learned and implicit values: Value-based methods alter the baseline through pretrained, frozen, or learned value estimates and adaptive sampling or stabilization.
- Objectives and supervision granularity: Other methods change objectives or supervision granularity through compute-indexed likelihoods, token-level recurrences, distillation targets, or aggregation rules.
- Asynchronous policy optimization: Asynchronous methods manage policy lag using stale-rollout ratios, staleness-aware anchors, or adapted trust regions.
5 Limitations
The study is bounded by small-model, limited-budget evaluations and several asynchronous and credit-assignment constraints.
- Experiments use small Qwen3.5 models under limited training budgets, leaving longer horizons and out-of-distribution tasks for future evaluation.
- The end-to-end comparison jointly changes event-time tracking, retention, and Dual-Clip, while the shorter ablation does not separate centering from batchwise scaling.
- A one-request same-prompt concurrency cap restricts event-time experiments to cross-prompt completion order.
- Persistent prompt values require repeatable task identities and offline initialization, first-completed collection can favor shorter trajectories, and outcome-level credit retains sparse-reward cold-start difficulty.
6 Conclusion
SPO++ aligns advantage normalization with the action-token measure and organizes persistent evidence by policy event time. Across ALFWorld and Math-TIR, it improves reward-curve area over SPO, with normalization strongest in the paired ablation.
- SPO++ aligns single-stream advantage normalization with the action-token measure consumed by a token-mean actor loss.
- Persistent prompt evidence is organized by policy event time rather than learner receipt order.
- Across two ALFWorld model scales and Math-TIR, SPO++ improves reward-curve area over SPO.
- A paired ablation identifies action-token-measure normalization as the strongest isolated contributor.
A.1 Metric definitions
The metrics use normalized reward-curve area and final-five reward, with C10 combining these measures for the ten-step normalization ablation.
- Metric definitions: Normalized reward-curve area is computed using trapezoidal integration, while the rectangle-rule area is used where specified.A matched comparison truncates both methods to their common number of completed reward points before computing either area.
- Metric definitions: C10 equally weights normalized trapezoidal reward-curve area and the final-five reward mean.C10 is reported only for the ten-step normalization ablation; Table 1 reports the two component metrics separately.
A.2 Training and implementation details
The reported experiments use a shared training configuration, with event-time evidence handling and model- or task-specific initialization details described alongside the training setup.
- Training and implementation details: Math-TIR uses a supervised cold start from verifier-positive, structurally valid Python-tool traces before online reinforcement learning.The retained 362 prompts are split 326/36, and the model is trained for two bfloat16 epochs with maximum length 40,960.
- Training and implementation details: Prompt-value initialization uses Jeffreys-smoothed estimates of eight binary verifier outcomes per prompt.Each pair shares its initialization: a model-specific base policy for ALFWorld and a shared cold-start policy for Math-TIR.