PaperlayerView exact revision on arXiv

Source-linked AI summary

Single-stream Policy Optimization

Zhongwen Xu, Zihan Ding

arXiv:2509.13232v2cs.LGcs.AIstat.ML

TL;DR

Group-based LLM policy optimization can waste computation on degenerate groups and lose scalability at synchronization barriers. SPO addresses these issues with a persistent value tracker and global advantage normalization, and it outperforms GRPO on challenging reasoning benchmarks while improving throughput in variable-time settings. Its main limitation is that using no baseline remains valid but can suffer from high policy-gradient variance.

  • Problem

    Group-based methods can erase learning signals when all responses share an outcome and can bottleneck training through group synchronization.

  • Method

    SPO uses a persistent KL-adaptive value tracker, global advantage normalization, and a group-free single-stream architecture with prioritized sampling.

  • Results

    SPO improves average maj@32 by +3.4 percentage points (pp) over GRPO across five benchmarks and achieves a 4.35× training-throughput speedup in variable-time simulations.

  • Takeaways & Limitations

    SPO provides a more robust, scalable, and efficient foundation for LLM reasoning and agentic training while challenging incidental algorithmic complexity.

  • Takeaways & Limitations

    Using no baseline is valid but may suffer from high policy-gradient variance.

Abstract

from arXiv · show

We revisit policy-gradient optimization for Large Language Models (LLMs) from a single-stream perspective. Prevailing group-based methods like GRPO reduce variance with on-the-fly baselines but suffer from critical flaws: frequent degenerate groups erase learning signals, and synchronization barriers hinder scalability. We introduce Single-stream Policy Optimization (SPO), which eliminates these issues by design. SPO replaces per-group baselines with a persistent, KL-adaptive value tracker and normalizes advantages globally across the batch, providing a stable, low-variance learning signal for every sample. Being group-free, SPO enables higher throughput and scales effectively in long-horizon or tool-integrated settings where generation times vary. Furthermore, the persistent value tracker naturally enables an adaptive curriculum via prioritized sampling. Experiments using Qwen3-8B show that SPO converges more smoothly and attains higher accuracy than GRPO, while eliminating computation wasted on degenerate groups. Ablation studies confirm that SPO's gains stem from its principled approach to baseline estimation and advantage normalization, offering a more robust and efficient path for LLM reasoning. Across five hard math benchmarks with Qwen3 8B, SPO improves the average maj@32 by +3.4 percentage points (pp) over GRPO, driven by substantial absolute point gains on challenging datasets, including +7.3 pp on BRUMO 25, +4.4 pp on AIME 25, +3.3 pp on HMMT 25, and achieves consistent relative gain in pass@$k$ across the evaluated $k$ values. SPO's success challenges the prevailing trend of adding incidental complexity to RL algorithms, highlighting a path where fundamental principles, not architectural workarounds, drive the next wave of progress in LLM reasoning.

1. Introduction

SPO reframes policy optimization as a single-stream process to avoid degenerate-group learning failures and synchronization bottlenecks. It combines persistent value estimation, global advantage normalization, and prioritized sampling, and outperforms GRPO on challenging reasoning benchmarks.

  • Group-based methods lose their learning signal when every response in a group shares the same outcome.All-correct and all-incorrect groups produce zero relative advantage, wasting computation.
  • Group-based training creates synchronization barriers because the entire group must wait for its slowest response, especially in long-horizon or tool-integrated tasks.Variable interaction times make a single slow trajectory capable of stalling the whole group.
  • SPO replaces per-group baselines with a persistent Bayesian value tracker, globally normalizes advantages, and supports prioritized sampling of informative prompts.The tracker provides a temporally informed success-probability estimate, while global normalization avoids unstable per-group statistics.

2. Related Work

Related work develops several alternatives to group-based policy optimization, including filtering, Bayesian prompt estimation, simplified objectives, leave-one-out baselines, and offline value estimation. These methods differ in baseline construction, but group-based approaches retain shared architectural limitations.

  • GRPO constructs an on-the-fly baseline by normalizing rewards across multiple responses generated for each prompt.This removes the need for a separate critic network but depends on group sampling for each update.
  • GRESO filters predicted-uninformative prompts, Bayesian prompt estimation forms a bandit strategy, and Lite PPO reduces RLVR training to advantage normalization and token-level loss aggregation.These approaches target rollout overhead, sampling strategy, or objective simplification.
  • RLOO uses a leave-one-out reward baseline and OPO uses a length-weighted average of rewards as a simplified baseline.Both methods retain group-derived baseline construction.
  • A*-PO uses offline estimation of an optimal value function and then constructs optimal advantages for least-squares regression during online training.Its two-stage procedure differs from approaches that update value estimates persistently during training.

3. Background

Policy gradients optimize expected reward but can have high-variance estimates, motivating action-independent baselines and advantage functions. LLM methods estimate these baselines with critics or small response groups, though group baselines can be noisy and sample-inefficient.

  • Policy gradients optimize expected reward for prompt-response trajectories, while REINFORCE updates the policy using an estimated reward-weighted gradient.For LLMs, a trajectory consists of generating one response y from a prompt x.
  • Subtracting an action-independent baseline from reward preserves an unbiased gradient estimator while reducing variance.The resulting advantage is A(x, y) = R(x, y) − b(x).
  • The optimal baseline is the policy value function, the expected reward for a prompt, but it must be estimated in practice because the true value is unknown.Baseline-estimation quality affects RL stability and efficiency.
  • PPO learns a parameterized critic, whereas GRPO estimates its baseline from the mean rewards of multiple responses sampled for one prompt.PPO critics are resource-intensive, while GRPO relies on concurrent group samples.
  • GRPO requires multiple generations per prompt and can produce noisy advantages because its baseline is estimated from a small group.Its raw advantage subtracts the group baseline and is normalized by the group standard deviation.

4. Method

SPO combines a KL-adaptive persistent value tracker, globally normalized advantages, and prioritized sampling within a group-free single-stream policy-gradient framework. This design targets stable learning while improving scalability and retaining compatibility with standard policy-optimization updates.

  • 4.1. A KL-Adaptive Value Tracker: SPO uses a Bayesian value tracker with KL-adaptive forgetting to estimate each prompt’s changing success probability from binary rewards.The tracker discounts older observations more aggressively when the current policy diverges from the last policy that acted on the prompt.
  • 4.2. Advantage Estimation and Policy Optimization: The tracker’s pre-update estimate supplies the advantage baseline, preserving action independence and unbiased policy-gradient estimation.For each sample, SPO computes the raw advantage as reward minus the previous tracker estimate before updating the tracker.
  • 4.2. Advantage Estimation and Policy Optimization: SPO globally normalizes batch advantages and applies them to response tokens with a standard PPO-Clip policy loss.The batch statistics are the mean and standard deviation of advantages across the sampled batch.
  • 4.2. Advantage Estimation and Policy Optimization: SPO’s no-baseline variant remains valid but can suffer from high policy-gradient variance.The tracker and advantage-normalization components address the variance challenge that motivates SPO’s design.
  • 4.3. Adaptive Curriculum: SPO samples prompts by learning potential, emphasizing uncertainty while retaining non-zero exploration probability for every prompt.The uncertainty term favors prompts whose estimated success probabilities are neither near zero nor near one, while the exploration bonus prevents curriculum collapse.
  • 4.4. Advantages over GRPO: As a group-free method, SPO collects independent prompt-response pairs instead of waiting for complete response groups, reducing synchronization delays from slow trajectories.This design is especially suited to multi-turn tool use and long-horizon rollouts with variable interaction times.

5. Experiments

Experiments on Qwen3-8B across five math benchmarks show SPO outperforming GRPO while providing more efficient and stable learning signals. Simulations further demonstrate that SPO’s group-free design improves throughput in variable-latency agentic settings.

  • Empirical Comparison with GRPO: +3.4 percentage points in maj@32: SPO scores 63.8 versus GRPO’s 60.4 across five benchmarks.SPO outperforms GRPO on maj@32 across all five benchmarks, including +7.3 pp on BRUMO 25, +4.4 pp on AIME 25, and +3.3 pp on HMMT 25.
  • Empirical Comparison with GRPO: SPO’s weighted-average pass@k curve stays above GRPO’s across all evaluated k values, with an average improvement of approximately 2.4 pp.The comparison is reported across five math competition benchmarks.
  • Analysis of Signal Efficiency and Stability: GRPO’s degenerate-group rate rises from roughly 60% to over 80%, whereas SPO’s near-zero advantages remain much lower and reflect increasingly accurate value predictions.SPO tracks |A| ≤ 10^-4 and |A| ≤ 0.02; the tighter near-zero ratio remains rare while the looser ratio grows as residuals shrink on mastered prompts.
  • Analysis of Signal Efficiency and Stability: SPO’s history-informed baseline reduces variance by nearly 50%, while GRPO’s effective non-degenerate advantage is highly volatile and has the largest variance.GRPO’s apparently stable overall variance is attributed to zero-variance degenerate samples rather than robust learning signals.
  • Agentic Training Demonstrations: In a high-variance agentic simulation, group synchronization forces faster trajectories to wait for a 508-second straggler, wasting compute and reducing throughput.The bottleneck is particularly relevant to multi-turn tool use and long-horizon reasoning with variable interaction times.
  • Agentic Training Demonstrations: Group-free sampling assembles 24 samples in 112 seconds versus 486 seconds for parallelized group-based sampling, yielding a 4.35× speedup.The group-free strategy starts 48 independent samples and selects the first 24 to finish, avoiding slow outliers.

6. Conclusions

SPO addresses degenerate-group waste and synchronization bottlenecks by combining single-stream training with persistent value tracking and global advantage normalization. The paper reports stronger, more scalable training while noting offline initialization cost as a practical consideration.

  • SPO eliminates computational waste from degenerate groups and synchronization bottlenecks in group-based policy optimization.These inefficiencies arise when groups provide no learning signal or must wait for their slowest member.
  • SPO replaces group-based baselines with a persistent KL-adaptive value tracker and global advantage normalization for more stable learning.The method returns to a single-stream policy-optimization paradigm.
  • SPO is reported to outperform GRPO on complex reasoning tasks while supporting robust, scalable, and efficient training.The conclusion frames this result as evidence that a simpler single-stream approach can surpass more complex methods.
  • Offline initialization of SPO’s value tracker introduces extra cost, though it can be amortized across experiments or avoided by loading shared datasets.The authors identify this as a practical concern and describe dataset-sharing mitigations.
  • SPO can be adapted to prompt repetition while retaining its primary asynchronous advantage over GRPO.Repeated prompts are treated as independent trajectories, enabling management of slow or problematic rollouts.

C.1. Inefficiency of Dynamic Sampling

Dynamic sampling avoids degenerate groups by collecting both successful and failed responses, but its expected cost becomes large for very easy or very hard prompts. SPO instead uses one sample per prompt and deprioritizes inefficient prompts adaptively.

  • Dynamic sampling guarantees a non-zero advantage by continuing until a prompt has both a success and a failure.This strategy addresses information loss from groups whose rewards are all identical.
  • Reported dynamic-sampling steps refer to learning steps, whereas sampling steps can be multiple times larger.The paper gives 5× as an example multiplier.
  • The expected number of samples grows hyperbolically as success probability approaches 0 or 1.For p = 0.1, collecting both outcomes requires approximately 10.11 generations on average.
  • SPO requires exactly one sample per prompt and uses adaptive curriculum to deprioritize inefficient prompts.This allocates resources toward prompts where learning is more effective.

C.2. Variance Reduction for Policy Gradient

The variance analysis attributes GRPO’s instability to noisy group baselines, noisy scaling, and degenerate groups. SPO decouples advantage estimation from current samples and normalizes across a large batch to reduce these sources of variance.

  • The policy-gradient signal is determined by the advantage multiplying the score-function gradient, making advantage construction central to training efficiency.The analysis therefore compares how GRPO and SPO estimate and normalize advantages.
  • GRPO’s group advantage suffers from noisy baselines, noisy scaling, and information loss when all group rewards match.Identical rewards make every advantage zero, eliminating the gradient signal.
  • SPO uses a historical action-independent value tracker and global batch normalization to produce lower-variance advantages.The tracker estimates success probability independently of the current sample group, while normalization uses one large-batch standard deviation.
  • For easy or hard prompts, GRPO’s information-loss term can dominate and the variance ratio reaches approximately 1.97.The cited example contrasts p = 0.9 or p = 0.1 with a moderately difficult p = 0.5 prompt using G = 8.
  • Increasing GRPO group size reduces information loss only at higher generation cost and does not remove small-sample baseline and scaling noise.SPO is presented as achieving lower variance without these group-size costs.

D. Training and Evaluation Details

The experiments use fixed training and sampling configurations, compare GRPO and SPO at matched response budgets, and evaluate hard math reasoning with avg@k, pass@k, and maj@k metrics.

  • Training uses maximum response length 16,384 tokens, learning rate 1 × 10^-6, and Clip-Higher clipping with ε_low = 0.2 and ε_high = 0.28.Sampling uses temperature 1.0, top-p = 1.0, and top-k = -1 during training.
  • The fair comparison gives GRPO 256 prompts with 8 responses each, while SPO processes 2,048 prompts for the same total response count.Both algorithms perform 8 gradient updates per rollout step.
  • Evaluation covers AIME 24, AIME 25, BeyondAIME, BRUMO 25, and HMMT 25 with temperature 0.6, top-p = 0.95, and top-k = 20.Responses receive binary reward 1 for a correct final answer and 0 otherwise.
  • avg@k measures expected individual-response correctness.
  • pass@k measures the probability of solving a problem within k attempts, whereas maj@k measures correctness of the majority-voted answer.The metrics are computed from independently sampled responses per problem.

E. Ablation Studies

The ablations show that SPO’s adaptive value tracker is central to stable learning, while offline initialization provides a necessary warm start. Comparisons with a static baseline further support adapting the baseline as the policy evolves.

  • Experimental Setup: The ablations use a streamlined setting with 256 prompt-response pairs, four gradient steps per batch, and AIME 25 avg@16 evaluation.Generation length is capped at 16,384 tokens.
  • SPO vs. A*-PO: SPO’s adaptive baseline outperforms the static baseline of A*-PO, supporting a value function that evolves with the policy.The comparison is reported in Figure 6a under the ablation setting.
  • Baseline Ablation: Removing the value tracker causes substantial performance degradation because globally normalized raw rewards remain noisy at the per-sample level.The history-informed baseline accounts for prompt-specific difficulty and reduces gradient variance.
  • Offline Initialization Ablation: Removing offline initialization produces initial instability and a performance dip, and the tracker does not recover to the properly initialized model’s final performance.The standard tracker receives a warm start from a small set of offline samples.
Loading 2509.13232v2…