Source-linked AI summary
SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
Tianyi Wang, Yixia Li, Long Li, Yibiao Chen, Shaohan Huang, Yun Chen, Peng Li, Yang Liu, Guanhua Chen
TL;DR
Long-horizon sparse-reward reasoning exposes instability in token-level PPO, while GRPO’s multi-sample baselines limit throughput. SPPO reformulates reasoning as a Sequence-Level Contextual Bandit with a decoupled scalar value function, achieving strong benchmark performance with single-sample updates. Its demonstrated scope is RLVR tasks with verifiable outcomes.
Problem
Token-level PPO suffers unstable credit assignment over long CoT horizons, while GRPO requires computationally expensive multi-sampling for baseline estimation.
Method
SPPO treats the prompt as context and the full reasoning chain as one atomic action, using a learned scalar value function for low-variance advantage estimation without multi-sampling.
Results
SPPO achieves the highest overall performance, surpassing GRPO (N = 8) on most benchmarks with Avg 48.06 vs. 47.08 on 1.5B while using N = 1.
Takeaways & Limitations
SPPO offers a scalable sparse-reward alignment paradigm combining optimization stability with high-throughput single-sample efficiency.
Takeaways & Limitations
The approach is tailored to verifiable-outcome RLVR tasks, and extending it to open-ended generation without objective verifiers remains future work.
Abstract
from arXiv · showhide
Proximal Policy Optimization (PPO) is central to aligning Large Language Models (LLMs) in reasoning tasks with verifiable rewards. However, standard token-level PPO struggles in this setting due to the instability of temporal credit assignment over long Chain-of-Thought (CoT) horizons and the prohibitive memory cost of the value model. While critic-free alternatives like GRPO mitigate these issues, they incur significant computational overhead by requiring multiple samples for baseline estimation, severely limiting training throughput. In this paper, we introduce Sequence-Level PPO (SPPO), a scalable algorithm that harmonizes the sample efficiency of PPO with the stability of outcome-based updates. SPPO reformulates the reasoning process as a Sequence-Level Contextual Bandit problem, employing a decoupled scalar value function to derive low-variance advantage signals without multi-sampling. Extensive experiments on mathematical benchmarks demonstrate that SPPO significantly surpasses standard PPO and matches the performance of computation-heavy group-based methods, offering a resource-efficient framework for aligning reasoning LLMs.
1 Introduction
Standard PPO becomes unstable for long-horizon sparse-reward reasoning because token-level credit assignment propagates delayed signals across thousands of tokens. SPPO instead uses a sequence-level bandit formulation and a learned scalar value function to retain stability without GRPO-style multi-sampling.
- Long CoT horizons make PPO’s token-level GAE credit assignment highly biased when rewards are delayed across thousands of tokens.
- The Critic overfits semantic cues near the sequence tail, causing advantages to vanish on correct trajectories and failing to penalize intermediate steps on incorrect ones.
- GRPO removes the learned Critic but introduces high-variance Monte Carlo baselines and requires multiple responses per prompt, bottlenecking throughput.
- SPPO treats the prompt as context and the entire reasoning chain as one atomic action, collapsing the time horizon and reducing token-level credit-assignment bias.
- SPPO uses a learned scalar value function to reduce baseline variance and enable stable single-sample updates, while a decoupled lightweight Critic can align a larger policy.
2 Background
The background contrasts PPO’s token-level value estimation with GRPO’s sequence-level, group-normalized alternative. It motivates SPPO by identifying token-level decomposition as a source of PPO instability and contextual-bandit reformulation as a possible remedy.
- PPO: PPO maximizes a clipped surrogate objective using a token-level Critic and GAE-based advantage estimates.
- PPO: In sparse-reward CoT tasks, setting γ = λ = 1 makes GAE propagate terminal rewards and simplify toward the Monte Carlo return minus the value estimate.
- GRPO: GRPO’s advantage dynamically scales rewards according to estimated prompt difficulty under a Bernoulli model.
- PPO: The PPO Critic’s late value convergence makes advantages vanish for correct trajectories, while underestimation on incorrect ones produces position-based rather than contribution-based credit.
- GRPO: GRPO samples N outputs per prompt, normalizes rewards against a group mean, and implicitly evaluates each response as an atomic action against a prompt-dependent baseline.
- Motivation: These observations motivate asking whether PPO instability arises from token-level MDP decomposition rather than from value estimation itself.
3 Method
SPPO reformulates reasoning as a sequence-level contextual bandit, using a prompt-level solvability value function and one advantage signal shared across the response. This redesign aligns PPO’s optimization with sparse, outcome-based reasoning rewards while addressing token-level credit-assignment ambiguity.
- Sequence-Level Contextual Bandit: SPPO collapses reasoning from a token-level MDP into a sequence-level contextual bandit with the prompt as context and the full response as one atomic action.The reward evaluates holistic correctness, and the formulation uses a collapsed horizon H = 1.
- Value Function and Advantage Estimation: The value function Vϕ(sp) estimates prompt solvability rather than future returns from arbitrary intermediate token states.This scalar baseline is intended to align value estimation directly with sparse-reward reasoning.
- Value Function and Advantage Estimation: SPPO constructs advantages from the sequence outcome and the prompt-level value estimate, with BCE used to calibrate the value model.The outcome is treated as a single-sample Bernoulli realization, while BCE trains the baseline.
- Sequence-Level Policy Optimization: SPPO adapts PPO’s clipped surrogate objective while replacing token-specific GAE advantages with one sequence-level advantage.The policy ratio compares the current and behavior policies, and clipping constrains the update.
- Sequence-Level Policy Optimization: The same advantage is propagated uniformly to every token, reinforcing all steps of successful chains and penalizing all steps of failed chains.This decouples the update signal from sequence length and targets temporal credit-assignment ambiguity.
4 Experiments
Experiments evaluate SPPO on 1.5B and 7B reasoning models across five held-out mathematical benchmarks against token-level and sequence-level baselines. SPPO consistently leads the baselines, supports a smaller critic for a larger policy, and its gains are attributed to sequence-level formulation rather than BCE alone.
- Experimental Setup: SPPO is evaluated on DeepSeek-R1-Distill-Qwen models at 1.5B and 7B scales using Average@16 accuracy across five held-out mathematical benchmarks.The benchmarks are AIME24, AIME25, AMC23, MATH500, and Minerva Math.
- Experimental Setup: The comparison includes the base model, Standard PPO, ReMax, RLOO, and GRPO (N = 8), using outcome-based correctness rewards.Experiments use +1 for correct boxed answers and 0 for incorrect answers.
- Main Results: SPPO achieves the highest overall performance, with Avg 48.06 versus 47.08 for GRPO (N = 8) on the 1.5B model.The reported comparison covers most benchmarks, while Standard PPO does not consistently improve over the base model.
- Main Results: A 1.5B critic aligning a 7B policy achieves the highest average score of 58.56, showing that SPPO retains effectiveness with a smaller value model.The authors associate this critic decoupling with reduced RLVR memory footprint.
- Ablation Study: Impact of Loss Function: PPO + BCE fails to reproduce SPPO’s success, and both PPO-based runs are stopped at 500 steps after performance collapse and degrading scores.The ablation attributes SPPO’s efficacy to the sequence-level contextual-bandit formulation and unified advantage propagation rather than BCE alone.
5 Analysis
SPPO improves training efficiency, memory accessibility, and sparse-reward robustness through sequence-level value estimation. Its critic correlates with task difficulty and supports stable performance across mathematical and control-task evaluations.
- Training Efficiency: SPPO reaches peak performance (mean score ≈58) in approximately 22 hours with single-sample efficiency (N = 1).GRPO (N = 8) and RLOO take longer because they generate multiple samples per prompt for baseline estimation.
- Resource Efficiency: SPPO with a decoupled critic maintains a low memory profile, enabling alignment of large reasoning models under constrained GPU budgets.The implementation also leverages memory management and sharding techniques from verl.
- Value Model Analysis: The Value Model captures relative prompt difficulty, with Pearson correlation 0.642 and Spearman rank correlation 0.664 against empirical results.The evaluation used 200 prompts and empirical AVG@k pass rates as difficulty labels.
- Value Model Analysis: The critic predicts conservatively, producing a unimodal distribution centered around 0.6-0.7 despite bimodal empirical difficulty.For hard prompts, it predicts ≈0.5; for easy prompts, ≈0.8, yielding stronger advantages for rare successes and failures.
- RLVR Benchmark: Across five sparse-reward control tasks, SPPO matches or outperforms Standard PPO, solving long-horizon Hopper and MountainCar tasks where PPO remains near zero.SPPO also maintains monotonic improvement in LunarLander and rapidly converges to high-precision CartPole behavior.
6 Related Work
Related methods address instability or sequence-level optimization through different mechanisms, but group-based approaches remain computationally expensive. SPPO instead isolates the sequence-level contextual-bandit formulation and replaces empirical baselines with a learned scalar value function.
- Group-Based Methods: GRPO mitigates token-level credit-assignment problems through group-based multi-sampling, but remains computationally expensive.DAPO and Dr.GRPO refine gradient dynamics while retaining the multi-sampling paradigm.
- SPPO Positioning: SPPO isolates the Sequence-Level Contextual Bandit formulation rather than adding orthogonal optimizations to group-relative baselines.It replaces the empirical baseline with a learned scalar value function.
- Sequence-Level RL: RLOO criticizes token-level PPO optimization but is built on the REINFORCE algorithm.Recent work identifies clipping and token masking as important to learning stability.
- Sequence-Level RL: GSPO and GMPO argue that sequence-level rewards conflict with PPO’s token-level design, but target routing instability in Mixture-of-Experts architectures.The paper excludes them to focus on general reasoning alignment.
7 Conclusion
SPPO addresses the trade-off between standard PPO’s high-bias credit assignment and GRPO’s high-variance inefficiency. It combines a sequence-level contextual-bandit formulation with scalar-critic advantage estimation for stable, high-throughput sparse-reward learning.
- Conclusion: SPPO reformulates reasoning as a Sequence-Level Contextual Bandit and employs a scalar critic for advantage estimation.The method is designed for high-throughput single-sample efficiency in sparse-reward tasks.
Limitations
SPPO is evaluated primarily on RLVR tasks with verifiable outcomes, because its formulation relies on estimating prompt solvability. Extending it to open-ended generation without objective verifiers remains future work.
- Scope Boundary: SPPO’s sequence-level bandit formulation is explicitly tailored to tasks with verifiable outcomes and prompt-solvability estimates.Open-ended generation without objective ground-truth verifiers remains outside the demonstrated scope.
Ethical Considerations
The study uses controlled, text-only benchmark environments and excludes human subjects and personal-data collection. It also identifies misuse and bias-propagation risks requiring safeguards and oversight.
- The experiments use controlled, text-only benchmark environments.
- The study does not involve human subjects or collect personal data.
- Potential misuse and bias propagation motivate responsible deployment with safeguards and oversight.
A Derivation and Analysis of GRPO Advantage
The analysis derives GRPO’s standardized advantage from Bernoulli outcome modeling, then examines how token-level critics behave on long-horizon reasoning trajectories. The reported trajectories show that correct and incorrect responses remain difficult to distinguish until their tails.
- Derivation: GRPO models sampled responses as independent Bernoulli trials with success probability p = P(R = 1|sp).
- Derivation: The GRPO advantage standardizes each reward relative to the group sample mean and standard deviation.
- Analysis: Across ten validation problems, token-level critic value curves for correct and incorrect trajectories typically remain entangled until the final few tokens.
- Scope: SPPO relies on verifiable outcome rewards to estimate prompt solvability, limiting its supported scope for tasks without objective ground truths.
D Resources and Implementation Details
The paper documents its experimental resources, reproducing commands, and execution configurations for SPPO and several comparison methods. It also includes a critic-dynamics visualization and links resource information through HuggingFace.
- Analysis: Figure 9 visualizes value estimates for correct and incorrect trajectories across ten randomly sampled mathematical problems.
- Resources: Table 2 summarizes the resources and licenses used, with resource names linking to their HuggingFace pages.
- Implementation: The implementation section provides snapshots of exact execution commands rendered from the actual training scripts.