Soft Adaptive Policy Optimization

Chang Gao; Chujie Zheng; Xiong-Hui Chen; Kai Dang; Shixuan Liu; Bowen Yu; An Yang; Shuai Bai; Jingren Zhou; Junyang Lin

Source-linked AI summary

Soft Adaptive Policy Optimization

Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, Junyang Lin

arXiv:2511.20347v2Nov 25, 2025cs.LGcs.AIcs.CL

TL;DR

Stable policy optimization for LLM reinforcement learning is difficult because token-level importance ratios can vary substantially, especially in MoE models, and hard clipping discards or admits learning signals unevenly. SAPO replaces hard clipping with a temperature-controlled soft gate, combining token adaptivity with sequence coherence and asymmetric treatment of negative-token updates. The paper reports improved stability and Pass@1 performance on reasoning benchmarks, plus consistent gains across Qwen3-VL tasks and model scales.

Problem
High-variance token importance ratios make stable and effective group-based policy optimization difficult, while hard clipping limits the trade-off between suppressing noisy updates and retaining valid learning signals.
Method
SAPO uses a temperature-controlled token-level soft gate with asymmetric positive and negative temperatures to attenuate off-policy updates smoothly while preserving near-on-policy gradients.
Results
SAPO improves training stability and task performance versus GSPO and GRPO, with higher Pass@1 performance under comparable budgets and consistent gains across Qwen3-VL tasks and scales.
Takeaways & Limitations
SAPO provides a more stable and informative optimization signal by preserving useful gradients while selectively suppressing high-variance updates.
Takeaways & Limitations
The SAPO–GSPO sequence-level connection relies on assumptions including low intra-sequence dispersion of token log-ratios.

Abstract

from arXiv · show

Reinforcement learning (RL) plays an increasingly important role in enhancing the reasoning capabilities of large language models (LLMs), yet stable and performant policy optimization remains challenging. Token-level importance ratios often exhibit high variance-a phenomenon exacerbated in Mixture-of-Experts models-leading to unstable updates. Existing group-based policy optimization methods, such as GSPO and GRPO, alleviate this problem via hard clipping, making it difficult to maintain both stability and effective learning. We propose Soft Adaptive Policy Optimization (SAPO), which replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful learning signals. Compared with GSPO and GRPO, SAPO is both sequence-coherent and token-adaptive. Like GSPO, SAPO maintains sequence-level coherence, but its soft gating forms a continuous trust region that avoids the brittle hard clipping band used in GSPO. When a sequence contains a few highly off-policy tokens, GSPO suppresses all gradients for that sequence, whereas SAPO selectively down-weights only the offending tokens and preserves the learning signal from the near-on-policy ones, improving sample efficiency. Relative to GRPO, SAPO replaces hard token-level clipping with smooth, temperature-controlled scaling, enabling more informative and stable updates. Empirical results on mathematical reasoning benchmarks indicate that SAPO exhibits improved training stability and higher Pass@1 performance under comparable training budgets. Moreover, we employ SAPO to train the Qwen3-VL model series, demonstrating that SAPO yields consistent performance gains across diverse tasks and different model sizes. Overall, SAPO provides a more reliable, scalable, and effective optimization strategy for RL training of LLMs.

1 Introduction

Group-based RL for LLMs faces unstable updates from high-variance token importance ratios, while hard clipping trades off noisy gradients against discarded learning signals. SAPO replaces hard clipping with smooth, adaptive gating that preserves useful updates and selectively suppresses problematic tokens.

High-variance token importance ratios, amplified in MoE models by routing heterogeneity and long responses, increase the likelihood of unstable updates.
Hard clipping stabilizes optimization by zeroing gradients outside a fixed band, but tight bands discard valid samples while loose bands admit noisy off-policy gradients.
SAPO replaces hard clipping with a temperature-controlled soft gate that smoothly attenuates off-policy token updates instead of truncating them.
SAPO is sequence-coherent and token-adaptive: under low within-sequence log-ratio dispersion, its average gate approximates a smooth sequence-level gate.
When tokens are heterogeneous or outliers, SAPO down-weights only offending tokens while retaining informative gradients from near-on-policy tokens in the same sequence.
SAPO shows improved stability and task performance versus GSPO and GRPO, sustaining coherent learning longer and reaching higher Pass@1 accuracy before divergence.

2 Preliminaries

The preliminaries define autoregressive language-model policies, group-based sampling and advantages, and sequence-level versus token-level policy objectives. Figure 1 frames their comparison through surrogate values and gradient weights as functions of policy ratios.

An autoregressive language model parameterized by θ is modeled as a stochastic policy πθ over token sequences generated for queries q from a query set D.
GRPO samples G responses from the behavior policy, computes group rewards, and optimizes a token-level objective weighted by importance ratios and group-normalized advantages.
The group-normalized advantage bAi,t is shared across tokens within a response, while G denotes the number of responses in a group.
GSPO uses a sequence-level optimization objective and applies clipping at the sequence level rather than independently per token.
Length normalization in GSPO reduces variance and places sequence scores on a consistent numerical scale across responses.
Figure 1 compares policy-update objectives under positive advantage using surrogate objective values and gradient weights plotted against policy ratios.

3 Soft Adaptive Policy Optimization

SAPO combines a token-level soft trust region with sequence-level coherence and asymmetric attenuation for positive and negative updates. Its smooth gate preserves near-on-policy learning signals while reducing unstable off-policy updates.

SAPO replaces hard clipping with a temperature-controlled sigmoid gate that smoothly attenuates token-level updates as importance ratios depart from one.The gate preserves the unclipped gradient at the on-policy point and decays smoothly away from it.
Under small steps and low within-sequence log-ratio dispersion, average token gates concentrate into a smooth sequence-level gate resembling GSPO.This provides sequence-level coherence while retaining SAPO’s continuous trust region.
When a few tokens become highly off-policy, SAPO down-weights those tokens while retaining informative gradients from near-on-policy tokens.Unlike sequence-wide suppression, this preserves useful learning signals and improves sample efficiency.
Relative to GRPO, SAPO replaces fixed-range token clipping with smooth scaling that balances useful learning signals against unstable policy shifts.The resulting updates avoid zeroing gradients outside a hard clipping range.
The method uses distinct temperatures for positive and negative tokens, with τneg > τpos because negative updates can diffuse across many irrelevant vocabulary logits.This asymmetric design makes negative-token gradients decay more rapidly to improve stability and performance.

4 A Gating-Function Perspective on SAPO’s Connections to GRPO and GSPO

SAPO uses algorithm-specific gating to combine sequence-level coherence with token-level adaptivity. Its smooth gates approximate GSPO under low dispersion while avoiding GRPO’s all-or-nothing clipping and preserving informative gradients for near-on-policy tokens.

Unified gating formulation: SAPO formulates policy optimization with an algorithm-specific gating function applied to token importance ratios.The unified surrogate defines f_i,t as the algorithm-specific gate, while sequence ratios use a geometric mean of token ratios.
Connections to GSPO and GRPO: GSPO uses a token-invariant gate within each sequence, whereas SAPO and GRPO use token-dependent gates.This distinction determines whether off-policy behavior is handled uniformly across a sequence or separately for each token.
SAPO–GSPO connection: Under low intra-sequence log-ratio dispersion, SAPO’s average token gate is well-approximated by a smooth sequence-level gate.The analysis invokes small steps and low dispersion, with the average-gate approximation obtained through a second-order Taylor expansion.
Empirical assumptions: On more than 10^5 sequences and 10^9 tokens, token ratios concentrated around 1 and per-sequence variance typically stayed below 0.02 across MoE and dense models.The MoE distribution was relatively wider than the dense-model distribution, while both supported the assumptions in most cases.
SAPO–GSPO connection: When sequences contain heterogeneous or outlier tokens, SAPO selectively down-weights those tokens while retaining informative gradients from near-on-policy tokens.GSPO instead suppresses gradients for the sequence when hard clipping is triggered by a few off-policy tokens, reducing sample efficiency.
SAPO–GRPO connection: GRPO assigns full gradients inside its clipping band and zero gradients outside it, whereas SAPO shrinks updates smoothly as importance-ratio deviation grows.The smooth gate avoids the discontinuity and gradient vanishing associated with GRPO’s binary trust region.
Empirical comparison: SAPO maintained stable learning and achieved higher final performance than GSPO and GRPO-R2, which experienced early training collapse.Figure 4 compares training reward and validation performance for the cold-start Qwen3-30B-A3B-Base model.

5 Experiments

Controlled mathematical-reasoning experiments show that SAPO maintains more stable training and achieves stronger performance than GSPO and GRPO-R2, while assigning higher temperatures to negative-token updates improves stability. Qwen3-VL experiments further show steady gains over both baselines under equal compute budgets.

Controlled mathematical-reasoning experiments: SAPO maintains stable training and achieves higher final performance than GSPO and GRPO-R2 across AIME25, HMMT25, and BeyondAIME.GSPO and GRPO-R2 exhibit early-stage training collapse, whereas SAPO remains stable and performs better across all reported benchmarks.
Temperature ablation: τneg = 1.05 > τpos = 1.0 produces the most stable training, while τneg = 0.95 < τpos = 1.0 produces the most instability.The equal-temperature configuration is also evaluated, but the reported stability ranking favors higher temperatures for negative tokens.
Qwen3-VL training: SAPO consistently improves Qwen3-VL-30B-A3B validation performance and outperforms GSPO and GRPO-R2 under equal compute budgets.The comparison reports training rewards and mean validation performance on AIME25, LiveCodeBench v6, ZebraLogic, and MathVision.

6 Conclusion

SAPO replaces hard-clipped policy optimization with smooth, temperature-controlled, token-adaptive gating and asymmetric treatment of negative-token gradients. Across mathematical reasoning and Qwen3-VL experiments, it extends stable training and improves performance across tasks, model sizes, and architectures.

Method: SAPO uses a temperature-controlled soft gate and asymmetric temperatures to regulate token updates more smoothly than hard-clipped policy optimization.The method is designed to address instability and inefficiency associated with hard clipping while preserving a more informative optimization signal.
Results: SAPO extends stable training and achieves higher Pass@1 performance under comparable budgets on mathematical reasoning benchmarks.The conclusion reports these gains as empirical results from controlled experiments.
Results: SAPO delivers consistent improvements across diverse text and multimodal tasks, model sizes, and dense or MoE architectures.These large-scale experiments use the Qwen3-VL model family.
Conclusion: The results support smooth and adaptive gating as a promising direction for more robust and effective RL training of large language models.This is the paper’s stated broader conclusion.