Source-linked AI summary
Group Sequence Policy Optimization
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, Junyang Lin
TL;DR
Large-language-model RL scaling is hindered by instability in GRPO, whose token-level importance weighting can produce collapse. GSPO instead uses sequence-likelihood ratios with sequence-level clipping, rewarding, and optimization, and achieves superior stability, efficiency, and performance, including for MoE training.
Problem
GRPO can suffer severe instability and catastrophic model collapse when scaling RL to large language models, limiting continued capability improvement.
Method
GSPO defines importance ratios from sequence likelihood and performs sequence-level clipping, rewarding, and optimization.
Results
GSPO demonstrates superior training stability, efficiency, and performance over GRPO, while eliminating the need for Routing Replay in MoE RL training.
Takeaways & Limitations
GSPO provides a scalable algorithmic foundation for large-scale RL and contributed to the performance improvements of the latest Qwen3 models.
Abstract
from arXiv · showhide
This paper introduces Group Sequence Policy Optimization (GSPO), our stable, efficient, and performant reinforcement learning algorithm for training large language models. Unlike previous algorithms that adopt token-level importance ratios, GSPO defines the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization. We demonstrate that GSPO achieves superior training efficiency and performance compared to the GRPO algorithm, notably stabilizes Mixture-of-Experts (MoE) RL training, and has the potential for simplifying the design of RL infrastructure. These merits of GSPO have contributed to the remarkable improvements in the latest Qwen3 models.
1 Introduction
Large-scale RL can expand language-model reasoning capabilities, but scaling it requires stable training. GSPO addresses instability in GRPO with sequence-level importance sampling and achieves stronger stability, efficiency, and performance.
- Large-scale RL enables language models to tackle sophisticated mathematics and programming problems through deeper, longer reasoning.
- Stable and robust training dynamics are essential for scaling RL, yet GRPO can cause catastrophic and irreversible model collapse in gigantic language models.
- GRPO instability arises from misapplied importance-sampling weights, whose high-variance noise grows with response length and is amplified by clipping.
- GSPO defines importance ratios from sequence likelihood and normalizes group rewards as advantages, aligning sequence-level rewarding with optimization.
- GSPO outperforms GRPO in stability, efficiency, and performance, stabilizes MoE RL training, and contributed to improvements in Qwen3 models.
2 Preliminaries
The preliminaries define language-model policies, likelihoods, rewards, PPO clipping, and GRPO's group-relative alternative to value-model-based optimization. They also identify the computational and estimation burdens associated with PPO.
- An autoregressive language model parameterized by θ is represented as the policy πθ, with x denoting a query and D the query set.
- A response likelihood is formed from token-level conditional probabilities, while a verifier assigns each query-response pair a reward r(x, y) ∈ [0, 1].
- PPO constrains updates relative to the old policy using token importance ratios, advantages, and a clipping range ε.
- PPO relies on a value model that adds substantial memory and computation demands and is difficult to scale reliably to longer, more complex tasks.
- GRPO avoids a value model by computing each response's relative advantage within a group of responses to the same query.
- Within GRPO, all tokens in a response share the same response-level advantage.
3 Motivation
Large rollout batches and mini-batch updates make RL off-policy, motivating clipping, but the paper argues that GRPO's token-level importance weighting is fundamentally ill-posed. The resulting noise accumulates on long responses and can cause irreversible collapse, motivating sequence-level optimization.
- Large models, sparse MoE activation, and long responses require large rollout batches, whose mini-batch updates create an off-policy setting.
- GRPO's objective is ill-posed because its importance-sampling weights are misapplied, especially for long-response training of large models.
- Standard importance sampling corrects distribution mismatch by reweighting samples from a behavior distribution while averaging over many samples.
- GRPO applies importance weights to individual tokens, but each weight is based on a single next-token sample and therefore fails to provide the intended distribution correction.
- Token-level weighting introduces high-variance gradient noise that accumulates over long sequences and is worsened by clipping.
- The resulting collapse can be irreversible despite checkpoint reversion, hyperparameter tuning, longer generation, or changing RL queries.
- Because rewards apply to entire sequences, the paper motivates matching the optimization unit to the reward unit through sequence-level importance weighting and optimization.
4 Algorithm
GSPO replaces GRPO’s token-level importance weighting with sequence-level likelihood ratios, clipping, and optimization aligned to sequence-level rewards. A token-level variant preserves GSPO’s numerical behavior while enabling per-token advantage customization.
- GSPO: GSPO defines its importance ratio from sequence likelihood and applies optimization directly at the sequence level.The sequence-level ratio reflects deviation between the sampling and current policies and aligns with sequence-level rewards.
- GSPO: Entire responses are clipped to exclude overly off-policy samples, matching sequence-level rewarding and optimization.Length normalization reduces variance and keeps the sequence-level ratio within a unified numerical range.
- Gradient analysis: GRPO assigns unequal token weights that can accumulate into unpredictable training effects, whereas GSPO weights all tokens in a response equally.The unequal GRPO weights vary across token positions, while GSPO removes this instability factor.
- GSPO-token: GSPO-token introduces token-wise advantage customization for settings such as multi-turn reinforcement learning.Its token-level ratio uses a stopped-gradient sequence ratio and has the same numerical value as the sequence-level ratio.
- GSPO-token: When every token in a response receives the same advantage, GSPO-token and GSPO are numerically identical in objective, clipping condition, and theoretical gradient.GSPO-token nevertheless allows advantages to be adjusted independently across tokens.
5 Experiments and Discussion
Experiments show that GSPO trains more stably and efficiently than GRPO, including for MoE models, while avoiding Routing Replay and supporting simpler RL infrastructure.
- MoE Training: GRPO requires Routing Replay for normal convergence in MoE RL, whereas GSPO does not.Routing Replay reuses activated experts but adds memory and communication overhead and can limit MoE capacity.
- Empirical Results: GSPO delivers stable training and better accuracy and benchmark performance than GRPO under the same training compute and consumed queries.Training remains stable while compute increases, the query set is updated, and generation length is extended.
- Clipping Fractions: GSPO achieves higher training efficiency than GRPO despite clipping a much larger fraction of tokens.The clipped-token fractions differ by two orders of magnitude, yet GSPO uses fewer tokens for gradient estimation and remains more efficient.
- MoE Training: GSPO’s sequence-level likelihood is less sensitive to individual-token likelihood fluctuations, resolving expert-activation volatility in MoE training.This enables conventional importance-ratio computation, normal convergence, and stable optimization without complex workarounds.
- RL Infrastructure: GSPO can use inference-engine likelihoods directly for optimization, avoiding training-engine recomputation when precision discrepancies exist.This is especially beneficial for partial rollout, multi-turn RL, and disaggregated training-inference frameworks.
6 Conclusion
GSPO defines sequence-level importance ratios and optimization for large-language-model reinforcement learning. It outperforms GRPO in stability, efficiency, and performance, particularly for large-scale MoE training.
- GSPO defines importance ratios from sequence likelihood and performs sequence-level clipping, rewarding, and optimization.
- GSPO demonstrates superior training stability, efficiency, and performance compared with GRPO, especially for large-scale MoE reinforcement learning.