Source-linked AI summary
S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models
Muzhi Dai, Chenxu Yang, Qingyi Si
TL;DR
Reasoning models often generate redundant chain-of-thought because outcome-only reinforcement learning overlooks whether intermediate steps are sufficient. S-GRPO serially evaluates early exits from one reasoning path and decays rewards by exit position, achieving shorter sequences with improved accuracy across diverse benchmarks.
Problem
Reasoning models produce redundant, lengthy thought sequences, while conventional outcome-reward RL overlooks the sufficiency of intermediate reasoning steps.
Method
S-GRPO generates one complete reasoning path, evaluates multiple serial early exits, and assigns higher rewards to earlier correct answers.
Results
35.4%∼61.1% sequence-length reduction accompanied 0.72%∼6.08% absolute accuracy improvement across five benchmarks and reasoning models including Qwen3 and Deepseek-distill.
Takeaways & Limitations
S-GRPO improves reasoning efficiency while maintaining or improving accuracy and is compatible with state-of-the-art reasoning models.
Abstract
from arXiv · showhide
As Test-Time Scaling emerges as an active research focus in the large language model community, advanced post-training methods increasingly emphasize extending chain-of-thought (CoT) generation length, thereby enhancing reasoning capabilities to approach Deepseek R1-like reasoning models. However, recent studies reveal that reasoning models (even Qwen3) consistently exhibit excessive thought redundancy in CoT generation. This overthinking issue arises from the inherent limitations of conventional outcome-reward reinforcement learning, which systematically overlooks the regulation of intermediate reasoning processes. This paper introduces Serial-Group Decaying-Reward Policy Optimization (S-GRPO), a novel reinforcement learning paradigm that enables models to implicitly evaluate the sufficiency of intermediate reasoning steps, thereby facilitating early exit in CoT generation. Unlike GRPO, which samples multiple possible reasoning paths in parallel (parallel group), S-GRPO only samples one reasoning path and serially selects multiple temporal positions from the path to exit thinking and directly generate answers (serial group). For correct answers within a serial group, rewards gradually decrease based on the exit positions along the reasoning path from front to back. This design encourages the model to produce more accurate and concise thoughts, while also incentivizing early thinking termination when appropriate. Empirical evaluations demonstrate that S-GRPO is compatible with state-of-the-art reasoning models, including Qwen3 and Deepseek-distill. Across diverse benchmarks such as GSM8K, AIME 2024, AMC 2023, MATH-500, and GPQA Diamond, S-GRPO achieves a substantial reduction in sequence length (35.4% - 61.1%) while simultaneously improving accuracy (absolute 0.72% - 6.08%).
1 Introduction
Reasoning models often overthink because outcome-only reinforcement learning rewards final correctness while overlooking whether intermediate reasoning is sufficient. S-GRPO addresses this by serially testing early exits and decaying rewards for later correct answers, improving both reasoning efficiency and accuracy.
- Motivation: Extended chain-of-thought generation can strengthen reasoning, but reasoning models frequently produce redundant and unnecessarily lengthy thought sequences.The redundancy increases computational overhead and can undermine accuracy by diverting reasoning toward incorrect pathways.
- Motivation: 0/1 outcome rewards reinforce correct final answers while failing to identify when intermediate reasoning has become sufficient.GRPO samples multiple reasoning chains in parallel, but its reward design overlooks intermediate inefficiencies.
- Approach: S-GRPO constructs a serial group by generating one complete reasoning path and adding early-exit answers at multiple positions along that path.This directly exposes different levels of reasoning sufficiency during reinforcement-learning training.
- Approach: Earlier correct exits receive higher rewards than later correct exits, encouraging higher-quality early reasoning and termination once sufficient reasoning is achieved.The method is designed to regulate intermediate reasoning rather than only final outcomes.
- Results: 0.72%∼6.08% absolute accuracy improvement accompanied 35.4%∼61.1% average token reduction across five benchmarks and four reasoning models.The experiments include GSM8K, AIME 2024, AMC 2023, MATH-500, and GPQA, using Qwen3 and Deepseek-series reasoning models.
2 Method
S-GRPO regulates intermediate reasoning by creating serial groups from one complete chain and multiple early-exit continuations. Position-dependent rewards favor correct earlier exits, while group-relative advantages update the policy toward concise reasoning without discarding the complete rollout.
- Method overview: S-GRPO uses rule-based outcome rewards to regulate intermediate reasoning and improve reasoning efficiency through sufficient-reasoning early exits.Earlier thinking exits receive relatively higher rewards, encouraging the model to terminate when its reasoning is sufficient.
- Serial-Group Generation: Unlike GRPO’s parallel sampling, S-GRPO intervenes at multiple positions within one complete chain so reinforcement learning can compare reasoning sufficiency along that path.The serial group combines the full thought rollout with early-exit responses.
- Serial-Group Generation: The full-thought rollout generates one complete reasoning path, then randomly selects uniformly distributed temporal positions for truncation during training.Random truncation exposes the model to diverse reasoning-sufficiency scenarios.
- Serial-Group Generation: At each selected position, truncated reasoning is followed by an answer rollout, producing intermediate answers for the serial group.The framework inserts an instruction to stop thinking and start answering at each truncation point.
- Serial-Group Generation: Oversampling filters queries so the serial group contains early-exit samples with correct answers.This is used to obtain the required training examples from early-exit thought rollouts.
- Decaying Reward Strategy: Correct-answer rewards decay with early-exit order, while incorrect answers receive zero reward.The design combines exponentially diminishing rewards for later correct answers with correctness-first optimization.
- Decaying Reward Strategy: The decaying-reward design balances reasoning sufficiency and efficiency by favoring reasoning sequences that are both accurate and concise.It specifically encourages adequate and correct reasoning steps at earlier stages of chain-of-thought generation.
- Advantage Computation and Parameter Update: S-GRPO computes each response’s advantage as its reward minus the serial-group mean, then broadcasts that sequence-level advantage to its tokens.The standard-deviation normalization used in GRPO is removed for training stability.
3 Experiments
S-GRPO is evaluated across multiple reasoning models, benchmarks, baselines, token budgets, and ablations. The experiments assess whether it improves accuracy while shortening reasoning trajectories and preserving reasoning capability.
- Experimental Setup: S-GRPO is compared with DEER, ConCISE, original GRPO, RL + Length Penalty, and ShorterBetter.These baselines include training-free, off-policy, and on-policy reinforcement-learning approaches for efficient reasoning.
- Experimental Setup: S-GRPO is evaluated on four large reasoning models and five benchmarks spanning mathematical and scientific reasoning.The models include DeepSeek-R1-Distill-Qwen-7B, DeepSeek-R1-Distill-Qwen-14B, Qwen3-8B, and Qwen3-14B; the benchmarks include GSM8K, AIME 2024, AMC 2023, MATH-500, and GPQA.
- Experimental Setup: The evaluation measures Accuracy (pass@1) and Token Count to assess correctness alongside inference length.Multiple evaluation runs are averaged because long-sequence generation is unstable and some benchmarks have limited sample sizes.
- Main Results: 35.4%–61.1% lower sequence length accompanies 0.72%–6.08% absolute accuracy improvement across five benchmarks and four reasoning models.The reported gains cover both in-domain mathematical tasks and the out-of-domain scientific reasoning benchmark GPQA.
- Comparison with Baselines: S-GRPO achieves the best combination of reasoning-length reduction and accuracy among the compared training-based efficient-reasoning methods.Against original GRPO, it attains comparable or better accuracy while shortening inference trajectories.
- Performance with Different Token Budgets: Across token budgets, S-GRPO generates shorter sequences with higher accuracy than vanilla CoT, while tight budgets produce especially higher accuracy at comparable length.Its accuracy and actual generation-length trends are smoother across budgets, and a representative example reaches the correct answer using less than half the reasoning budget.
4 Related Work
The paper situates S-GRPO within research on reasoning efficiency, outcome-reward reinforcement learning, and early-exit methods for reducing redundant chain-of-thought generation.
- Rule-based outcome-reward reinforcement learning uses binary correctness rewards and reduces training overhead by avoiding separate reward models.
- Reasoning-efficiency methods include training-free approaches, such as dynamic prompting, Best-of-N pruning, optimization, and early exit.
- DEER and S-GRPO both produce intermediate answers through early exit, but DEER decides during inference while S-GRPO learns the behavior through training.
- Training-based efficiency methods include variable-length-CoT supervision, length-reward reinforcement learning, concise-CoT construction, and shorter-correct-chain reward schemes.
5 Conclusion
The paper concludes that S-GRPO regulates intermediate reasoning with serial-group decaying rewards, improving efficiency while preserving or increasing accuracy across multiple benchmarks and reasoning models.
- S-GRPO reduces sequence length by 35.4%∼61.1% while improving accuracy by 0.72%∼6.08% across five benchmark datasets.
- S-GRPO uses rule-based outcome rewards to regulate intermediate reasoning and encourages earlier high-quality thoughts followed by prompt termination when sufficient.
- The method is compatible with Qwen3 and Deepseek-distill and is presented as a final optimization stage for post-training pipelines.
A An example of Training Data
The example illustrates how S-GRPO truncates a complete reasoning path at an early-exit position and assigns rewards according to the resulting answer.
- S-GRPO training data includes a complete reasoning path followed by a truncated thought rollout that produces an answer earlier.
- The example’s truncated rollout answers the cyclic-group embedding problem with n = 12.
- Figure 5 presents training-data truncation together with decaying reward assignment across early-exit positions.