Source-linked AI summary
Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, Tat-Seng Chua
TL;DR
GRPO offers stable but coarse reward-based credit assignment, while SDPO gives targeted supervision but can become unstable during prolonged training. SRPO routes correct samples to GRPO and failed samples to SDPO, adding entropy-aware weighting, and outperforms both baselines across five benchmarks and two model scales. The method also maintains moderate response lengths and lowers per-step compute cost.
Problem
GRPO’s uniform token credit assignment is coarse for failed rollouts, whereas SDPO’s targeted supervision can become unstable because correct-sample distillation is ambiguous and self-teacher reliability degrades.
Method
SRPO routes correct samples to GRPO and failed samples with teacher information to SDPO, using entropy-aware weighting to suppress unreliable distillation targets.
Results
Across five benchmarks and two model scales, SRPO consistently achieves the highest peak performance; on Qwen3-8B, its five-benchmark average is 77.4%, +3.4 over GRPO and +6.3 over SDPO.
Takeaways & Limitations
SRPO preserves SDPO’s rapid early improvement while maintaining GRPO’s long-horizon stability, with moderate response lengths and up to 17.2% lower per-step compute cost.
Takeaways & Limitations
The paper notes that improved reasoning quality may increase dual-use risks, so deployment should retain standard safety controls.
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models. While Group Relative Policy Optimization (GRPO) is widely adopted, its coarse credit assignment uniformly penalizes failed rollouts, lacking the token-level focus needed to efficiently address specific deviations. Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision that facilitates rapid early improvement, yet it frequently collapses during prolonged training. We trace this late-stage instability to two intrinsic flaws: self-distillation on already-correct samples introduces optimization ambiguity, and the self-teacher's signal reliability progressively degrades. To resolve these issues, we propose Sample-Routed Policy Optimization (SRPO), a unified on-policy framework that routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction. SRPO further incorporates an entropy-aware dynamic weighting mechanism to suppress high-entropy, unreliable distillation targets while emphasizing confident ones. Evaluated across five benchmarks and two model scales, SRPO achieves both the rapid early improvement of SDPO and the long-horizon stability of GRPO. It consistently surpasses the peak performance of both baselines, raising the five-benchmark average on Qwen3-8B by 3.4% over GRPO and 6.3% over SDPO, while simultaneously yielding moderate response lengths and lowering per-step compute cost by up to 17.2%.
1 Introduction
The introduction identifies complementary weaknesses in GRPO and SDPO and proposes routing samples to the method best suited to their learning status. SRPO combines this routing with entropy-aware weighting to achieve rapid improvement, stable long-horizon optimization, and stronger benchmark performance.
- Motivation: GRPO applies one scalar advantage uniformly across rollout tokens, limiting credit assignment for localized errors in failed samples.This coarse assignment is generally suitable for successful rollouts but cannot focus optimization on specific deviations.
- Motivation: SDPO provides denser logit-level guidance and improves faster early, but its advantage is not sustained under prolonged training.The paper attributes late-stage instability to optimization ambiguity on correct samples and declining self-teacher reliability.
- Motivation: Restricting SDPO updates to failed samples retains most of its benefit, whereas applying them only to correct samples degrades performance and destabilizes training.This diagnostic supports routing optimization signals according to sample correctness.
- Method: SRPO routes correct samples to GRPO and failed samples with teacher information to SDPO, while entropy-aware weighting downweights uncertain distillation targets.The design supports early dense correction and increasingly reward-aligned updates as more rollouts become correct.
- Results: Across five benchmarks and two model scales, SRPO achieves the highest peak performance while maintaining moderate response lengths and reducing per-step compute cost.On Qwen3-8B, the five-benchmark average reaches 77.4%, +3.4 over GRPO and +6.3 over SDPO; per-step compute cost falls by up to 17.2%.
2 Preliminaries
GRPO uses group-normalized rewards for reward-aligned but coarse sequence-level updates, while SDPO adds dense logit-level supervision from a feedback-conditioned self-teacher. Both methods remain on-policy, but they differ in whether supervision is driven by outcome rewards or teacher–student distribution discrepancies.
- Group Relative Policy Optimization: GRPO generates a group of on-policy rollouts and normalizes their scalar rewards to estimate each rollout’s relative advantage.The method does not require a learned critic.
- Group Relative Policy Optimization: Because GRPO assigns one sequence-level advantage uniformly across tokens, it reinforces or suppresses whole rollouts without identifying responsible tokens.This yields reward-aligned but coarse-grained credit assignment.
- Self-Distillation Policy Optimization: SDPO constructs a self-teacher from auxiliary rollout information and supplies dense logit-level supervision alongside scalar rewards.The student uses context x, while the teacher uses enriched context (x, f).
- Self-Distillation Policy Optimization: SDPO trains the student to match the self-teacher’s distribution along the original rollout by minimizing a logit-level divergence.The divergence may use KL, reverse KL, or Jensen–Shannon divergence.
- Self-Distillation Policy Optimization: The self-teacher re-scores the student’s own rollout under enriched context rather than generating a new trajectory, keeping the procedure on-policy.GRPO is reward-driven, whereas SDPO is teacher-driven and depends on self-teacher quality.
3 The SRPO
SRPO routes rollouts to GRPO or entropy-weighted SDPO according to learning status, combining sequence-level reward alignment with targeted token-level correction in one on-policy objective.
- Sample-level routing: SRPO routes correct rollouts to GRPO and incorrect rollouts with teacher information to SDPO, selecting the supervision signal suited to each sample.The framework uses the same on-policy trajectories while changing the advantage estimator across branches.
- Sample-level routing: The routing mask sends only incorrect rollouts with available teacher information to SDPO; all remaining rollouts use GRPO.
- Advantage estimators: GRPO provides sequence-level, reward-derived advantages, whereas SDPO provides teacher-derived, logit-level advantages for more localized correction.Both branches update the same policy on the same on-policy trajectories.
- Dynamic-weighted SDPO: Entropy-aware dynamic weighting reweights SDPO tokens so low-entropy teacher predictions contribute more and high-entropy predictions contribute less.The unnormalized weight is ˜wi,t = exp(−βHi,t), with β > 0 controlling sensitivity to entropy differences.
- Dynamic-weighted SDPO: The weighted SDPO loss preserves SDPO’s functional form while modulating each token’s contribution according to teacher confidence.
- Training procedure: SRPO’s token-normalized objective automatically shifts effective emphasis from dense SDPO correction early in training toward reward-aligned reinforcement as more rollouts become correct.This avoids an additional mixing hyperparameter and adapts to changing sample composition.
4 Experiments
Across five benchmarks and two Qwen3 scales, SRPO combines strong early learning with stable long-horizon improvement, while maintaining moderate response lengths and reducing long-horizon compute time.
- Experimental Setup: SRPO is evaluated on five benchmarks using Qwen3-4B and Qwen3-8B, with most analyses conducted at the 8B scale.The benchmarks are Chemistry, Physics, Biology, Materials, and Tool Use.
- Main Results: 77.4 avg@16 on Qwen3-8B and 74.2 on Qwen3-4B are SRPO’s highest five-benchmark averages within the 10h budget.These results exceed the corresponding GRPO and SDPO averages at both model scales.
- Main Results: SRPO matches SDPO’s rapid early improvement, continues improving over longer horizons, and ultimately exceeds both baselines’ peak performance.SDPO saturates early, whereas SRPO maintains steadier gains before reaching the highest reported results.
- Ablation Study: Advantage Mix is slightly better at 1h but trails SRPO by 2.5 points at 5h and 3.3 points at 10h.The control shows no further gain after 5h, while sample routing remains stronger over long horizons.
- Ablation Study: Dynamic weighting adds 0.4, 0.7, and 1.8 average-result points at 1h, 5h, and 10h, respectively, on top of sample routing.The widening gain is consistent with entropy-aware weighting becoming more useful as self-teacher reliability declines.
- Efficiency and Response Length: SRPO produces moderate response lengths between GRPO and SDPO and becomes 17.2% faster than GRPO per training step at 10h.At 1h, SRPO has a 17.4% overhead relative to GRPO, but at 5h and 10h it is faster than both baselines.
5 Conclusion
The paper proposes SRPO to combine reward-driven reinforcement with targeted self-distillation through sample-level routing and entropy-aware weighting. Across five benchmarks and two model scales, SRPO outperforms GRPO and SDPO while maintaining moderate response lengths and lower long-horizon per-step compute time.
- Conclusion: SRPO routes successful samples to GRPO and failed samples with teacher information to SDPO, while entropy-aware weighting suppresses unreliable distillation signals.This unified on-policy framework assigns each sample the optimization signal suited to its learning status.
- Conclusion: Across five benchmarks and two model scales, SRPO consistently outperforms pure GRPO and SDPO while preserving early efficiency and long-horizon stability.The conclusion also reports moderate response lengths and lower per-step compute time over long training horizons.
Ethics Statement
The work studies post-training optimization and reports no targeted harmful capabilities, but acknowledges dual-use risks and recommends standard safety controls. Its experiments use public benchmarks, automatic rewards, and no personal data or sensitive metadata.
- Ethics Statement: The work does not introduce new capabilities targeted at harmful applications, but improved reasoning may increase dual-use risks.The authors specifically mention misleading or unsafe content as possible risks.
- Ethics Statement: The authors recommend deployment with content moderation, policy-based filtering, and rate limiting.
- Ethics Statement: Experiments use publicly available benchmark datasets and automatic verifiable rewards without collecting personal data or involving human subjects.
- Ethics Statement: The training objective does not use private annotations or sensitive metadata.
- Related Context: The paper situates its work within RLVR, distillation, and self-distillation methods for language-model post-training.
B.1 Technical Setup
Experiments ran on a single 8-GPU H20 node with 768 GB total VRAM. The implementation used verl with FSDP2 for distributed training and SGLang for rollout generation.
- Experiments used a single node with 8 NVIDIA H20 GPUs interconnected via NVLink and 768 GB total VRAM.The software environment used GPU driver 550.144.03, CUDA 12.4, and PyTorch 2.8.0.
- The implementation was based on verl and used PyTorch FSDP2 for distributed training across GPUs.
- Rollout generation used SGLang instead of the original SDPO implementation’s vLLM backend for better environment compatibility.Both engines implement the same sampling algorithms and support identical decoding parameters.
B.2 Hyperparameters
The experiments retained SDPO’s prompt templates and benchmark splits for fair comparison, while SRPO largely inherited baseline hyperparameters. The five benchmarks comprise four Science Q&A tasks and one Tool Use task.
- Hyperparameters: SRPO inherited all non-learning-rate hyperparameters from SDPO and used a 5 × 10−6 learning rate between the GRPO and SDPO rates.This setting was intended to balance reward-driven and self-distillation signals.
- Prompt templates: The study used unchanged SDPO prompt templates, with a shared multiple-choice format for Science Q&A and a separate tool-calling format for Tool Use.
- Datasets: The evaluation covered five benchmarks: Chemistry, Physics, Biology, Materials, and Tool Use.The four Science Q&A benchmarks came from SciKnowEval’s Level 3 reasoning subset, while Tool Use came from ToolAlpaca.
B.5 Teacher Information Construction
Teacher information comes from correct sibling rollouts within each group and is used to route feasible incorrect samples to SDPO. Correct samples and cases without usable teacher context default to GRPO.
- Teacher information: Teacher information is a correct sibling rollout from the same group, excluding the rollout being updated from serving as its own teacher.The sibling’s full response text supplies the teacher information.
- Construction procedure: For each prompt, the policy generates G = 8 rollouts and identifies correct responses using the threshold ri ≥0.5.
- Fallback: When all group rollouts are incorrect, no teacher exists and every rollout is assigned to GRPO.
- Training dynamics: As training improves, the SDPO fraction decreases while the GRPO fraction increases correspondingly.The figure reports 5-step rolling means for the routing fractions.
- Routing rule: Only incorrect rollouts with available teacher information are routed to the SDPO branch; all other cases default to GRPO.
C Routing Statistics Over Training
SRPO shifts sample routing from SDPO toward GRPO as rollout correctness improves. This adaptive transition also reduces distillation-related compute overhead, while teacher information remains available for most samples.
- Routing composition: Approximately 40% of samples initially route to SDPO and 60% to GRPO, reflecting frequent incorrect rollouts early in training.As the policy improves, the increasing fraction of correct rollouts sends more samples to GRPO.
- Routing composition: The SDPO share decreases over training while the GRPO share increases, empirically supporting SRPO’s adaptive mixing behavior.SDPO contributes dense correction early, whereas more samples are handled by GRPO as policy correctness improves.
- Compute implications: Decreasing SDPO usage lowers per-step compute because self-teacher log-probabilities are computed only for SDPO-routed samples.The resulting compute time steadily decreases and eventually falls below standalone GRPO and SDPO.
- Teacher availability: Teacher information remains constructable for most samples throughout training, so routing changes are driven mainly by increasing rollout correctness rather than teacher unavailability.Fallback to GRPO from unavailable teacher information is relatively infrequent.