PaperlayerView exact revision on arXiv

Source-linked AI summary

Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance

Zhuowen Han, Jinwei Xiao, Zhengxi Lu, Renren Jin, Zhiyuan Yao, Yuxin Liu, Hongyan Hao, Yueqing Sun, Yu Yang, Qi GU, Xunliang Cai, Deyi Xiong

arXiv:2608.00782v1cs.CL

TL;DR

GRPO can lose learning signals on zero-variance groups, while naive OPD combinations may limit exploration and suppress useful updates. RSTG selectively guides difficult samples and tokens and adds teacher-generated correct trajectories, improving over naive GRPO+OPD by 4.02% on math and 3.05% on code.

  • Problem

    GRPO provides sparse rewards and vanishing gradients on groups whose responses receive identical rewards, motivating denser supervision.

  • Method

    RSTG selectively applies confidence-weighted OPD to negative zero-variance prompts and informative tokens, while adding SFT on teacher-generated correct trajectories.

  • Results

    RSTG consistently improves over naive GRPO+OPD across mathematics and code, achieving +4.02% on mathematics and +3.05% on code.

  • Takeaways & Limitations

    RSTG enables effective integration of GRPO and OPD, with improvements confirmed across benchmarks in two domains and three model pairs.

  • Takeaways & Limitations

    The experiments are limited to scales feasible under computational constraints and require a meaningful capability gap between teacher and student.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models (LLMs). While Group Relative Policy Optimization (GRPO) is widely adopted, it suffers from sparse reward signals and loses gradients entirely when all responses within a group receive identical rewards. On-policy distillation (OPD) offers a natural remedy by providing dense, token-level supervision from a teacher model. However, naively combining GRPO with OPD leads to degraded performance, due to three underlying causes: not all samples benefit from distillation; fitting too quickly to the teacher undermines the exploratory capacity of RL; and OPD's advantages are asymmetric, suppressing most tokens. To address these challenges, we propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most. At the sample level, OPD is restricted to negative zero-variance prompts with each sample weighted by the teacher's confidence score. At the token level, distillation targets only tokens with high student entropy or large teacher-student divergence. We further augment training with SFT on correct trajectories generated by the teacher model, injecting positive gradient signals where RL yields none. Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and +3.05% on code.

1 Introduction

The introduction argues that naive GRPO+OPD fails because distillation is indiscriminate, teacher-bounded, and asymmetrically suppresses token-level learning signals. RSTG selectively applies teacher guidance to recover these signals and improves performance over naive GRPO+OPD on mathematics and code.

  • Motivation: GRPO produces sparse rewards and vanishing gradients when all rollouts in a group are correct or incorrect, creating positive and negative zero-variance prompts.GRPO applies a normalized scalar advantage uniformly to every token.
  • Motivation: Naive GRPO+OPD fails because OPD benefits samples unevenly, quickly caps performance at the teacher’s level, and suppresses learning through asymmetric token advantages.Student-generated prefixes can also make OPD gradients unreliable because distillation is conditioned on potentially erroneous local context.
  • RSTG: RSTG restricts OPD to negative zero-variance prompts where the student fails but the teacher succeeds, weighting each distillation signal by the teacher’s mastery.This selective setting uses only 3.63% of the full data and outperforms standard OPD on all samples.
  • RSTG: RSTG also targets distillation at tokens with high student entropy or large teacher-student divergence and adds SFT on correct teacher-generated trajectories.These mechanisms provide selective token-level guidance and positive gradients where RL produces none.
  • Results: +4.02% on mathematics and +3.05% on code are RSTG’s improvements over naive GRPO+OPD across Qwen2.5 and Qwen3 model families.RSTG also mitigates advantage asymmetry, slows convergence toward the teacher, and prevents abrupt response length inflation.

2 Related Work

On-policy distillation transfers teacher capabilities through dense, token-level supervision on student-sampled trajectories. Related work combines reinforcement learning with distillation, including GKD-style and policy-gradient-style OPD paradigms.

  • On-Policy Distillation (OPD): OPD samples trajectories from the student and aligns its behavior with the teacher’s token-level logit distribution, providing dense on-policy supervision.This supervision naturally complements GRPO.
  • On-Policy Distillation (OPD): GKD-style OPD directly minimizes token-level KL divergence, whereas PG-style OPD uses per-token reverse KL as a dense reward with negated reverse-KL advantage.The paper adopts the PG-style paradigm.
  • Combining RL and OPD: Prior work integrates reinforcement learning with knowledge distillation, beginning with GKD for text summarization and extending through KDRL.More recent studies investigate reinforcement learning with on-policy self-distillation.
  • Combining RL and OPD: A growing body of work studies combining reinforcement learning with on-policy self-distillation, including contributions by Yang et al., Li et al., and Lu et al.The supplied passage indicates that this paradigm still faces significant challenges, but does not specify them here.

3 Preliminaries

The preliminaries unify GRPO, OPD, and SFT within a common reinforcement-learning framework by expressing their updates through policy gradients and method-specific advantage functions. GRPO uses group-average rewards as a baseline, OPD assigns token-level credit through teacher–student divergence, and SFT treats teacher-generated tokens as uniformly positive supervision.

  • The framework defines D as the input distribution and πθ and π∗ as the student and teacher policies, respectively.
  • Policy-gradient updates use token-level relative advantages, with rewards measuring the quality of sampled response trajectories.
  • GRPO avoids PPO’s learned critic by using the average reward across multiple responses sampled for the same prompt as the baseline.
  • OPD trains on student-generated trajectories by minimizing reverse KL divergence, with teacher–student log-probability differences providing token-level advantages.
  • SFT is framed as an off-policy RL method using teacher-generated trajectories and a constant advantage, giving every teacher token uniform positive credit.

4 Method

RSTG routes negative zero-variance prompts to adaptive teacher guidance while retaining standard GRPO otherwise. It recovers collapsed learning signals through confidence-weighted, high-value-token OPD and teacher-generated SFT on correct trajectories.

  • RSTG framework: RSTG activates on negative zero-variance prompts and otherwise follows standard GRPO, combining confidence-weighted OPD, high-quality-token updates, and teacher-generated SFT.The three components operate at sample, token, and trajectory levels.
  • Sample selection: 3.63% of D yields the best OPD performance when training is restricted to samples where the teacher excels.Dsw contains 9k student-failed samples, while Dswtr contains 2k samples where the teacher achieves mean@8 = 1.
  • Confidence-weighted OPD: OPD targets negative zero-variance prompts, where failed rollouts collapse GRPO advantages to zero, and weights token-level advantages by the teacher’s mean@8 confidence score ω_i.The weighting coefficient gives greater optimization weight to tokens when the teacher is more confident.
  • Token selection: High-value tokens are selected using high student entropy or large teacher-student discrepancy, then OPD is applied to the top-k% ranked by the combined Soft-OR score.This selection slows convergence toward the teacher and reduces gradient noise while retaining tokens carrying uncertainty or richer information.
  • Auxiliary SFT: RSTG adds an auxiliary SFT loss on teacher-generated correct trajectories for negative zero-variance prompts to inject positive gradients where RL provides no learning signal.The SFT objective is equivalent to assigning uniform positive advantage A^SFT_t = 1 to every teacher-trajectory token.

5 Experiments

Experiments across three teacher–student pairs show that RSTG consistently improves over naive GRPO+OPD on mathematical reasoning and code generation. Additional analyses indicate that RSTG preserves learning signals, slows teacher convergence, controls response length, and depends on selective OPD design.

  • Main Results: +4.02%, +3.52%, and +0.89% are RSTG’s gains over naive GRPO+OPD across the three mathematical model pairs.Naive GRPO+OPD changes relative to standard GRPO were −0.2%, −1.57%, and +0.36%, respectively.
  • Main Results: +2.56%, +3.05%, and +1.97% are RSTG’s further gains over naive GRPO+OPD across the three code-generation model pairs.Naive GRPO+OPD itself improves over standard GRPO by +4.52%, +7.44%, and +9.64%, respectively.
  • Training Dynamics: RSTG yields higher advantage values than naive GRPO+OPD and alleviates OPD-induced advantage asymmetry, allowing more tokens to receive positive learning signals.Training curves show RSTG outperforming nearly all baselines at every step, although gains are less pronounced for the small-gap 4B–4B pair.
  • Training Dynamics: At step 200, teacher–student top-k overlap is 69.7% for OPD, 67.6% for naive GRPO+OPD, and 65.81% for RSTG, indicating slower convergence.RSTG also maintains response length comparable to GRPO while mitigating OPD’s abrupt length inflation.
  • Ablation Studies: Removing teacher-guided advantage weighting and token selection lowers Pair 1’s average mathematical accuracy to 51.74%, below ReLIFT’s 52.52%.The ablation attributes the drop to inappropriate sample targeting and premature convergence toward the teacher distribution.

6 Conclusion

RSTG addresses performance degradation from naively combining GRPO and OPD by selectively distilling difficult, insufficiently mastered samples according to teacher proficiency. It further uses fine-grained token selection and auxiliary SFT to slow convergence, reduce gradient noise, and inject positive learning signals.

  • Adaptive teacher guidance: RSTG selectively applies OPD to difficult samples that the student has not yet mastered, weighting each by the teacher’s degree of proficiency.This targets distillation where it is needed rather than applying it indiscriminately.
  • Adaptive teacher guidance: Fine-grained token selection slows convergence and reduces gradient noise during distillation.The method operates at token level to control the pace and quality of learning.
  • Auxiliary supervision: An auxiliary SFT objective injects positive gradient signals and provides a global perspective on the solution space.SFT complements selective OPD with an additional training signal.

Limitations

RSTG is evaluated in math and code, but its applicability beyond these domains and at larger model scales remains untested. Experiments are limited by computational resources and OPD’s requirement for a meaningful teacher–student capability gap.

  • Experiments cover two domains, math and code, demonstrating the method’s generalizability across these settings.
  • The method’s applicability to other settings, such as agentic tasks, remains to be explored.
  • Computational constraints and OPD’s need for a meaningful teacher–student capability gap restrict experiments to the largest feasible scale.
  • Larger model scales are left for future work, although the authors expect RSTG to remain effective there.

A Appendix · A.1 Preliminaries

OPD trains the student on its own generated trajectories by minimizing reverse KL divergence to the teacher. Its gradient can be written in an RL-like form, with a token-level advantage enabling credit assignment.

  • A.1 Preliminaries: OPD uses student-generated trajectories as the basis for distillation.The student generates its own trajectories before comparing its behavior with the teacher.
  • A.1 Preliminaries: The distillation objective minimizes reverse KL divergence between student and teacher distributions on those trajectories.This defines the core OPD training objective.
  • A.1 Preliminaries: Expanding the KL divergence and applying probability’s chain rule yields a token-level decomposition.The derivation converts sequence-level divergence into conditional token terms.
  • A.1 Preliminaries: Taking the gradient with respect to θ and applying the log-derivative trick expresses OPD as an expectation over student trajectories.The resulting gradient uses the identity ∇θE[f(y)] = E[f(y)∇θ log πθ(y)].
  • A.1 Preliminaries: The resulting OPD gradient has the same form as the reinforcement-learning objective, unifying OPD with the RL framework.This correspondence is stated by comparing Eq. (17) with Eq. (2).
  • A.1 Preliminaries: OPD defines its token-level advantage as the negative teacher-student log-probability difference.AOPD_t = −(log πθ(yt | x, y<t) − log π∗(yt | x, y<t)).
  • A.1 Preliminaries: This OPD advantage enables token-level credit assignment during training.Each token receives supervision according to the teacher-student probability discrepancy.

A.2 Method

RSTG selects high-value tokens using a differentiable Soft-OR of normalized student entropy and teacher-student divergence. This targets tokens where either signal is large while avoiding double-counting when both are large.

  • Mitigating Premature Convergence via Token Selection: Tokens with large h_t or d_t are treated as high-value for distillation.The method combines these criteria through a differentiable Soft-OR selection score.
  • Mitigating Premature Convergence via Token Selection: The Soft-OR score becomes large when either normalized h_t or d_t is large, while avoiding double-counting when both are large.Both h_t and d_t are min-max normalized to [0, 1] before computing the selection score.

A.3 Experiment Details … A.7 Ablation Studies

The experiments evaluate RSTG across controlled datasets, baselines, implementation settings, training dynamics, and component ablations. Ablations show that each RSTG component contributes meaningfully to performance, while training-cost analysis indicates manageable overhead.

  • A.3 Experiment Details: The APPS evaluation set contains 500 uniformly sampled instances, evenly distributed across Introductory, Interview, and Competition difficulty levels.The full APPS dataset contains 10,000 samples.
  • A.4 Baselines: Naive GRPO+OPD evaluates OPD on all samples, failed samples, or negative zero-variance prompts, selecting the best variant for each model pair.The OPD coefficient matches that used by RSTG.
  • A.5 Implementation Details.: RSTG trains for 550 mathematics steps and 400 code steps on 8 A100 GPUs, with model-pair runtimes ranging from approximately 2.5 to 5 days.The reported mathematics runtimes are approximately 3, 4, and 2.5 days; code runtimes are approximately 4, 5, and 3 days.
  • A.5 Implementation Details.: 12 hours is the additional cost of RSTG over standard GRPO for 550 steps on 8 A100 GPUs, because OPD and SFT run only on negative zero-variance prompts.These prompts initially account for approximately 15.6% of steps, with the proportion decreasing as the model improves.
  • A.5 Implementation Details.: 2 hours on 2 A100 GPUs is sufficient to pregenerate offline SFT data for the 57K training set with one rollout per prompt, adding no training-time cost.The paper characterizes the overall computational cost of RSTG as manageable.
  • A.6 Training Dynamics: Training dynamics are reported for one model pair from each Qwen3 and Qwen2.5 family on mathematical benchmarks throughout training in Figures 9 and 10.The figures compare the selected model pairs against standard GRPO.
  • A.7 Ablation Studies: Ablations on Qwen3-1.7B-Instruct→Qwen3-4B-Instruct progressively add RSTG components to naive GRPO+OPD, and each component contributes meaningfully to overall performance.Detailed ablation results are provided in Table 3 for mathematical reasoning benchmarks.
Loading 2608.00782v1…