PaperlayerView exact revision on arXiv

Source-linked AI summary

SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation

Zhenrui Yue, Huimin Zeng, Yueqi Wang, Yaokun Liu, Fengran Mo, Jinghan Zhang, Mung Yao Jia, Gyuseok Lee, Yang Zhang, Na Wei, Dong Wang

arXiv:2609.36742v1cs.AIcs.CL

TL;DR

RLVR’s sparse outcome rewards leave intermediate reasoning steps without token-level credit, while self-distillation can add dense feedback but a self-teacher may be overconfident. SIPO combines reward-based updates with contrastive self-teacher feedback, and outperforms RL and self-distillation baselines across reasoning and code-generation tasks without an external teacher or additional generation.

  • Problem

    RLVR’s outcome-level rewards lack token-level credit assignment for intermediate reasoning steps, while OPSD’s self-teacher can be overconfident and over-penalize long reasoning trajectories.

    Source: §1 · pp. 1–2

  • Method

    SIPO combines reward-based policy gradients with contrastive self-distillation, using the difference between self-teacher evaluations under correct and mistaken contexts as token-level feedback.

    Source: §1 · p. 2

  • Results

    SIPO outperforms RL and self-distillation baselines across math, science QA, tool use, and code generation, and remains stable on long math reasoning where self-distillation collapses.

    Source: §5 · p. 9

  • Takeaways & Limitations

    SIPO preserves direct task-reward optimization while redistributing credit across tokens, and requires neither an external teacher nor additional generation.

    Source: §1 · p. 2, §5 · p. 9

    For code generation without a reference answer, uniformly failed groups receive no contrastive feedback because no successful sibling rollout is available.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for improving large language models (LLMs) on various tasks, yet its sparse outcome rewards lack token-level credit assignment for intermediate steps. To address this, on-policy self-distillation (OPSD) leverages a self-teacher with privileged context to provide additional dense learning signals. However, because the self-teacher is often overconfident and imposes excessive penalties on long reasoning trajectories, OPSD frequently struggles in practice. To mitigate this, we propose self-instructing policy optimization (SIPO) with a contrastive self-teacher to provide dense credit. At each iteration, SIPO samples multiple rollouts per prompt from the current policy, scores them with environment rewards, and constructs two teacher contexts for each rollout by pairing the reference answer with mistakes made within the group. The model then re-evaluates its own responses under both contexts, using the difference between the two teacher log-probabilities as token-level feedback, so that biases shared by both contexts are expected to largely cancel. The resulting objective yields a token-level advantage for every rollout: the reward still sets the main direction of each update while the self-teacher redistributes credit across tokens. Even in groups where every rollout fails and group-relative advantages vanish, SIPO still provides a learning signal. By preserving direct optimization of the task reward while providing dense, token-level feedback, this approach bridges reinforcement learning and on-policy self-distillation. Extensive experiments across multiple reasoning and code-generation benchmarks demonstrate that SIPO outperforms both RLVR and OPSD baselines without an external teacher or additional generation.

1 INTRODUCTION

RLVR provides task-grounded rewards but lacks token-level credit, while on-policy self-distillation offers dense feedback with drawbacks of its own. SIPO combines reward-based updates with contrastive self-teacher feedback, and experiments show it outperforms RLVR and OPSD baselines under comparable computational budgets.

1 INTRODUCTION

  • RLVR’s trajectory-level outcome reward lacks token-level credit assignment, and group-relative advantages vanish when every sampled rollout fails.Such uniformly failed groups account for 20–30% of groups early in Qwen3-8B training on DAPO-Math-17k and about 10% at the end.Source: §1 · p. 1
  • OPSD uses privileged self-teacher context for dense supervision, but its overconfidence can over-penalize long reasoning trajectories.Its distillation signal may also be sparse or absent when successful rollouts are rarely sampled or the teacher provides no meaningful supervision.Source: §1 · p. 2
  • SIPO’s contrastive self-teacher uses the difference between correct- and mistaken-context log-probabilities as token feedback, with shared biases expected to largely cancel.Because the contrast does not require a successful rollout, uniformly failed groups still contribute a learning signal.Source: §1 · p. 2
  • SIPO combines reward-based policy gradients and on-policy self-distillation through a token-level advantage that adds contrastive feedback to group-relative advantage.It samples and scores multiple rollouts, pairs reference answers with group mistakes, and re-evaluates responses under both contexts without generating new text.Source: §1 · p. 2
  • SIPO outperforms RLVR and OPSD baselines on STEM reasoning and code-generation benchmarks under comparable computational budgets.Analyses also suggest its contrastive self-teacher mitigates standard self-teacher overconfidence and supplies a signal to uniformly failed groups.Source: §1 · p. 3

2 RELATED WORK

Prior RL methods reduce variance or memory overhead with group- or batch-based reward baselines, but retain coarse trajectory-level credit. On-policy distillation supplies token-level supervision, and SIPO combines it with reward-based learning.

2.1 REINFORCEMENT LEARNING

  • RL methods including GRPO use group- or batch-level reward statistics to reduce memory overhead or variance, but outcome rewards still provide coarse trajectory-level credit.The section places SIPO as a way to combine reward-based policy gradients with dense token-level supervision while retaining direct task-reward optimization.Source: §2.1 · p. 3

2.2 ON-POLICY DISTILLATION

  • On-policy distillation trains on the student’s rollouts with token-level teacher targets, while later work explores privileged self-teachers and correct-versus-incorrect teacher contrasts.The cited contrastive approaches leave uniformly failed groups without learning signals; SIPO combines sparse outcome signals with dense token-level supervision without additional generation.Source: §2.2 · p. 3

3 METHODOLOGY

SIPO combines group-based reward optimization with token-level feedback from a self-teacher contrasting correct and incorrect contexts. Its token-level advantage preserves reward direction while redistributing credit and can provide a signal when all sampled rollouts fail.

3.1 PRELIMINARIES

  • The policy samples a group of responses per prompt and scores each with a binary reward before group-based optimization.The preliminaries describe this as the standard RLVR setting.Source: §3.1 · pp. 3–4
  • GRPO assigns every token in a rollout the same group-relative advantage, while on-policy distillation uses teacher-scored tokens to provide dense learning signals.The described OPD objective estimates a reverse-KL gradient, but does not directly optimize task reward.Source: §3.1 · p. 4

3.2 SELF-INSTRUCTING POLICY OPTIMIZATION

  • For each rollout, SIPO contrasts a positive context containing the reference answer with a negative context containing a group mistake, often the rollout’s own wrong answer.Without a reference answer, it uses successful and failed sibling rollouts; rollouts without a valid pair receive no contrastive feedback.Source: §3.2 · pp. 4–5
  • SIPO evaluates the same response tokens under both augmented contexts, adding two teacher forward passes without additional generation.The teacher parameters are synchronized with the current policy every K training steps.Source: §3.2 · p. 5
  • SIPO adds clipped contrastive teacher evidence to each rollout’s group-relative advantage, preserving reward direction while assigning token-level feedback even when all rollouts fail.In uniformly failed groups, the group-relative advantage is zero, so the clipped teacher evidence supplies the learning signal.Source: §3.2 · pp. 4–5
  • The math table compares Qwen3-8B methods across six benchmarks, while the science QA and tool-use table reports best avg@16 within 1h and 5h.The math table uses different avg@k protocols across benchmarks.Source: Table 2 · p. 6, Table 1 · p. 6

4 EXPERIMENTS

Experiments compare SIPO with reward-optimization and self-distillation baselines across math, science QA, tool use, and code generation. SIPO shows stronger benchmark results, while training-dynamics and ablation analyses examine its contrastive teacher and reward-based updates.

4 EXPERIMENTS

  • SIPO is evaluated on math, science QA, tool use, and code generation, using Qwen3-8B throughout and Olmo3-7B-Instruct for science QA and tool use.Code correctness is determined by unit tests; held-out IFEval, ArenaHard-v2, and MMLU-Pro measure generalization.Source: §4 · p. 6

4.1 RESULTS

  • SIPO reaches the best average math accuracy, 59.4, improving over GRPO by 2.1% and RLSD by 2.7%.It also improves over GRPO on all six math benchmarks.Source: §4.1 · pp. 6–7
  • On Chemistry and Materials, SIPO converges faster and reaches higher final reward than SDPO and GRPO; on math it stays above GRPO while RLSD stalls until step 50.The authors attribute SIPO’s continued reward optimization to its policy-gradient term and suggest that contrastive credit adds to the reward signal without slowing learning.Source: Figure 3 · p. 7, Figure 5 · p. 8, §4.1 · p. 8
  • Table 3 reports SIPO improving LiveCodeBench v6 accuracy from 27.9 to 56.3, above the strongest baseline, RLSD at 55.4.Its average holdout score is 42.7, close to the base model’s 43.5; among trained methods, it loses only 0.8% on average.Source: Table 3 · p. 7, §4.1 · p. 8
  • 70.4 average accuracy gives SIPO gains over SDPO of 1.3%, RLSD of 7.0%, and GRPO of 7.7% across science QA and tool-use settings.SIPO is best in 6 of 10 settings; with Olmo3-7B, it improves over SDPO by 1.4% and GRPO by 6.7% on average.Source: §4.1 · p. 7
  • On math, SDPO’s reward, entropy, and response length decrease as its gradient norm grows; responses shrink below 1,000 tokens.The authors hypothesize that answer-aware self-distillation shortens reasoning needed for math, while shared answer-directed preferences in SIPO’s two contexts largely cancel.Source: Figure 4 · p. 8, §4.1 · pp. 8–9
  • Replacing RLSD’s single-teacher evidence with contrastive evidence raises its math average from 56.7 to 58.3, above GRPO’s 57.3.Adding contrastive evidence to every rollout, including uniformly failed groups, raises the average further to 59.4.Source: Table 4 · p. 9, §4.1 · p. 9
  • The single-teacher evidence has a negative mean of −0.002 to −0.004, while contrastive evidence fluctuates around zero with |mean| < 0.001.The authors interpret the contrast as canceling shared answer-directed preferences while leaving evidence specific to correct or incorrect answers.Source: §4.1 · p. 9

5 CONCLUSION

SIPO contrasts a correct and incorrect privileged context to provide dense credit while unifying reinforcement learning with on-policy self-distillation. Across math, science QA, tool use, and code generation, it outperforms RL and self-distillation baselines, remains stable on long math reasoning, and uses no external teacher or additional generation.

5 CONCLUSION

  • Across math, science QA, tool use, and code generation, SIPO outperforms RL and self-distillation baselines.It also forgets little on holdout evaluation.Source: §5 · p. 9
  • SIPO contrasts teacher evaluations under correct and incorrect privileged contexts, so shared biases largely cancel and even uniformly failed groups receive dense credit.Source: §5 · p. 9
  • SIPO remains stable on long math reasoning where self-distillation collapses and requires no external teacher or additional generation.Source: §5 · p. 9
  • SIPO adds only teacher forward passes and is a drop-in replacement for GRPO when environmental feedback is available.Source: §5 · p. 9

A METHOD DETAILS

SIPO constructs contrastive teacher contexts from reference answers and mistakes, or sibling rollouts when no reference is available, then uses a sign-preserving projection to keep teacher feedback from reversing reward direction.

A.1 CONSTRUCTION OF CONTRASTIVE PAIRS

  • When no reference answer is available, the teacher prompts use a successful sibling from the same group; science QA also uses sibling pairs when both exist.For the described sibling pair, both siblings differ from the rollout being evaluated, and environment feedback is not added to the prompts.Source: §A.1 · p. 13
  • Contrastive feedback is omitted unless both teacher prompts can be constructed, excluding uniformly correct groups and incorrect rollouts without valid judged answers.For code generation without a reference answer, uniformly failed groups also receive no contrastive feedback; science QA instead falls back to reference-answer pairs.Source: §A.1 · p. 13

A.2 SIGN-PRESERVING PROJECTION

  • The sign-preserving projection constrains the token-level advantage so the teacher term cannot reverse the direction set by reward.For binary rewards, it is inactive when κδ < 1/G; the supplied settings satisfy κδ ≤ 0.1 < 1/8, and uniformly failed groups remain unchanged.Source: §A.2 · p. 13

B EXPERIMENT SETTINGS

SIPO is evaluated in binary-reward settings with and without reference answers, across math, science QA, tool use, and code generation. The experiments compare on-policy RL and self-distillation baselines under task-specific datasets and training protocols.

B EXPERIMENT SETTINGS

  • The evaluation covers binary-reward RLVR tasks with reference answers and code generation where unit tests determine correctness without a reference answer.For code, reward is one only when all training tests pass, with separate held-out tests used for evaluation.Source: B EXPERIMENT SETTINGS · p. 14
  • SIPO is evaluated in math, science QA, tool use, and code generation, using Qwen3-8B across tasks and Olmo3-7B-Instruct for science QA and tool use.The math training set contains 14,116 DAPO-Math-17k problems; LiveCodeBench v6 uses held-out tests for evaluation.Source: B EXPERIMENT SETTINGS · p. 14
  • Comparisons include GRPO with asymmetric clipping, SDPO, and RLSD; methods take one on-policy step per generation batch except LiveCodeBench v6, where they take four mini-batch steps.The study also reports four-step off-policy GRPO on science QA and tool use.Source: B EXPERIMENT SETTINGS · p. 14
  • All settings use rollout group size G = 8 and a self-teacher snapshot synchronized every 10 training steps.Science QA and tool use report avg@16 against wall-clock training time, while math trains for 125 steps and reports the best checkpoint.Source: B EXPERIMENT SETTINGS · p. 14

C.1 RESULTS WITH OLMO3-7B

With Olmo3-7B, SIPO achieves the highest average accuracy on science QA and tool use, improving over SDPO and GRPO and obtaining the best result in most reported settings.

C.1 RESULTS WITH OLMO3-7B

C.2 OFF-POLICY GRPO

Off-policy GRPO does not account for SIPO’s gains, while reward curves show SIPO improving earlier on Physics and Biology.

C.2 OFF-POLICY GRPO

  • GRPO with four off-policy mini-batch steps scores 65.6 vs. 62.7 on average with Qwen3-8B and 57.4 vs. 59.8 with Olmo3-7B, both below SIPO’s 70.4 and 66.5.The results indicate SIPO’s gains do not stem from restricting GRPO to on-policy updates.Source: §C.2 · p. 15

C.3 REWARD CURVES ON PHYSICS AND BIOLOGY

  • On Physics, SIPO reaches 74.6 at 1h versus SDPO’s 66.6.The reward curves show SIPO reaches higher reward earlier and keeps improving.Source: §C.3 · p. 15
  • On Biology, SDPO leads at 1h, 50.6 vs. 46.8, but SIPO leads at 5h, 58.7 vs. 56.8.The reward curves show SIPO reaches higher reward earlier overall and keeps improving.Source: §C.3 · p. 15

C.4 COMPARISON WITH THE ANALYTIC KL VARIANT

The analytic KL variant performs comparably or better on shorter tasks but collapses on long math reasoning; the contrastive formulation remains stable across short and long tasks. On code generation, the analytic KL variant performs slightly better, while its training dynamics show lower teacher agreement and maintained entropy.

C.4 COMPARISON WITH THE ANALYTIC KL VARIANT

  • On science QA and tool use, analytic KL scores 71.2 vs. 70.4 with Qwen3-8B and 69.7 vs. 66.5 with Olmo3-7B, but collapses during long math reasoning.The authors attribute the collapse to an overconfident self-teacher imposing excessive penalties over long trajectories; the contrastive formulation remains stable on short and long tasks.Source: §C.4 · p. 15, Table 7 · p. 16
  • On Physics and Biology, SIPO reaches higher reward earlier in training and maintains a more stable upward trajectory than SDPO.The comparison is shown for Qwen3-8B.Source: Figure 7 · p. 16
  • On code generation, analytic KL scores 58.5 vs. 56.3 on LCBv6 and 43.7 vs. 42.7 on the holdout average.The authors note that generated programs are compact compared with long mathematical derivations and that execution feedback indicates where a program fails.Source: §C.4 · p. 16
  • On Physics and Biology, the analytic KL variant has lower top-k agreement with its self-teacher than SDPO and higher entropy than GRPO.Its entropy is higher than SDPO on Physics and comparable on Biology.Source: §C.4 · p. 16

C.5 SELF-TEACHER DESIGN OF THE ANALYTIC KL VARIANT

Ablations of the analytic KL variant show that its self-teacher needs informative successful context and must remain separate from the actor. Qualitative examples illustrate how the earlier self-teacher uses correct solutions across science subjects and analyzes an incorrect attempt on Materials.

C.5 SELF-TEACHER DESIGN OF THE ANALYTIC KL VARIANT

  • Replacing a successful sibling rollout with an incorrect one in SIPO-M leads to a lower reward plateau and stagnating entropy.The authors explain that a correct rollout provides a viable reasoning path, whereas a mistake alone offers limited guidance; in the final method, mistakes are contrasted with a reference answer rather than used alone.Source: §C.5 · p. 16, Figure 9 · p. 18
  • Using the actor’s current weights as the teacher in SIPO-OP causes reward to stay at initialization and entropy to collapse toward zero.With shared weights, the divergence has a trivial minimum at the actor’s own distribution and supplies no useful gradient; the final method instead keeps a separate teacher synchronized every 10 training steps.Source: §C.5 · pp. 16–17, Figure 9 · p. 18

D QUALITATIVE EXAMPLES

  • The earlier self-teacher examples show it using correct solutions in Chemistry, Physics, and Biology, and analyzing an incorrect reference attempt to deduce the answer on Materials.The final method uses a similar prompt structure but places privileged information before the question and re-scores the student response instead of generating a new one.Source: D QUALITATIVE EXAMPLES · p. 17, Figure 10 · p. 19, Figure 11 · p. 20

Related papers

Loading 2609.36742v1…