On-Policy Self-Distillation with Teacher Evolution

1 Introduction

• Sequence-level RL methods assign the same reward to every response token, overlooking differences in token significance.The paper presents this as a limitation that token-level OPSD seeks to address.Source: §1 · p. 1
• OPSD supplies dense token-level supervision from a self-teacher prompted with self-generated privileged context, usually a correct solution.It computes the self-teacher’s per-token logits on student rollouts while preserving the on-policy property.Source: §1 · pp. 1–2
• OPSD can fail to improve continuously because its self-teacher may give unreliable guidance, while training optimizes the model only as a student.The student is guided indiscriminately across samples, allowing misleading signals to enter optimization.Source: §1 · p. 2
• TESD uses reward-informed sample-wise weights to scale student losses and suppress updates from samples with unreliable teacher guidance.This is intended to mitigate the negative impact of unreliable guidance on student optimization.Source: §1 · p. 2
• TESD consistently outperforms representative OPSD and sequence-level RLVR methods in experiments.The authors evaluate pre-trained LLMs of different parameter sizes across multiple downstream tasks.Source: §1 · p. 2
• TESD lets the self-teacher generate rollouts and receive rewards for task correctness and context independence.These rewards are used for teacher-oriented optimization and adaptive student-oriented optimization.Source: §1 · p. 2

2.1 RL with Verifiable Rewards

• RLVR methods including GRPO use sequence-level rewards to optimize the model as a student prompted solely by input queries.Related work modifies GRPO’s objective, normalization, and clipping while retaining sequence-level rewards.Source: §2.1 · p. 2
• TESD uses task-correctness and context-independence rewards to optimize the self-teacher, enhancing token-level OPSD supervision with sequence-level feedback.The self-teacher receives both the input query and privileged context.Source: §2.1 · p. 3

2.2 On-Policy Self-Distillation

• Unlike OPSD approaches relying solely on student-generated rollouts, TESD rewards self-teacher rollouts and adapts sample-wise supervision intensity.The paper positions this approach as a way to achieve more consistent performance gains through enhanced supervision quality.Source: §2.2 · p. 3

3.1 Preliminaries: On-Policy Self-Distillation

• OPSD computes per-token distribution gaps between the self-teacher and student along student-sampled rollouts, providing dense, on-policy supervision.The divergence can be KL or JS, and the self-teacher is usually updated as an exponential moving average of the student.Source: §3.1 · p. 3

3.2 Teaching Ability Investigation

• The self-teacher gives low confidence to some correct rollouts and high confidence to many incorrect ones, with higher confidence tending to favor rollouts resembling privileged context.Teaching ability also trends downward and is unstable during training, while confidence-resemblance bias leaves many correct, context-dissimilar rollouts with low confidence.Source: §3.2 · p. 4

3.3 Teacher-Oriented Optimization

• Teacher-oriented optimization trains the self-teacher on its own rollouts using task correctness and normalized edit distance from privileged context as rewards.The edit-distance reward encourages distinct rollouts, while task correctness encourages guidance toward correct solutions; this optimization mitigates declining teaching ability.Source: §3.3 · pp. 4–5

3.4 Adaptive Student-Oriented Optimization

• TESD estimates guidance quality from the mean reward of teacher rollouts for each query and adaptively weights student losses, while jointly optimizing teacher and student.Batch-softmax weights are scaled by batch size to preserve the overall loss scale; the joint formulation is intended to mitigate unreliable guidance while improving the self-teacher.Source: §3.4 · p. 5

4.1 Experimental Setup

• Experiments use Qwen3-1.7B/4B/8B and Olmo3-7B-Instruct across science reasoning, tool-use, coding, and holdout tasks.All methods use 16 generated rollouts; TESD allocates 8 each to teacher and student, while other methods use all 16 for student rollouts.Source: §4.1 · pp. 5–6

4.2 Performance on Downstream Tasks

• TESD outperforms SDPO and GRPO across base models in both feedback settings and achieves a better performance–forgetting tradeoff on holdout tasks.With rich feedback, SDPO can outperform GRPO, while TESD remains ahead across base models.Source: Table 1 · p. 6, Table 2 · p. 6, §4.2 · p. 6

4.3 Ablation Studies

• 70.61% vs. 67.06% for student sequence-level optimization over OPSD, while teacher sequence-loss optimization reaches 69.35% vs. 67.06%.Correctness-only teacher training can encourage copying the privileged context; adding an edit-distance reward mitigates this behavior and improves guidance of diverse rollouts.Source: Table 4 · p. 7, §4.3 · p. 7
• 75.49% vs. 70.56% after scaling and clipping, while mean-reward weighting alone yields 69.53% vs. 73.76%.Softmax normalization reaches 70.56% vs. 73.76%; scaling and clipping help maintain the overall loss scale.Source: Table 5 · p. 7, §4.3 · p. 7

4.4 Analysis

• During training, TESD’s teaching ability improves with smaller local variance than SDPO, while its adaptive weights shift toward 1.0 as teacher reward rises.TESD also has higher mid- and late-training distillation loss than SDPO and a sparser token-credit pattern focused on query-relevant keywords.Source: §4.4 · pp. 7–8, Figure 4 · p. 8, Figure 5 · p. 8

• TESD combines correctness and independence rewards to optimize the self-teacher and adaptively suppress student updates on queries with potentially unreliable guidance.The paper reports stronger performance than representative OPSD methods and proposes investigating optimization geometry and multimodal reasoning in future work.Source: §5 · p. 8

• All methods use the verl framework with FSDP on one 8×H20 node; training uses 32 prompts per step, with 30 epochs for science tasks, 15 for ToolUse, and 40 steps for LCBv6.Inputs and responses are capped at 2048 and 8192 tokens, respectively.Source: §A.1.1 · p. 12
• SDPO uses an EMA self-teacher, a generalized Jensen–Shannon divergence distillation loss approximated with top-100 logits, and successful student rollouts as privileged context.The implementation enables self-success exclusion and removes thinking traces from the demonstration.Source: §A.1.1 · p. 12
• TESD samples 8 student and 8 teacher rollouts, uses an EMA teacher update rate of 0.01, and clips adaptive weights to [0.05, 5.0].Its teacher reward is task correctness minus λedit times normalized token-level edit distance, with λedit set to 1.0.Source: §A.1.1 · p. 12

A.1.2 Evaluation Strategies

• Evaluations use 16 sampled responses per test problem and report mean@16 accuracy, with the maximum value across training used to compare convergence upper bounds.In-distribution tests run every 10 training steps; sampling uses temperature 0.6 and top-p 0.95.Source: §A.1.2 · pp. 12–13
• Task correctness is binary and task-specific: SciKnowEval checks the answer letter, ToolUse checks the predicted tool call, and LCBv6 checks whether code passes hidden tests.For LCBv6, the hidden tests must pass within the time limit in a sandbox.Source: §A.1.2 · p. 13
• The eight hold-out tasks use task-designated metrics, including strict instruction-following accuracy, exact-match accuracy, and pass@1, under specified decoding settings.Most tasks use greedy decoding; MATH500 and GSM8K use temperature 0.6 and top-p 1.0.Source: §A.1.2 · p. 13

A.1.3 Prompt Templates

• Student rollout and evaluation use the same dataset prompts, and Qwen prompt construction disables the thinkingmode chat-template option.Concrete ScienceQA and ToolUse templates are referenced in Figures 7, 8, and 9.Source: §A.1.3 · p. 13
• The appendix identifies case studies of different methods on SciKnowEval’s Physics subset using Qwen3-8B.Source: Figure 11 · p. 14
• The train/test sizes are tabulated for six tasks.Source: Table 7 · p. 14

A.1.4 Normalized Edit Distance

• Normalized edit similarity uses matching token blocks from SequenceMatcher; identical sequences score Sedit = 1 and Dedit = 0, while none matching score Sedit = 0 and Dedit = 1.The teacher rollout and privileged-context solution are lowercased and whitespace-tokenized after special-token removal.Source: §A.1.4 · p. 14

A.2.1 Scaling of Rollouts

• TESD improves consistently as rollout count grows, while SDPO drops from 55.25% to 55.12% when N increases from 32 to 48.The comparison uses Biology-subset SciKnowEval with Qwen3-4B over 10 training epochs.Source: Table 8 · p. 14, §A.2.1 · p. 15

A.2.2 Qualitative Results

• TESD’s case-study responses are concise and substance-driven, unlike GRPO’s redundant long reasoning and SDPO’s shorter but shallow answers.The SDPO responses include excessive self-affirmations, while GRPO responses contain repetitive “wait” tokens.Source: §A.2.2 · p. 15

A.2.3 Effects of λedit

• λedit = 1.0 gives the highest average across three subsets, although the subset-specific optima differ, so the authors fix it for main experiments.Biology’s optimum is 1.0 and materials’ is 0.7.Source: §A.2.3 · p. 15, Table 9 · p. 15

A.2.4 Training Efficiency

• On LCBv6, TESD takes increasingly less time than GRPO and SDPO as training progresses.The authors attribute this to more structured, efficient-to-execute programs and reduced code-execution and sandbox-evaluation time.Source: §A.2.4 · p. 15, Figure 12 · p. 15

A.3.1 Relation with Methods improving Privileged Context; A.3.2 Relation with Methods integrating GRPO and SDPO

• TESD targets teacher capability rather than only context quality, and differs from GRPO–OPSD blends by optimizing teacher and student roles separately.Related methods focus on increasing or strengthening privileged context; TESD uses GRPO rewards for the self-teacher and adaptive OPSD weighting for the student.Source: §A.3.1 · p. 15, §A.3.2 · p. 16

A.4 Ethical Considerations

• The paper notes that stronger reasoning may be misused and emphasizes robust safety alignment when deploying such models.The experiments use open-source datasets according to their licenses and intended purposes.Source: §A.4 · p. 16

A.4.1 Limitations

• TESD adds hyperparameters such as λedit, and EMA teacher updates smooth capability improvement, limiting how quickly the teacher evolves.The EMA mechanism is retained from the current OPSD paradigm to ensure training stability.Source: §A.4.1 · p. 16