PaperlayerView exact revision on arXiv

Source-linked AI summary

TISD: On-Policy Self-Distillation with Trajectory Intervention

Taeckyung Lee, Rinat Amankos, Jeonghye Kim, Hyungjun Yoon, Woogyeol Jin, Sung-Ju Lee

arXiv:2609.30878v1cs.AIcs.LG

TL;DR

OPSD evaluates teacher targets only on student-sampled rollouts, leaving alternative-branch successor contexts unsupervised unless the student visits them. TISD forces a teacher-selected branch action, returns suffix generation to the student, and distills the resulting trajectory. It improves average performance over SDPO across coding and science under the reported step- and time-based comparisons.

  • Problem

    OPSD can target a teacher-favored alternative action at a visited prefix but cannot supervise its successor contexts unless the student samples that branch, creating a training-time data-collection bottleneck.

    Source: §1 · p. 1

  • Method

    TISD forces the teacher-preferred token at peak disagreement, lets the student regenerate the suffix, and distills teacher predictions along the resulting trajectory.

    Source: §1 · p. 2

  • Results

    TISD improves Avg@4 over SDPO by 1.2 percentage points across three coding models, and Avg@128 by 0.8 points across four science domains at equal steps and 0.3 points at equal time.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Results across coding and science support teacher-guided branching as a way to expose additional contexts for privileged supervision.

    Source: §6 · p. 9

    TISD depends on the student’s ability to continue from a teacher-selected action and on the quality of privileged teacher targets; branch selection and suffix regeneration add training-time computation.

    Source: §6 · p. 9

Abstract

from arXiv · show

On-policy self-distillation (OPSD) provides dense teacher targets, but evaluates them only along student-sampled rollouts. When the privileged teacher favors an alternative action at a visited prefix, OPSD can provide a target for the branch decision but cannot supervise the successor contexts induced by that action unless the student samples it. This creates a training-time data-collection bottleneck and suggests a different role for teacher-student disagreement: proposing a trajectory branch rather than identifying a sufficient local repair. Our diagnostic framework using controlled token interventions reveals that a teacher-preferred token at peak disagreement can improve student continuation success, while its local corrective value is limited. Motivated by this finding, we introduce a simple branch-regenerate-distill algorithm, Trajectory-Intervention Self-Distillation (TISD). TISD forces a teacher-selected branch action, returns suffix generation to the student, and distills the full trajectory under the privileged-context-conditioned teacher. Across the coding models, TISD improves average Avg@4 over SDPO by 1.2 percentage points. Across the science domains, it improves average Avg@128 by 0.8 points under an equal-step budget and by 0.3 points under an equal-time budget. These results support teacher-guided branching as a way to expose useful successor contexts for self-distillation.

1 INTRODUCTION

OPSD can teach the student to prefer a teacher-favored action but does not supervise the alternative branch’s successor contexts unless the student visits them. TISD forces that action, lets the student regenerate the suffix, and distills the resulting trajectory; reported averages exceed SDPO across coding and science.

Problem

  • Standard OPSD evaluates teacher targets along student-sampled trajectories, so it supervises alternative-branch successor contexts only when the student visits them.Teacher-regenerated trajectories offer another route, but TRD showed late-training accuracy collapse as responses reached the generation limit in SciKnowEval L3 biology.Source: §1 · p. 1

Motivation

  • On failed LiveCodeBench rollouts, forcing the teacher-preferred token at peak disagreement raised Qwen3-8B continuation success to 9.9%, versus 1.7% when the original token was retained.The student regenerated the suffix in both conditions.Source: §1 · p. 2

Method

  • TISD forces the teacher-preferred token at peak divergence, lets the student regenerate the suffix, and distills teacher predictions along the resulting trajectory.This supervises the branch decision and subsequent predictions at newly visited prefixes.Source: §1 · p. 2

Results

  • TISD improves Avg@4 over SDPO by 1.2 percentage points across three coding models, and Avg@128 by 0.8 points across four science domains at equal steps and 0.3 points at equal time.The results support teacher-guided branching both with and without rich execution feedback.Source: §1 · p. 2

2 RELATED WORK

Related work studies on-policy distillation, selective token guidance, and teacher-revised trajectories. TISD instead uses a teacher-selected branch action before returning suffix generation to the student.

On-policy (self-)distillation

  • OPSD derives dense targets from student rollouts using a teacher conditioned on privileged information; SDPO and the Self-Distilled Reasoner use successful rollouts or environment feedback, while RLTF formalizes learning from text feedback (H¨ubotter et al., 2026; Zhao et al., 2026; Song et al., 2026).Source: §2 · p. 2
  • Related OPD methods guide student trajectories through teacher argmax targets, near-future trajectory information, or teacher rewriting, whereas TISD changes the trajectory through one branch action and then returns generation to the student (Liu et al., 2026; Jiang & Ferraro, 2026; Jiang et al., 2026).The student regenerates the suffix without privileged context, while the privileged-context teacher supplies targets along that trajectory.Source: §2 · p. 3

Credit assignment and selective token signals

3 TEACHER-STUDENT DIVERGENCE AS A TRAJECTORY STEERING SIGNAL

The diagnostics examine where teacher–student disagreement concentrates, whether peak disagreement yields a local repair, and whether a teacher-selected action redirects student generation. They find that branching can expose new student prefixes for privileged-context supervision even when token substitution alone rarely repairs a failed response.

3 TEACHER-STUDENT DIVERGENCE AS A TRAJECTORY STEERING SIGNAL

  • The section investigates disagreement concentration, local correction, and whether taking the teacher-preferred action exposes a better student continuation.Source: §3 · p. 3

3.1 WHERE IS TEACHER–STUDENT DISAGREEMENT CONCENTRATED?

  • The peak-divergence position carries 9–20% of total response KL across model scales, a 40–80× concentration relative to mean KL divergence.The diagnostics use reverse KL on Qwen3 rollouts in the SDPO setting on LiveCodeBench v6.Source: §3.1 · pp. 3–4

3.2 PEAK DIVERGENCE RARELY PROVIDES A LOCAL REPAIR

  • Teacher-preferred token substitution repairs only 0.0–1.7% of failures across model scales, while allowing the privileged teacher to generate the suffix raises success to 7.9%, 10.5%, and 13.3% at 1.7B, 4B, and 8B.At larger model scales, peak divergence often occurs in non-code text rather than at an explicit algorithmic or control-flow decision.Source: §3.2 · p. 4

3.3 A TEACHER-SELECTED ACTION CAN REDIRECT THE STUDENT

  • At Qwen3-8B, forcing the teacher token at peak disagreement produced 9.9% successful student continuations versus 1.7% with the original token forced, with suffix regeneration in both conditions.Of failed rollouts solved by teacher continuation at 8B, 17 of 24 were also solved by the intervened student continuation (70.8%).Source: §3.3 · pp. 4–5

3.4 IMPLICATIONS FOR TRAJECTORY-LEVEL LEARNING

  • Teacher-guided branching changes the student’s successor prefixes, creating contexts absent from the original rollout that can receive new privileged-context teacher supervision.The diagnostics frame peak divergence as a candidate branch point even when its preferred token does not repair the original response.Source: §3.4 · p. 5

4 TISD: SELF-DISTILLATION WITH TRAJECTORY INTERVENTION

TISD branches from failed student rollouts using the teacher’s preferred action at peak divergence, then returns suffix generation to the student and distills the resulting trajectory. This exposes teacher supervision at successor contexts while keeping continuations student-generated.

4.1 TISD ALGORITHM

  • At peak divergence, TISD forces the teacher’s preferred action, lets the student regenerate the suffix, and distills teacher predictions along the retained trajectory.Successful responses remain unchanged; eligible failed responses are regenerated when the required privileged context is available.Source: §4 · p. 5, §4.1 · pp. 5–6

4.2 WHY RETURN GENERATION TO THE STUDENT?

  • Returning suffix generation to the student keeps continuations aligned with student behavior, unlike teacher regeneration that can produce responses unlikely under the deployment student.The cited TRD run on SciKnowEval L3 biology falls from 64.8% to 0.0% validation Avg@128 as response lengths approach the generation limit.Source: §4.2 · p. 6

4.3 TISD CREATES NEW OPPORTUNITIES FOR TEACHER SUPERVISION

  • TISD supplies teacher supervision at branch-induced successor prefixes within the same collection iteration, including contexts absent from the original trajectory.On Qwen3-8B LiveCodeBench v6, regenerated suffixes have 2.37× the mean token reverse KL of their paired original suffixes.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

Across coding and science evaluations, TISD outperforms SDPO on reported averages, with additional experiments examining training-data scale, branch selection, and regeneration depth. The experiments compare methods that distill along original student, teacher-regenerated, or teacher-initiated student-regenerated trajectories.

5.1 LEARNING WITH RICH ENVIRONMENT FEEDBACK

  • Across three coding models, TISD averages 50.4% Avg@4 and 51.4% Pass@4, versus SDPO’s 49.2% Avg@4 and 50.3% Pass@4.Evaluation uses rich environment feedback on LiveCodeBench v6 and selects the checkpoint with the highest Avg@4.Source: §5.1 · p. 7

5.2 LEARNING WITHOUT RICH ENVIRONMENT FEEDBACK

  • Across four science domains, TISD improves average Avg@128 over SDPO by 0.8 points under equal steps and 0.3 points under equal time.The comparisons use 200 training steps and a 10-hour budget without rich environment feedback.Source: §5.2 · pp. 7–8

5.3 ADDITIONAL RESULTS

  • On LCB-Large, TISD reaches 57.3 Avg@4 versus GRPO’s 54.3; on LCB-Small, it reaches 66.5 versus SDPO’s 65.5.TISD reaches the best baseline performance after 33% fewer updates on LCB-Large and 56% fewer updates on LCB-Small.Source: §5.3 · p. 8, Figure 6 · p. 8
  • Peak-divergence branching reaches 66.5 Avg@4, versus 65.2 for random-position and 64.6 for first-disagreement branching; retaining the student’s original token reaches 64.8.Only peak-divergence branching improves over SDPO at 65.5 Avg@4 among the evaluated selectors.Source: §5.3 · pp. 8–9
  • At initialization, student-generated continuations succeed more often after peak-divergence than first-disagreement intervention (9.9% vs. 6.1%), while teacher-generated continuations favor first disagreement (17.1% vs. 13.3%).The diagnostic is on failed Qwen3-8B LiveCodeBench v6 rollouts.Source: §5.3 · p. 9
  • Single-round regeneration reaches 66.5 Avg@4 and 92.4 Pass@4, outperforming two rounds (65.2, 90.8) and three rounds (65.5, 90.1).Additional rounds use the original privileged context and add no verifier calls between rounds.Source: §5.3 · p. 9

6 DISCUSSION AND CONCLUSION

The paper treats trajectory collection as a design dimension of privileged-context self-distillation and presents teacher-guided branching as a way to expose additional contexts for supervision. Its analysis distinguishes trajectory steering from local correction, while TISD’s effectiveness depends on student continuation ability and teacher-target quality.

6 DISCUSSION AND CONCLUSION

  • Teacher-guided branching initiates an alternative student continuation and distills the resulting trajectory, exposing additional contexts for privileged supervision.The discussion frames the trajectories on which targets are evaluated as a design dimension alongside the targets themselves.Source: §6 · p. 9
  • A teacher-preferred token can steer subsequent student generation even when substituting it alone does not repair the response.The paper suggests evaluating interventions by the new contexts they expose for teacher supervision, not only by immediate success.Source: §6 · p. 9
  • TISD depends on the student’s ability to continue from a teacher-selected action and on the quality of privileged teacher targets, while adding training-time computation.The paper identifies more efficient branch selection or generation as a direction for reducing this overhead.Source: §6 · p. 9

AI USE STATEMENT

The authors disclose using generative AI tools for writing polish, LLM-judge analysis in Section 3, and method implementation, and state that they reviewed the AI-assisted work.

AI USE STATEMENT

  • Generative AI tools were used to polish writing, perform LLM-judge analysis in Section 3, and implement methods; the authors state they reviewed all AI-assisted work.They report no other use and take responsibility for the final content, claims, and artifacts.Source: AI USE STATEMENT · p. 10

A IMPLEMENTATION OF TISD

The appendix describes the TISD procedure and diagnostic protocol, then reports analyses of KL concentration, semantic roles, and branch-position selection. The diagnostics use failed rollouts to examine teacher guidance and student-regenerated continuations.

A IMPLEMENTATION OF TISD

  • Figure 9 summarizes TISD’s branch–regenerate–distill procedure, with records containing prompts, responses, binary rewards, and privileged contexts.Peak divergence selects the eligible position with the largest teacher–student divergence.Source: Figure 9 · p. 13, A IMPLEMENTATION OF TISD · p. 13

B PRELIMINARY ANALYSIS: FULL DETAILS

  • The diagnostic analysis uses failed Qwen3-{1.7B, 4B, 8B} SDPO rollouts on LiveCodeBench v6 to examine teacher guidance beyond the original student trajectory.A typical training step collects 256 rollouts; the appendix also extends intervention analysis to other architectures and science domains.Source: B PRELIMINARY ANALYSIS: FULL DETAILS · p. 13

B.1 ANALYSIS PROTOCOL

  • The protocol classifies student and teacher token roles near selected positions and compares teacher-forced and original-token student regenerations using execution-based correctness.For SciKnowEval L3, correctness requires an exact match to the reference answer option; pooled science rates cover 553 failed rollouts.Source: §B.1 · pp. 13–14

B.2 KL CONCENTRATION CDF

  • At Qwen3-1.7B, peak divergence carries a median 19.5% of total response KL; across scales, concentration falls to approximately 9% at 4B and 8B.Even at approximately 9%, the peak position carries roughly 40× the per-position average KL.Source: §B.2 · p. 14, Figure 10 · p. 14

B.3 SEMANTIC ROLE BREAKDOWN

  • At larger model scales, peak-divergence tokens are predominantly non-code text, and student–teacher disagreement usually stays within the same broad semantic role.The dominant diagonal in the transition matrix indicates within-category differences, such as alternative variable names or comments.Source: §B.3 · p. 14, Table 3 · p. 14, Figure 11 · p. 15

B.4 EFFECT OF BRANCH-SELECTION POSITION

  • 9.9% of student continuations succeed after forcing the teacher token at peak divergence, versus 6.1% at first divergence and 1.7% at random positions.Teacher continuation instead reaches 17.1% at first divergence and 13.3% at peak divergence, supporting peak divergence for TISD’s student-regenerated suffixes.Source: §B.4 · pp. 14–15, Table 4 · p. 15

B.5 PERSISTENCE DURING TRAINING

The appendix reports that non-code text remains the largest semantic category at peak divergence during Qwen3-1.7B training and compares validation accuracy when failed rollouts are included or excluded from supervision.

B.5 PERSISTENCE DURING TRAINING

  • Across Qwen3-1.7B SDPO training steps 0–80, rollout pass rate rises from 0.211 to 0.434 while the algorithmic share of peak-divergence positions among failures falls from 36.1% to 26.2%.Non-code text remains the largest semantic category at peak divergence throughout.Source: §B.5 · p. 15, Table 5 · p. 16

B.6 CONTRIBUTION OF FAILED-ROLLOUT SUPERVISION

  • At step 80, validation accuracy is 0.480 when successful and failed rollouts are supervised, versus 0.372 with successful rollouts only.The fixed-step comparison motivates examining teacher guidance on failed rollouts as training examples under standard SDPO.Source: §B.6 · p. 15

B.7 CROSS-ARCHITECTURE VALIDATION

These appendix analyses test teacher-guided student continuation across architectures and task domains, then formalize how regenerated contexts can add constraints beyond original-trajectory targets.

B.7 CROSS-ARCHITECTURE VALIDATION

  • On OLMo3-7B-Instruct, teacher-token intervention yields 10.1% student continuation success versus 6.2% with the original token, reproducing the Qwen3 trajectory-steering pattern.The diagnostic used 221 non-truncated LiveCodeBench v6 rollouts; 81.4% of failed-rollout peak-divergence positions were non-code text.Source: §B.7 · p. 16, Table 6 · p. 16

B.8 CROSS-DOMAIN VALIDATION

  • On SciKnowEval L3, student continuation from the teacher token succeeds on 39.4% of failures versus 24.4% for the original-token control; on LiveCodeBench, the rates are 9.9% and 1.7%.The science improvement appears in all four domains; teacher continuation itself succeeds at 93.5% on science failures versus 13.3% on coding failures.Source: §B.8 · p. 16, Table 7 · p. 17

B.10 CONTINUATION OUTCOMES BY SEMANTIC ROLE

  • Peak-divergence semantic role is not a reliable branch-quality indicator: non-code positions fare better at 4B/8B, but the ordering reverses at 1.7B; students reproduce 17 of 24 teacher successes (70.8%).The role-stratified comparisons have small role-specific samples; the paired overlap is measured on the same 181 failed Qwen3-8B rollouts.Source: §B.10 · pp. 17–18, Table 9 · p. 17, Table 10 · p. 18

B.11 QUALITATIVE EXAMPLE OF TRAJECTORY REDIRECTION

  • In a failed LiveCodeBench response, forcing the teacher-preferred naming token redirects the student to a different algorithm that avoids the original bug.The example changes the iteration from (i, k) to (i, j), producing k = 2j − i.Source: §B.11 · p. 18, Figure 12 · p. 18

B.12 REPETITIVE META-COMMENTARY IN A TRD VALIDATION RESPONSE

  • A TRD validation response repeatedly invokes an absent “reference solution” and selects option D without explaining the task-specific basis for that choice.The example illustrates reference-oriented rewriting that does not provide task-specific reasoning.Source: §B.12 · p. 18, Figure 13 · p. 19

C WHY REGENERATED CONTEXTS CAN ADD SUPERVISION

  • The shared-parameter analysis asks whether fitting original-context teacher targets determines later predictions; TISD adds teacher targets at prefixes created by student regeneration.The tractable model uses a binary next-token logit with history-specific features and a shared parameter update.Source: C WHY REGENERATED CONTEXTS CAN ADD… · p. 19, §C.1 · pp. 19–20

C.2 PREDICTION AMBIGUITY AFTER EXACT TEACHER MATCHING

  • Exact matching on collected histories can leave predictions at a new history unresolved when its features depend on parameter directions those targets do not constrain.The remaining ambiguity is zero if the new history lies in the collected row space or no correction freedom remains.Source: §C.2 · pp. 20–21

C.3 EMPIRICAL EVIDENCE FOR REGENERATED TRAJECTORY SUPERVISION

TISD’s regenerated-trajectory training outperforms SDPO in the reported coding and science comparisons, while the analysis explains how new contexts can supply complementary constraints.

C.3 EMPIRICAL EVIDENCE FOR REGENERATED TRAJECTORY SUPERVISION

  • Mean coding Avg@4 rises from 49.2% with SDPO to 50.4% with TISD across three models, and mean Pass@4 rises from 50.3% to 51.4%.Table 2 also reports higher average performance for TISD under both equal-step and equal-time budgets.Source: §C.3 · p. 21
  • Regenerated-context targets can distinguish student parameter corrections that original-context targets leave indistinguishable, but TISD’s net effect also depends on constraints removed with replaced suffixes.The analysis reports 2.37× larger mean token reverse KL on regenerated Qwen3-8B suffixes; isolating the suffix-supervision contribution requires a separate loss ablation.Source: §C.3 · p. 21
  • The analysis and experiments support teacher-guided regeneration within the studied settings, with its contribution depending on collected histories, shared representation, and teacher-target quality.The empirical comparisons test the complete branch–regenerate–distill procedure.Source: §C.3 · p. 21

D EXPERIMENT DETAILS

The experiments specify separate training configurations for coding with rich feedback and science without it, and report TISD’s per-step cost on coding.

D.1 LEARNING WITH RICH ENVIRONMENT FEEDBACK

  • Coding experiments train Qwen3-8B, Qwen3-14B, and OLMo3-7B-Instruct for 80 updates, using successful sibling rollouts when available and otherwise execution-derived feedback as privileged context.TRD uses reverse KL because it performed better than forward KL in the LiveCodeBench experiments.Source: §D.1 · p. 21, Table 11 · p. 22

D.2 LEARNING WITHOUT RICH ENVIRONMENT FEEDBACK

  • Science experiments train Qwen3-8B separately on four SciKnowEval L3 domains, mainly following SDPO without rich environment feedback while searching over target and branch-selection divergences.The method-specific divergence settings are reported in Table 12.Source: §D.2 · p. 22, Table 12 · p. 23

D.3 COMPUTATIONAL EFFICIENCY

  • TISD averages 16.60 minutes per LiveCodeBench v6 training step, versus 15.05 for SDPO, 16.69 for GRPO, and 16.36 for TRD.The reported TISD step adds 1.55 minutes (10.3%) over SDPO, reflecting teacher scoring and student suffix regeneration.Source: §D.3 · p. 22, Table 13 · p. 23
Loading 2609.30878v1…