Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning

1 INTRODUCTION

• Standard OPD fragments corrections into one-token guidance rather than continuous, correct segments (Jiang et al., 2026).Recent DAgger-style work seeks SFT-like long-horizon supervision from student states (Lauffer et al., 2025). However, DAgger inserts teacher responses into student trajectories, substantially changing what the student would otherwise produce.Source: §1 · p. 1
• SOPD combines OPD’s on-policy property with SFT’s long-horizon supervision by generating a teacher correction from each student-step prefix after the student completes its trajectory.Each teacher correction consists of one step, providing step-level supervision over the complete student-generated trajectory.Source: §1 · pp. 1–2
• SOPD reduces to SFT as the number of steps approaches one and approximates forward-KL OPD as step length approaches one token.These limiting cases connect the two distillation paradigms.Source: §1 · p. 2
• 10 percentage points is SOPD’s average improvement over OPD on mathematical reasoning; it also outperforms SFT and OPD on agent tasks with fewer interaction rounds.Steps follow natural reasoning boundaries for mathematics and individual environment-interaction turns for agent tasks.Source: §1 · p. 2
• SOPD needs teacher-generated responses rather than logits, enabling black-box distillation with practical teacher-call costs comparable to standard SFT.Total teacher generation per sample is close to one complete response. For agents, single-step teacher calls avoid additional waiting for teacher–environment interaction and tool execution.Source: §1 · p. 2

Knowledge distillation

• SOPD combines student-induced prefixes with teacher-generated local targets while preserving the complete student trajectory.Sequence-level distillation uses teacher-generated sequences to reduce token-target train–test mismatch (Kim & Rush, 2016). MiniLLM instead minimizes reverse KL and samples from the student during optimization, distinguishing sequence generation from student sampling.Source: §2 · pp. 2–3

On-policy language-model distillation

• OPSD improves reasoning and efficiency without a separate teacher (Zhao et al., 2026), while ExOPD enables multi-expert students to surpass teachers (Yang et al., 2026).Generalized Knowledge Distillation queries teacher logits on student generations. OPSD uses ordinary and privileged contexts (Zhao et al., 2026), whereas ExOPD relaxes direct imitation through reward extrapolation (Yang et al., 2026).Source: §2 · p. 2
• SOD, StepOPSD, and SAF-OPD combine step-level signals with GRPO or RLVR to address sparse rewards and local credit assignment (Zhong et al., 2026; Zhang et al., 2026; Ding et al., 2026).GAD instead uses a generator–discriminator game for black-box on-policy distillation, outperforming sequence-level distillation while accessing only teacher-generated text (Ye et al., 2025).Source: §2 · p. 2
• Unlike TRD’s complete trajectory rewriting (Jiang et al., 2026), SOPD preserves the student trajectory and generates a coherent local correction within each natural step.TRD identifies why token-wise OPD yields fragmented repairs: later targets remain conditioned on erroneous student prefixes rather than prefixes incorporating earlier teacher corrections (Jiang et al., 2026).Source: §2 · pp. 2–3

Distillation and guidance for multi-turn agents

• Agent methods use expert suffixes or guidance-enhanced reward optimization rather than independent targets at every fixed student state (Lauffer et al., 2025; Da et al., 2025).OEC switches from student control to an expert and applies SFT to reward-filtered expert suffixes (Lauffer et al., 2025). Agent-RLVR augments failed attempts with external-model guidance used only during training, then applies RL with verifiable environment rewards (Da et al., 2025).Source: §2 · p. 3
• Other agent-distillation methods vary student-control intervals, execute curriculum-selected teacher actions, or replay teacher trajectories (Wang et al., 2026; Li et al., 2026; Liao et al., 2026).TCOD expands student control forward or backward through trajectories (Wang et al., 2026), while Guided-OPD anneals executed teacher interventions, producing teacher–student mixture states (Li et al., 2026). ReOPD avoids repeated training-time environment interaction by replaying teacher prefixes, letting students act at selected steps, and requesting dense teacher supervision (Liao et al., 2026).Source: §2 · p. 3
• SOPD provides independent teacher fragments at fixed student-generated steps, preserving more faithfully on-policy supervision than methods that intervene in trajectories.Each fragment conditions on its corresponding student prefix. Teacher generation requires no intervening environment interaction or returned tool results, so queries can run in parallel after the student trajectory finishes.Source: §2 · p. 3

Supervised fine-tuning (SFT)

• SFT teacher-forces complete teacher or expert sequences, providing coherent long-range targets but creating an off-policy state-distribution mismatch.Each target token is conditioned on preceding teacher tokens rather than the current student's trajectory.Source: §3 · p. 3

On-policy distillation (OPD)

• OPD samples student trajectories and queries teacher distributions at every student prefix, aligning training with inference states and supplying dense token-level feedback while requiring teacher logits.The standard implementation requires access to teacher logits.Source: §3 · p. 3
• After student errors, OPD's position-wise targets remain conditioned on erroneous student prefixes and need not form a coherently executable correction path (Jiang et al., 2026).As the erroneous trajectory continues, each successive teacher query receives an even longer erroneous student prefix.Source: §3 · p. 3

4 STEP-LEVEL ON-POLICY DISTILLATION

• SOPD combines student-selected step boundaries with coherent, multi-token teacher corrections, without allowing teacher targets to change subsequent student-visited states.The complete student trajectory is partitioned first; teacher generation unfolds autoregressively only within each local target.Source: §4 · p. 4, §4.4 · p. 6

4.1 STUDENT ROLLOUTS FIX THE VISITED STATES

• Student prefixes determine step-start states, including observations returned after executing student actions in interactive environments.Mathematics uses paragraph boundaries or explicit headings, with a 1,024-token safety cap only when no natural boundary appears. In ALFWorld, each assistant response and its parsed action form a step.Source: §4.1 · p. 4

4.2 TEACHER TARGETS DO NOT ALTER THE TRAJECTORY

• Teacher queries exclude earlier teacher targets, enabling independent, parallel generation using only teacher-generated text rather than logits or gradients.Each query contains the original prompt and preceding student steps, supporting black-box teachers. Mathematics targets stop at natural boundaries; agent queries run after episodes without tool execution or new environment observations.Source: §4.2 · p. 5

4.3 STEP BALANCING PREVENTS LONG TARGETS FROM DOMINATING

• SOPD averages token cross-entropy within each target, then weights collected steps equally, preventing long targets from dominating.Trajectories with more retained steps contribute more terms; stochastic targets estimate length-normalized risk, while greedy targets yield length-normalized KL rather than unnormalized sequence-level forward KL. Packing preserves this objective: targets attend only to their student prefix and own causal tokens, with loss restricted to teacher tokens.Source: §4.3 · p. 5

4.4 SOPD CONNECTS TWO GRANULARITY LIMITS

• Whole-response steps reduce SOPD to the sequence teacher-forcing objective used by SFT.Caching the teacher response yields ordinary teacher-data SFT; generating it online changes data acquisition, not the optimization objective.Source: §4.4 · p. 5
• Single-token steps with sampled teacher targets recover forward-KL OPD on student-visited prefixes in expectation.Teacher entropy is constant with respect to student parameters, making expected cross-entropy equivalent to forward-KL minimization. Greedy targets instead imitate the teacher’s argmax and are not unbiased estimates of full-distribution forward KL.Source: §4.4 · pp. 5–6

4.5 RELATION TO OTHER DISTILLATION PROCEDURES

• Unlike teacher-trajectory distillation (Kim & Rush, 2016) or probability-querying OPD, SOPD generates local teacher targets at student-induced states.Off-policy sequence distillation trains on teacher-generated trajectories; standard OPD minimizes token-level divergence along student outputs.Source: §4.5 · p. 6
• SOPD uses no reward, preserves natural temporal order, queries every student-visited step, and never executes a teacher target.ExOPD uses reward extrapolation (Yang et al., 2026), whereas TCOD assigns control intervals through a temporal curriculum (Wang et al., 2026). Guided-OPD introduces teacher actions into the environment through a curriculum (Li et al., 2026), while ReOPD replays offline teacher prefixes instead of interacting throughout training (Liao et al., 2026).Source: §4.5 · p. 6

5.1 ALFWORLD AGENT INTERACTION

• 84.29% Seen and 82.09% Unseen success exceed Vanilla OPD’s 65.72% and 60.45%, respectively, while reducing mean rounds by 3.53 and 4.33.SOPD achieves the highest success and fewest rounds among student methods on both splits, exceeding TCOD-F2B’s Unseen success by 2.90 points. Following TCOD (Wang et al., 2026), the experiment distills a domain-RL-trained Qwen2.5-7B teacher into a Qwen2.5-3B-Instruct student.Source: §5.1 · pp. 6–7
• Full-split ALFWorld evaluations show Seen and Unseen success improving overall during training while interaction lengths decrease.These checkpoint trajectories indicate that improved task completion accompanies more efficient interaction policies.Source: Figure 2 · p. 7, §5.1 · p. 7
• 10.74% Hard-split success accompanies 28.15 mean rounds, the shortest interaction trajectory among all student methods.The Hard split remains comparatively stable during training, unlike the substantial success gains on Seen and Unseen.Source: §5.1 · p. 7

5.2 MATHEMATICAL REASONING

• 57.7% four-benchmark mean accuracy exceeds OPD’s 47.7%, with SOPD leading on all four mathematical reasoning benchmarks in the same-size teacher–student comparison.SOPD improves over OPD by 9.9 points on AIME24, 10.4 on AIME25, 6.4 on HMMT25-Feb, and 13.1 on HMMT25-Nov. Each benchmark result aggregates 32 decoding samples for each of 30 problems at the final checkpoint.Source: Table 2 · p. 8, §5.2 · p. 8
• 45.2 mean teacher tokens per retained step at the final update accompanies more student reasoning steps with broadly stable step lengths.Mean teacher supervision grows from 29.0 tokens at update 1 to 35.0 at update 8, then reaches 41.9 and 45.2 at the final two updates. Step boundaries use the semantic-step rule and a 1,024-token safety cap.Source: Figure 4 · p. 8, §5.2 · p. 8
• SOPD partitions mathematical responses at natural reasoning boundaries, whereas OPD optimizes sampled-token reverse KL using the same prompt pool and one student response per prompt.The student starts from Qwen3-4B non-thinking, and its teacher is trained from the same base using GRPO (Shao et al., 2024). Following G-OPD (Yang et al., 2026), teacher training uses 57,000 DeepMath (He et al., 2025) examples with difficulty at least 6.Source: §5.2 · pp. 7–8

6 CONCLUSION

• SOPD yields consistent gains across four mathematical benchmarks and the ALFWorld Seen and Unseen splits.It provides coherent, dense local supervision without allowing the teacher to control later states.Source: §6 · p. 9
• SOPD preserves the complete student trajectory, generates one black-box teacher target at every natural step, and optimizes a step-balanced cross-entropy loss.Teacher queries use student-visited intermediate states.Source: §6 · p. 9
• At response-level granularity, SOPD recovers sequence SFT; at token-level granularity, it corresponds to a sampled forward-KL OPD objective.Source: §6 · p. 9

• Section 5 specifies model pairs, training data, rollout protocol, optimization settings, and evaluation suites for the main experiments.Source: REPRODUCIBILITY STATEMENT · p. 9
• Appendix A details per-update batch sizes, learning rates, decoding parameters, response limits, evaluation sample counts, seeds, and ALFWorld interaction limits.Source: REPRODUCIBILITY STATEMENT · p. 9
• The submission source includes figure-generation scripts and aggregated records used to produce the reported tables and figures.Source: REPRODUCIBILITY STATEMENT · p. 9

A.1 MATHEMATICAL REASONING

• Both methods use Qwen3-4B non-thinking as the student and Qwen3-4B-Non-Thinking-RL-Math-Step500 as the teacher.Following G-OPD (Yang et al., 2026), teacher training uses 57,000 DeepMath examples with difficulty at least 6 for 500 optimizer updates. Correct final answers receive reward 1.0; incorrect answers receive 0.0.Source: §A.1 · p. 11
• 10,240 prompt instances and 10,240 student rollouts are processed by each method across ten updates, using identical prompt order and 1,024 prompts per update.Each prompt receives one student rollout, sampled with temperature 1.0 and top-p 1.0. Maximum prompt and response lengths are 2,048 and 32,768 tokens, respectively.Source: §A.1 · p. 11
• 2 × 10^-6 is SOPD’s learning rate, versus 10^-5 for standard OPD; SOPD generates teacher steps greedily, while OPD uses sampled-token reverse KL.Greedy teacher-step generation uses temperature 0.Source: §A.1 · p. 11
• SOPD ends natural steps at blank-line paragraph boundaries or explicit reasoning headings, with a fallback token cap.The fallback cap is 1,024 tokens, and each retained student prefix receives one independently generated teacher step. Loss is averaged first over tokens within each target, then over retained steps.Source: §A.1 · p. 11
• 960 responses per benchmark are evaluated for each method: 32 independent generations for each of 30 problems in AIME24, AIME25, HMMT25-Feb, and HMMT25-Nov.Evaluation uses temperature 1.0, top-p 1.0, and a 32,768-token response cap. Every response is a separate request with seed 42 + 32i + j for problem i and sample j.Source: §A.1 · p. 11
• The evaluator extracts final boxed answers and applies rule-based symbolic verification with Math-Verify.The result extractor checks all numerators and denominators against the durable summaries.Source: §A.1 · p. 11

A.2 ALFWORLD

• The student is Qwen2.5-3B-Instruct, paired with the ‘langfeng01/GiGPO-Qwen2.5-7B-Instruct-ALFWorld’ teacher from the TCOD model pairing.The Explorer samples 16 tasks with one rollout each; the released asynchronous launcher lets Explorer and Trainer counters advance independently.Source: §A.2 · p. 11
• The Trainer consumes 64 experiences per batch with learning rate 10^-6, temperature 1.0, and staleness two.Training uses a 512-token response cap, a 10,240-token prompt cap, and at most 30 environment turns.Source: §A.2 · p. 11
• Full evaluation covers Valid Seen-140, Valid Unseen-134, and Hard-121, reporting task-level success mean and mean environment rounds.Evaluation samples at temperature 0.4 with a 4,096-token response cap.Source: §A.2 · p. 11

B LLM USAGE

• The authors determined the research claims, methods, experiments, and conclusions.Source: B LLM USAGE · p. 12
• Generative AI tools assisted language editing, translation, figure-layout iteration, and LaTeX preparation.Source: B LLM USAGE · p. 12
• The authors manually verified the final manuscript and reported results.Source: B LLM USAGE · p. 12