On the Off-Policy Teacher in On-Policy Distillation

Motivation

• OPD uses student-generated trajectories for training, but the teacher must supervise student prefixes that increasingly diverge from contexts sampled under its own policy.The resulting teacher-side distribution shift may make token-level guidance less reliable and harm OPD training.Source: §1 · p. 1
• Earlier approaches adapt how the student uses fixed-teacher supervision; SCOUT instead asks whether the teacher can be trained to supervise student-generated prefixes better.The paper presents teacher adaptation as a complementary optimization axis to regulating teacher supervision.Source: §1 · p. 2

SCOUT

• SCOUT combines dense OPD updates for the student with outcome-reward reinforcement learning that trains the teacher on student-generated prefixes.The evolving teacher then supervises subsequent student updates.Source: §1 · p. 2

Results

• Across three math settings, SCOUT improves average accuracy over standard OPD by 1.2–2.6 points and raises the code-generation average score from 56.6 to 59.7.The paper also reports that SCOUT is complementary to other OPD approaches and that sparse teacher updates can realize the benefit.Source: §1 · p. 2

Evaluation

• The evaluation spans multiple teacher–student pairs, model scales, and model families in mathematical reasoning, and also tests code generation.A control without student-prefix conditioning does not match SCOUT, supporting adaptation to student-generated states as central to the improvement.Source: §1 · p. 2

On-Policy Distillation

• Off-policy distillation typically trains on teacher-generated responses or a fixed dataset, whereas OPD collects responses from the current student policy.OPD therefore supplies dense token-level supervision on states the student visits.Source: §2 · p. 3
• OPD reduces the mismatch between training and inference-time contexts while retaining dense teacher distributional supervision.Its objective uses a divergence between student and teacher next-token distributions, such as forward or reverse KL or Jensen–Shannon divergence.Source: §2 · p. 3

Group Relative Policy Optimization

• GRPO samples responses from the current policy and uses sequence-level rewards, normalized within each response group, to form a clipped policy-gradient objective.The objective can include KL regularization to a reference policy.Source: §2 · p. 3

3.1 The Off-Policy Teacher in OPD

• The off-policy teacher issue is that a teacher optimized on its own trajectory contexts must supervise student-generated prefixes that may become increasingly unfamiliar as reasoning proceeds.The authors hypothesize that this makes the teacher less effective on longer student-generated prefixes.Source: §3.1 · pp. 3–4
• The continuation experiment truncates student responses to AIME 2025 questions at prefix ratios from 10% to 90%, then samples four teacher continuations per prefix.It uses Qwen3-1.7B in non-thinking mode as student and Qwen3-4B-Instruct-2507 as teacher, with a 16,384-token response limit for both.Source: §3.1 · p. 4
• Teacher continuation accuracy generally declines as student-generated prefixes lengthen, while teacher uncertainty on those prefixes rises relative to its own-prefix contexts.The continuation decline is not explained by insufficient generation length: sampled continuations remain within the response budget.Source: §3.1 · p. 4, Figure 1 · p. 4

3.2 Method

• SCOUT keeps the standard student OPD update and adds teacher-side RL on continuations generated from prefixes sampled from the same student trajectories.The teacher receives verifiable outcome rewards, and gradients apply only to teacher-generated continuation tokens; the updated teacher is synchronized for later OPD updates.Source: §3.2 · pp. 4–5
• SCOUT periodically updates the teacher and linearly increases the student-prefix ratio, progressing from shorter prefixes to more challenging longer ones during training.Periodic updates are intended to keep the teacher effective as the student’s context distribution evolves.Source: §3.2 · p. 5, Figure 2 · p. 5

4.1 Experimental Setup

• Experiments cover three mathematical-reasoning teacher–student pairs and one code pair, compare frozen-teacher OPD, GRPO, and OPD variants, and use three training seeds with repeated benchmark evaluation.The evaluation includes six math benchmarks and three code-generation benchmarks.Source: §4.1 · p. 6

4.2 Main Results

• 51.4 and 51.6 mean accuracy in the Qwen3-4B and Qwen3-8B settings improve over OPD by 2.2 and 2.6 points, respectively.SCOUT is best or tied for best among OPD-based methods across all six benchmarks in both settings.Source: §4.2 · p. 6, Table 2 · p. 7
• SCOUT achieves the strongest aggregate performance among competing OPD methods in all three mathematical-reasoning settings and code generation, across model scales, training configurations, and model families.The results cover both mathematical reasoning and code generation.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · pp. 7–8
• For the same Qwen3-4B-Instruct-2507 → Qwen3-1.7B pair, SCOUT improves over OPD by 2.2 points on mathematical reasoning and 3.1 points on code generation; Relay-OPD remains competitive on math but drops to 12.1 mean accuracy on code.Source: §4.2 · p. 6
• 53.6 mean accuracy improves over OPD by 1.2 points with Skywork-OR1-Math-7B teaching DeepSeek-R1-Distill-Qwen-1.5B, while GRPO collapses and Prune-OPD underperforms standard OPD.SCOUT outperforms competing OPD methods overall in this model-family setting.Source: §4.2 · p. 7, Table 3 · p. 8

• With student prefixes held fixed, the trained teacher achieves higher accuracy at every prefix length, attributing the improvement to teacher adaptation rather than stronger student prefixes.The matched-checkpoint comparison also shows increasing continuation accuracy across longer prefixes as training progresses.Source: §5 · pp. 8–9, Figure 3 · p. 9
• After SCOUT training, teacher entropy on student-generated prefixes falls over later response positions, narrowing its gap with entropy on the teacher’s own responses.The figure inset shows the corresponding behavior before training.Source: Figure 4 · p. 9, §5 · p. 9
• Avg@6 is 51.3, 52.5, and 51.5 for update intervals f = 1, 5, and 10, respectively; performance falls close to standard OPD at f = 20.The authors use f = 10 to preserve most gains with fewer teacher updates.Source: Figure 5 · p. 10, §5 · p. 10
• Across three settings, SCOUT performs strongest; OPD plus Teacher GRPO yields only modest gains or slightly underperforms OPD, so extra teacher training alone does not match SCOUT.The control uses the same outcome-reward GRPO objective but starts teacher rollouts from the original problem rather than student-generated prefixes.Source: §5 · pp. 9–10, Table 4 · p. 10
• With comparable training compute on the same data, OPD quickly plateaus while SCOUT continues improving; extra OPD epochs do not close the gap in this fixed-data setting.The comparison does not rule out further gains from fresh data.Source: Figure 6 · p. 10, §5 · pp. 10–11
• Adding SCOUT to OPTR yields a further 2.5-point gain on math and 1.2-point gain on code, reaching the strongest performance in both settings.OPTR itself improves over standard OPD on both tasks, indicating the teacher adaptation gains remain when using an improved distillation objective.Source: §5 · p. 11

• Related methods also revise trajectories or provide corrective teacher guidance, while Table 5 reports that adding SCOUT improves both OPD and OPTR across math and code.Trajectory interventions include MOTAB [36], Relay-OPD [18], TRD [37], and SOPD [38]; TrOPD also includes off-policy guidance [6].Source: Table 5 · p. 11, §6 · p. 12
• One family controls rollout horizons through early stopping, gradual extension, fixed truncation, or down-weighting and truncating later supervision when models drift apart.These approaches include prefix distillation and ESR [7], POPD and TOPD [29], and Prune-OPD [17].Source: §6 · p. 11
• Other fixed-teacher methods reweight, select, gate, or restrict teacher signals using discrepancy, uncertainty, reliability, or verifier feedback.Examples include IW-OPD [30], SOD [31], TIP, TrOPD [6], LGR [3], RG-OPD, OPDVR, and SPOT [35].Source: §6 · pp. 11–12
• Unlike prior approaches that keep the teacher fixed, SCOUT updates it to better supervise student-generated states, adding a complementary optimization axis for OPD.The paper reports that teacher adaptation can be combined with TrOPD’s loss-level OPTR component.Source: §6 · pp. 11–12

7 Conclusions

• SCOUT addresses teacher-side distribution shift in OPD by adapting the teacher to student-generated prefixes.Source: §7 · p. 12
• Across model pairs and task domains, SCOUT consistently improves over standard OPD and competing OPD methods.The analysis finds better continuation from student-generated prefixes and attributes gains primarily to teacher adaptation rather than simply further teacher training.Source: §7 · p. 12
• The results establish teacher adaptation as a promising, complementary optimization axis for improving on-policy distillation.Source: §7 · p. 12

A Limitations and Future Work

• The evaluation focuses on mathematical reasoning and function-level code generation, leaving longer-horizon agentic tasks for future study; SCOUT also adds wall-clock and memory costs and relies on verifiable rewards and a directly accessible, trainable teacher.The authors identify learned or weaker feedback, more efficient teacher updates, and prompt-based adaptation as future directions.Source: A Limitations and Future Work · p. 18

B.1 Hyperparameter Settings

• Learning rates are selected by grid search: student rates among {1, 3, 5, 7} × 10−6 on standard OPD, then teacher rates among {1, 2, 3, 4, 5} × 10−6 for SCOUT.The general hyperparameter settings are reported in Tables 6 and 7, with best learning rates in Table 8.Source: §B.1 · p. 18
• GRPO uses a learning rate of 3 × 10−6, rollout number n = 8, and reference weight β = 0.001; teacher GRPO uses the same settings except for its learning rate.Source: §B.1 · p. 18

B.2 Loss Implementation

• The single-sample K1 estimator avoids the computational and memory overhead of evaluating the teacher over the full vocabulary at every student-generated token.It provides a sampled approximation to the reverse-KL objective.Source: §B.2 · pp. 18–19
• The student is optimized with a clipped policy-gradient objective using a token-level advantage defined for each student-generated token.The implementation uses the policy-gradient formulation of OPD.Source: §B.2 · p. 18
• Separate tables list hyperparameters for mathematical reasoning, code generation, and learning rates across model pairs and tasks.Source: Table 6 · p. 19, Table 7 · p. 19, Table 8 · p. 19

B.3 Evaluation Details

• Benchmarks use task-specific repeated evaluations, with 32 repeats for AIME 24, AIME 25, AMC, and HMMT, and 16 for HumanEval+.MATH500 and MBPP use 8 repeats; OlympiadBench uses a filtered 581-question text-only English subset, and LiveCodeBench uses stable v5 with 880 questions and 4 repeats.Source: §B.3 · p. 19, Table 9 · p. 21
• Methods are generally trained with three independent runs; individual-run results report evaluation SEM, while aggregate rows report the mean and cross-run SD.One-sided Welch tests compare run-level macro-averages, and orange cells mark higher means with p < 0.1.Source: §B.3 · pp. 19–20

B.4 Reproducibility and Compute

• The reproducibility section reports that experiments run on AWS p4d instances with A100 GPUs, with method-specific node allocations and software versions.Training starts without an SFT warm-up stage; SCOUT places student and teacher training on separate physical nodes.Source: §B.4 · p. 21

C Complete Experiment Results

• The complete-results section provides per-benchmark performance and individual training runs in addition to the aggregate main-text results.Source: C Complete Experiment Results · p. 21

C.1 Results of the Main Experiments

• Main experiments use three independent training runs unless otherwise specified, with individual-run ± values denoting evaluation standard errors and Mean-row ± values denoting across-run standard deviations.Bold and underline mark the best and second-best three-run aggregates among OPD-based methods.Source: §C.1 · p. 21
• Relay-OPD generated excessive syntax errors on code generation, so the authors skipped repeated training for that method.Source: §C.1 · pp. 21–22
• Tables 10–13 cover Qwen3-4B-Instruct-2507 to Qwen3-1.7B, Qwen3-8B-DAPO to Qwen3-1.7B, and Skywork-OR1-Math-7B to DeepSeek-R1-Distill-Qwen-1.5B settings.Tables 10 and 11 cover the first setting; Tables 12 and 13 cover the latter two settings, respectively.Source: §C.1 · pp. 21–22

C.2 Results of the Analysis Experiments

• SCOUT significantly improves over OPD in average math performance in the Qwen3-4B, Qwen3-8B, and Skywork settings, with significant gains on 3 of 6, 5 of 6, and 4 of 6 benchmarks, respectively.Source: Table 10 · p. 23, Table 12 · p. 25, Table 13 · p. 26
• In the Qwen3-4B code setting, SCOUT significantly improves accuracy on 2 of 3 benchmarks and in average performance.Source: Table 11 · p. 24
• The teacher-GRPO control applies the same outcome-reward update as SCOUT but has the teacher generate from scratch rather than from a student-generated prefix.OPD and SCOUT are shown as three-run aggregate references in these math and code comparisons.Source: Table 14 · p. 27, Table 15 · p. 27
• Teacher updates every 10 or fewer steps consistently improve over OPD, while gains become marginal or disappear at intervals of 15 or more.An interval of 5 gives the highest average accuracy; an interval of 10 retains most of the improvement with fewer updates, and the learning rate is reduced for more frequent updates.Source: §C.2 · p. 22, Table 16 · p. 28
• SCOUT further improves OPTR in both the math and code settings, supporting complementarity between teacher adaptation and this loss-level OPD method.Source: Table 17 · p. 28, Table 18 · p. 29