PaperlayerView exact revision on arXiv

Source-linked AI summary

Teaching a Moving Student: Rethinking the Curriculum of On-Policy Distillation

Lingxiang Hu, Tianle Xia, Yiding Sun, Ming Xu, Linfang Shang, Lan Xu, Ning Zheng, Wei Xu, Jie Jiang

arXiv:2609.25048v2cs.CLcs.AI

TL;DR

In OPD, the student’s evolving policy shifts which prefixes receive teacher supervision, and earlier prefixes can become unlikely before teacher–student disagreement is resolved. R-OPD uses a gradient-triggered replay curriculum to revisit initial-student trajectories; across mathematics benchmarks, it improves accuracy over continued current-policy sampling.

  • Problem

    OPD’s supervised state distribution changes with the student, raising how to organize supervision when earlier prefixes become less likely despite persistent teacher–student misalignment.

    Source: §1 · p. 1

  • Method

    R-OPD monitors mean gradient changes relative to minibatch variation and switches from current-policy sampling to initial-policy replay after two consecutive qualifying comparisons.

    Source: §1 · p. 3, §7.1 · p. 7

  • Results

    Across eight mathematics benchmarks, R-OPD improves average accuracy over continued on-policy sampling by 2.25 and 4.44 percentage points at 16K and 32K, respectively, for a 0.6B student.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across three Qwen3 student sizes, R-OPD exceeds current-policy training and fixed R40F20 replay at 16K/32K, while its longer self-generated responses at 32K suggest timely replay helps students express more reasoning capacity.

    Source: §9 · p. 9

    The experiments focus on competition mathematics with Qwen3 students; transfer to other task families and student architectures remains to be established.

    Source: H LIMITATIONS · p. 26

Abstract

from arXiv · show

In on-policy distillation (OPD), the student determines which states receive teacher supervision. As its policy evolves, earlier response prefixes become less likely even though teacher-student disagreement on them persists. Under matched trajectory and optimization budgets, neither more queries nor more frequent rollout resampling is uniformly beneficial. Current-policy rollouts outperform initial-policy rollouts at shorter response budgets, but this ranking reverses at longer budgets. Replaying initial-policy rollouts after current-policy training improves accuracy, whereas replaying fixed recent rollouts does not reproduce the gain. We propose R-OPD, a gradient-triggered curriculum that adaptively selects when to revisit initial-student trajectories. When changes in mean gradients fall within minibatch-level variation for two consecutive comparisons, training switches from the next iteration onward to initial-policy replay. Across eight mathematics benchmarks and three training-data orders, R-OPD improves average accuracy over continued current-policy sampling by 2.25/4.44 percentage points at 16K/32K for a 0.6B student and 2.81/6.15 points for a 1.7B student. With a 30B-A3B teacher, it improves 8B accuracy by 4.05 points at 32K. At 32K, R-OPD exceeds a fixed schedule of 40 current-policy updates followed by 20 replay updates by 1.39/1.66/1.85 points for 0.6B/1.7B/8B. At the same generation cap, R-OPD also produces longer responses, suggesting that well-timed revisits help students use more of their reasoning capacity.

1 INTRODUCTION

In OPD, the student determines which response prefixes receive teacher supervision, so the supervised states shift as its policy evolves. The introduction shows that rollout freshness and query breadth have budget-dependent effects, then motivates R-OPD’s adaptive replay of earlier student trajectories.

1 INTRODUCTION

  • OPD’s state distribution changes with the student, which can move training away from earlier prefixes before their teacher supervision is fully absorbed.The student determines which states receive future supervision, but moving away from a prefix does not mean the student has matched the teacher there.Source: §1 · p. 1
  • With matched trajectory and optimization budgets, increasing query count helps under per-step resampling but hurts when rollouts are generated before training.The introduction also reports that frequent resampling does not consistently improve accuracy.Source: §1 · p. 2
  • Current-policy rollouts yield higher accuracy at shorter response-length budgets, but initial-policy rollouts become more accurate and produce longer responses at 32K.The comparison holds across two teachers.Source: §1 · p. 2
  • Most saved initial-policy prefixes become less likely under the trained student, while its predictions on them move closer to the teacher but remain imperfectly aligned.Earlier states therefore leave the training distribution before their teacher supervision has been fully absorbed.Source: §1 · p. 2
  • Replaying the initial-policy bank after current-policy training further reduces teacher–student disagreement and improves accuracy at both 16K and 32K.Freezing a newly generated bank from the trained student does not reproduce the gain.Source: §1 · p. 2
  • Training first on current-policy rollouts and then replaying the initial-policy bank outperforms the reverse order at 16K and 32K.This temporal asymmetry motivates learning from current-policy states before revisiting initial states.Source: §1 · pp. 2–3
  • R-OPD uses changes in mean gradients relative to minibatch variation to choose when replay begins, switching to the initial-policy bank after two consecutive qualifying comparisons.Replay starts from iteration τ + 1, avoiding a manually chosen transition iteration for each setting.Source: §1 · p. 3
  • Across eight mathematics benchmarks, R-OPD improves average accuracy over continued on-policy sampling by 2.25 and 4.44 percentage points at 16K and 32K, respectively, for a 0.6B student.For 1.7B students, the mean gains are 2.81 and 6.15 points; with a 30B-A3B teacher, R-OPD improves mean 8B accuracy by 4.05 points at 32K.Source: §1 · p. 3

2 SETUP: STATES, SUPERVISION, AND TRAINING

The setup treats each response prefix as a state and distinguishes the source of training states from the current student’s predictions on them. Experiments vary query, rollout, and replay schedules while controlling trajectory and optimization-step budgets within comparisons.

2 SETUP: STATES, SUPERVISION, AND TRAINING

  • A response to query x induces states st = (x, y<t), where different response prefixes determine where teacher supervision is applied.Current-policy rollouts come from the evolving student, while initial-policy rollouts come from θ0 before OPD training.Source: §2 · p. 3
  • At each recorded state, the teacher’s next-token probabilities supervise the current student’s predictions conditioned on the same query and prefix.Resampling supplies new prefixes; replay returns to existing prefixes while recomputing current student and fixed teacher token probabilities.Source: §2 · p. 3
  • Experiments vary query sets, rollout resampling, and replay schedules while controlling trajectory and optimization-step budgets within each comparison.The main analyses use Qwen3-0.6B with Qwen3-4B-Instruct-2507 and evaluate transfer with Qwen3-1.7B; accuracy is averaged equally across benchmarks.Source: §2 · p. 3

3 QUERY COUNT AND ROLLOUT SAMPLING INTERACT

Under a fixed trajectory and optimization-step budget, query count and rollout-resampling frequency interact rather than providing uniform gains. The section also notes that rollout schedules differ in response completion under the 16K limit, complicating accuracy comparisons.

3 QUERY COUNT AND ROLLOUT SAMPLING INTERACT

  • The experiment crosses 8, 48, and 14,080 queries with 1, 10, or 110 collection rounds, using 14,080 trajectories and 110 optimization steps per configuration.All configurations use the same distillation objective and a 16K response limit.Source: §3 · p. 4
  • With fixed initial-policy rollouts, expanding from 8 queries to the full training set lowers average accuracy from 21.16% to 19.05%, while per-step resampling raises it from 23.61% to 25.57%.The difference between these effects is 4.07 percentage points (95% paired question-bootstrap interval [2.00, 6.28]).Source: Table 1 · p. 4, §3 · p. 4
  • Increasing collection rounds from 10 to 110 lowers accuracy for 8- and 48-query settings but improves it for the full training set.Most of the gain appears between one and ten collection rounds.Source: §3 · p. 4
  • Only 21.2%–24.9% of fixed-initial-rollout responses both contain an extractable answer and finish below 16K, versus 92.6%–96.8% with resampling.Roughly half of fixed-rollout responses reach the cap without an extractable answer, so the 16K comparison mixes answer correctness with answer timing.Source: §3 · p. 4

4 THE VALUE OF FRESH ROLLOUTS DEPENDS ON RESPONSE-LENGTH BUDGET

Current-policy sampling leads at shorter response budgets, but initial-policy training overtakes it at longer budgets under two teachers. The reversal coincides with longer responses from initial-policy training at 32K.

4 THE VALUE OF FRESH ROLLOUTS DEPENDS ON RESPONSE-LENGTH BUDGET

  • Current-policy sampling has higher average accuracy at 16K across the eight-benchmark suite.The comparison uses matched initialization, queries, trajectories, optimization steps, and training response limit within each teacher comparison.Source: §4 · p. 5
  • The evaluation uses eight mathematics benchmarks and scores the first 4K, 8K, 16K, or 24K tokens of the same continuations generated with a 32K limit.Responses are re-tokenized with the student tokenizer before scoring.Source: §4 · p. 5
  • At 32K, initial-policy training leads by 4.39 percentage points with Qwen3 (95% interval [3.07, 5.75]) and 2.07 points with JustRL ([0.49, 3.67]).The ordering reverses between 16K and 24K with Qwen3 and between 24K and 32K with JustRL; initial-policy training leads on all eight benchmarks with Qwen3 and six of eight with JustRL, with one tie.Source: §4 · p. 5
  • At 32K, mean response length is 21,736 versus 12,011 tokens with Qwen3, and 24,359 versus 14,152 with JustRL, for initial-policy versus current-policy training.Source: §4 · p. 5

5 WHAT REMAINS AS SUPERVISION MOVES ON?

The section separates whether initial-policy prefixes remain likely from whether the student has learned the teacher’s predictions on them. Most prefixes become less likely, while teacher–student disagreement decreases but persists.

5 WHAT REMAINS AS SUPERVISION MOVES ON?

  • Evaluating identical saved prefixes across checkpoints separates changes in prefix visitation from changes in conditional predictions.The study samples diagnostic prefixes from initial-policy responses and measures their likelihood and student–teacher reverse KL.Source: §5 · p. 5
  • After 40 optimization steps, 99.15% of sampled nonempty initial-policy prefixes are less likely, while mean teacher–student KL falls from 0.774 to 0.448.KL is 0.548 after 20 steps; each response receives equal weight in the prefix analysis.Source: §5 · p. 5
  • The student becomes less likely to revisit exact initial-policy prefixes even though its predictions there remain different from the teacher’s.The section tests whether replaying these states after current-policy training improves accuracy.Source: §5 · p. 5

6 REVISITING EARLIER STUDENT STATES

Initial-policy replay improves accuracy after current-policy training, and training on current-policy rollouts before replay outperforms the reverse order at longer response budgets. The comparisons hold trajectory and optimization budgets fixed.

6.1 WHICH TRAJECTORIES SHOULD BE REPLAYED?

  • Initial-policy replay improves average accuracy over continued current-policy sampling at 16K and 32K, while fixed recent rollouts stay close to continued sampling.The compared continuations start from the same R40 checkpoint and use 7,200 trajectories over 60 updates.Source: §6.1 · pp. 5–6, Table 3 · p. 6
  • The initial-policy bank contains shorter training responses than the R40 bank—758.85 versus 8,171.47 tokens on average—so response length alone does not explain its accuracy advantage.These are training-bank lengths; the table’s Tokens column instead reports final-model evaluation output lengths.Source: §6.1 · p. 7
  • Replay source changes which prefixes receive the larger improvement in teacher alignment: initial-policy replay reduces KL more on initial-policy prefixes, while continued sampling reduces it more on R40 prefixes.The KL comparison evaluates fixed diagnostic prefixes from both sources after continuations from the same R40 checkpoint.Source: §6.1 · p. 7, Table 4 · p. 7

6 REVISITING EARLIER STUDENT STATES

  • The order comparison allocates 20 current-policy and 20 replay steps after a shared R20 checkpoint, with both orders using the same initial-policy bank.The section’s comparisons control total trajectory and optimization-step budgets.Source: §6.1 · p. 5, §6.2 · p. 7

6.2 WHEN SHOULD THESE TRAJECTORIES BE REPLAYED?

  • Current-policy training followed by replay beats the reverse order by 1.61 points at 16K and 3.31 at 32K, but the reverse order is more accurate at 4K and 8K.The longer-budget gains occur on all eight benchmarks.Source: §6.2 · p. 7

7 R-OPD: REVISITING INITIAL-POLICY ROLLOUTS

R-OPD switches from current-policy sampling to initial-policy replay when gradient changes stabilize relative to minibatch variation. Across student sizes, it improves accuracy over continued current-policy sampling and a fixed replay schedule.

7.1 ADAPTIVE REPLAY CURRICULUM

  • R-OPD uses current-policy states first, then replays a stored bank of 2,400 responses sampled from the initial student on the same 48 queries.When the trigger fires at iteration τ, that iteration finishes on current-policy rollouts; replay begins at τ + 1.Source: §7 · p. 7, §7.1 · p. 7

7.2 GRADIENT-TRIGGERED TRANSITION

  • The trigger compares changes in mean gradients across current-policy iterations with minibatch-level variation and requires two consecutive qualifying comparisons.The switch begins on the iteration after the trigger; if it never fires, training remains on current-policy rollouts.Source: §7.2 · pp. 7–8

7.3 RESULTS ACROSS STUDENT SIZES AND RESPONSE BUDGETS

  • R-OPD gains 2.25 and 4.44 points over continued current-policy sampling at 16K and 32K for Qwen3-0.6B, and 2.81 and 6.15 points for Qwen3-1.7B.Table 5 reports eight-benchmark accuracy across three data orders; the comparison uses the same evaluation settings.Source: Table 5 · p. 8, §7.3 · p. 9
  • With Qwen3-8B and a Qwen3-30B-A3B teacher, R-OPD gains 4.05 points over current-policy sampling at 32K and exceeds fixed replay by 1.85 points.At 16K, R-OPD gains 0.78 points over current-policy sampling and exceeds fixed replay by 0.87 points.Source: Table 5 · p. 8, §7.3 · p. 9
  • Trigger timing varies by student size and data order; a 0.6B fixed-onset scan shows R-OPD near the high-performing region without prescribing one switch iteration.Across three shuffle seeds, mean trigger iterations are 5.67, 6.67, and 7.67 for 0.6B, 1.7B, and 8B; R-OPD yields 28.62/30.94% at 16K/32K versus 28.46/30.79% for τ = 5.Source: §7.3 · p. 9
  • Current-policy sampling is more accurate at 4K and 8K, while R-OPD is more accurate at 16K and above and produces longer mean responses at the same 32K cap.The reported response-length increases are 52%, 73%, and 41% for 0.6B, 1.7B, and 8B, respectively.Source: Figure 3 · p. 8, §7.3 · p. 9

8 RELATED WORK

Related work studies OPD state alignment, changes to probability updates or prefixes, and informative-sample selection in reinforcement learning. This paper holds the per-state objective and candidate selection fixed while varying rollout origin, response budget, and training order.

8 RELATED WORK

  • Prior OPD analyses study token corrections and alignment, while OPD variants alter probability updates or prefixes; RL work selects informative samples and tokens.The paper holds the per-state objective and candidate selection fixed while varying rollout origin, response budget, and training order.Source: §8 · p. 9
  • Concurrent work attributes few-query OPD to broad state coverage and slow alignment, while this work examines trajectory policy of origin and when earlier states are revisited.Related trajectory-reuse work is also cited; this paper stores response text and recomputes student and teacher probabilities for each scoring batch.Source: §8 · p. 9

9 CONCLUSION

R-OPD revisits earlier student prefixes because OPD can move supervision to new states before teacher-student disagreement on earlier states is resolved. Across eight mathematics benchmarks, it improves accuracy over continued current-policy sampling and fixed R40F20 replay, with longer responses at 32K.

9 CONCLUSION

  • Across eight mathematics benchmarks and three data orders, R-OPD exceeds continued current-policy training and fixed R40F20 replay at 16K/32K for three Qwen3 student sizes.Its higher accuracy and longer self-generated responses at 32K suggest that timely replay helps students express more of their reasoning capacity.Source: §9 · p. 9
  • OPD students determine which response prefixes receive teacher supervision, so more queries or fresher rollouts do not uniformly help.Earlier states can retain useful supervision after becoming less likely under the evolving student.Source: §9 · p. 9
  • R-OPD adaptively selects when to revisit earlier states, and its accuracy advantage over continued current-policy sampling is reported at 16K and 32K.The conclusion reports gains across three data orders and three student sizes.Source: §9 · p. 9

AI USE STATEMENT

The appendix documents AI assistance and the shared training setup, distillation objective, and rollout schedules used in the experiments.

AI USE STATEMENT

  • Generative AI assisted with manuscript polishing, related-work retrieval, and debugging portions of the experimental code; the authors reviewed the contributions and take responsibility for the paper and artifacts.Source: AI USE STATEMENT · p. 10

A.1 ROLLOUTS, TRAINING UNITS, AND EXPERIMENT BUDGETS

  • Comparisons match consumed trajectories and optimization steps, with token use reported separately because response length also affects it.An optimization step is one AdamW update; a training iteration scores a rollout batch and applies one or more updates, while a collection round samples trajectories from one checkpoint.Source: §A.1 · p. 12, Table 6 · p. 12

A.2 DISTILLATION OBJECTIVE AND SHARED CONFIGURATION

  • The shared loss is a reverse-KL-derived surrogate over the student’s top-16 tokens, using full-vocabulary log probabilities and student weights normalized on that candidate set.Task rewards, verifier scores, selective-token masks, entropy bonuses, and auxiliary KL losses are disabled.Source: §A.2 · p. 13

A.3 ROLLOUT SAMPLING SCHEDULES

  • The rollout grid ranges from initial-policy responses generated before training to current-student responses resampled before every optimization step.The central grid matches 14,080 trajectories and 110 optimization steps; the matched source comparison shares queries, initialization, objective, and update schedule.Source: §A.3 · p. 13

B.1 EVALUATION SUITES AND RESPONSE BUDGETS

The appendix specifies the evaluation and training protocols and expands results on query selection, rollout sources, response behavior, and uncertainty. Initial-policy rollouts have an accuracy advantage at longer budgets, while current-policy rollouts lead in direct 16K evaluation.

B.1 EVALUATION SUITES AND RESPONSE BUDGETS

  • Evaluation covers three- and eight-benchmark suites, equally weights benchmarks, and evaluates response budgets using direct 16K generations or saved 32K responses and shorter prefixes.Response-level accuracy is averaged over questions within each benchmark; Pass@n measures questions with at least one correct response, and majority vote uses the most frequent extracted answer.Source: §B.1 · p. 14, §B.2 · p. 14, Table 8 · p. 14

B.3 UNCERTAINTY

  • The reported 95% intervals quantify question-level uncertainty for fixed models and stored samples, while curriculum variability uses three data-order runs on the same fixed 48-query set.The continuation results are descriptive.Source: §B.3 · p. 14

C TRAINING QUERY SETS

  • Random and difficulty-selected query sets give similar accuracy and observed pass@32 under the same training budget, supporting random selection in the main experiments.The difficulty-selected set is the only query set selected using model behavior.Source: C TRAINING QUERY SETS · p. 15, Table 9 · p. 15

D.1 QUERY COUNT AND ROLLOUT SAMPLING

  • The query-count interaction in average accuracy is 4.07 points, with a 95% paired question-bootstrap interval of [2.00, 6.28].The interaction compares the full and eight-query conditions across per-step and single-round rollout collection.Source: §D.1 · p. 16

D QUERY-COUNT AND ROLLOUT-SOURCE RESULTS

  • Figure 4 reports response categories for each benchmark and sampling configuration as four combinations of answer extraction and reaching the 16K limit; extraction counts answers regardless of correctness, and the average weights the three benchmarks equally.Source: Figure 4 · p. 17

D.2 MATCHED ROLLOUT SOURCES AT A DIRECT 16K GENERATION LIMIT

  • At direct 16K, current-policy mean accuracy exceeds initial-policy accuracy by 2.63 points with Qwen3 and 6.72 points with JustRL.The respective 95% intervals are [−0.03, 5.33] and [3.65, 9.80].Source: Table 11 · p. 17, §D.2 · p. 17

D.3 ACCURACY AND GENERATION BEHAVIOR ACROSS RESPONSE-LENGTH BUDGETS

  • At 32K, initial-policy training reaches higher final accuracy with longer responses, while current-policy accuracy saturates earlier.Initial-policy/current-policy mean-length ratios are 1.81 under Qwen3 and 1.72 under JustRL at 32K.Source: §D.3 · p. 17, Table 12 · p. 18, Figure 5 · p. 18

E.1 SHARED CONTINUATION PROTOCOL

  • Continuation experiments restore both model and AdamW state and compare fixed initial-policy or recent rollout banks under a shared 2,400-response, 20-step stage.The evaluation uses the eight-benchmark suite and one training run per condition.Source: §E.1 · pp. 18–19

E.2 FIXED-PREFIX DIAGNOSTICS

The appendix probes how replay affects disagreement on earlier prefixes, compares replay timing and source controls, and reports R-OPD results across student sizes. Its evidence is scoped to competition mathematics with Qwen3 students.

E.2 FIXED-PREFIX DIAGNOSTICS

  • Fixed-prefix diagnostics show residual disagreement on initial prefixes after current-policy training, with a larger reduction when those prefixes are replayed.The KL matrix uses the same diagnostic responses and sampled positions across scoring students, with responses weighted equally.Source: §E.2 · p. 19, Table 13 · p. 19
  • 2.437975 versus 2.758502 nats on the highest 1,229 positions; this group contributes 94.66% of signed KL differences between continued current-policy training and replay.Replay has lower KL on 41.57% of positions and 77.60% of response-level averages; the top-10% partition is post hoc.Source: §E.2 · p. 19
  • 99.15% under R40 and 99.29% after its 20-step initial-policy continuation have lower exact-prefix probability than at initialization.These measurements concern recorded prefixes, not semantically equivalent responses.Source: §E.2 · pp. 19–20

E.3 CONTINUATION SOURCE AFTER TWO TRAINING HISTORIES

  • At 32K, initial-policy continuation is more accurate from both starting histories; after current-policy training it wins on five of eight benchmarks at 16K and seven at 32K.Early replay improves 32K accuracy but lowers 16K accuracy; the budget breakdown also shows the current-policy advantage at short prefixes giving way to replay gains at longer budgets.Source: §E.3 · p. 20, §E.4 · p. 21, §F.5 · p. 24

E.5 TRAINING ORDER AT FIXED SOURCE ALLOCATION

  • 1.61 and 3.31 percentage-point mean gains at 16K and 32K favor current-policy training followed by replay over the reverse order at fixed source allocation.Current-then-initial is more accurate on all eight benchmarks at 16K, 24K, and 32K; reverse order is stronger on all eight at 4K and seven at 8K.Source: §E.5 · p. 21, Table 17 · p. 21

E.6 MIXED SOURCES AND DIRECT EOS SUPERVISION

  • Source balancing increases 32K accuracy and response length but lowers answer extraction; removing the direct EOS term leaves accuracy and length close to full replay.EOS-only supervision yields shorter responses, higher answer extraction, and lower accuracy; initial-policy gradients also have 2.53- and 2.82-times the current-policy norms at updates 55 and 110.Source: §E.6 · pp. 21–22, Table 18 · p. 22, §E.7 · p. 22

F.3 PAIRED DATA-ORDER REPETITIONS AND TRIGGER ITERATIONS

  • R-OPD exceeds fixed replay in all three paired runs at 16K and 32K for every student size.The trigger sequences were (5, 5, 7), (7, 6, 7), and (7, 8, 8) for 0.6B, 1.7B, and 8B; two consecutive qualifying gradient comparisons start replay on the next iteration.Source: §F.2 · p. 23, §F.3 · p. 24, Table 21 · p. 24

H LIMITATIONS

  • The experiments focus on competition mathematics with Qwen3 students, so transfer to other task families and student architectures remains to be established.The authors identify evaluating those other tasks and architectures as the way to establish how broadly the results transfer.Source: H LIMITATIONS · p. 26

Related papers

Loading 2609.25048v2…