PaperlayerView exact revision on arXiv

Source-linked AI summary

Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models

Zheng Zhang, Xinyue Tan, Lufei Li, Xinyi Zhang, Yexin Li, Kan Ren

arXiv:2609.37132v1cs.AI

TL;DR

Existing OPSD teachers are typically tied to the learner’s current optimization stage, motivating whether later progress can provide stronger supervision for an earlier policy. B-OPSD trains a future teacher and uses it to supervise a restarted student; it consistently improves OPSD in both answer-available and answer-free settings.

  • Problem

    Existing OPSD self-teachers are typically built from initial, current, or slowly averaged policy states, leaving open whether optimization progress can provide stronger supervision for an earlier policy.

    Source: §1 · p. 1

  • Method

    B-OPSD temporarily trains a future policy, restores the student to its initial state, and selectively uses the future teacher for privileged trajectories and token-level targets.

    Source: §1 · p. 2

  • Results

    B-OPSD consistently improves standard OPSD in both settings; in rollout-based answer-available evaluation, average scores rise from 27.50 to 41.30 for Qwen3-4B and from 48.80 to 64.44 for Qwen3-8B.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    Optimization progress can serve as reusable supervision for an earlier policy, extending self-improvement beyond simply accumulating better parameters.

    Source: §6 · p. 9

Abstract

from arXiv · show

On-policy self-distillation (OPSD) improves large language models by letting a self-teacher with privileged information provide dense token-level supervision on the model's own trajectories. Yet existing methods typically construct the self-teacher from the current, initial, or slowly averaged policy state, leaving the quality of supervision constrained by the teacher's ability to exploit privileged information. We ask whether the model's own optimization progress can instead be recycled into a stronger self-teacher. In this paper, we introduce Bootstrapped On-Policy Self-Distillation (B-OPSD), which temporarily trains the policy ahead to obtain a future teacher, restores the student to the original policy state, and then uses the future teacher to supervise the restarted student. The future teacher improves supervision in two complementary ways, it can generate more reliable privileged trajectories and, conditioned on them, provide more informative token-level targets along the restarted student's on-policy trajectories. Experiments on mathematical reasoning with Qwen3-4B and Qwen3-8B show consistent improvements over standard OPSD in both settings, including gains from 27.50 to 41.30 and from 48.80 to 64.44 in the rollout-privileged setting. Our findings point to a broader principle for self-improving models that future learning progress can be distilled backward, preserving acquired knowledge while bootstrapping beyond the optimization state that produced it.

1 INTRODUCTION

OPSD uses privileged information for token-level supervision, but existing self-teachers remain tied to roughly the learner’s optimization stage. B-OPSD recycles later progress as supervision for an earlier policy state.

1 INTRODUCTION

  • B-OPSD temporarily trains a future policy, restores the student to its original state, and distills from the future teacher.The future teacher can generate more reliable privileged trajectories and provide more informative token-level targets along the restarted student’s own trajectories.Source: §1 · p. 2
  • Answer-available training routes verified future trajectories to future-teacher supervision, while answer-free training uses consensus to select when to query that teacher.Otherwise, the method retains standard OPSD supervision.Source: §1 · p. 2, Figure 1 · p. 2
  • Across three benchmarks and two open-source LLMs, B-OPSD consistently improves standard OPSD, with analyses indicating gains beyond extra updates alone.The reported analyses also find future supervision most effective early after restart and a different learning trajectory than continuing from the future checkpoint.Source: §1 · p. 2

2 RELATED WORK

Prior OPSD methods vary in privileged information and teacher construction, while research on optimization states has mostly reused past states rather than future states as teachers.

2 RELATED WORK

  • OPSD uses student-generated trajectories and privileged-context token supervision, with designs varying both the privileged information and the teacher’s parameter state.Examples include final answers, reference solutions, rollouts, initial-policy copies, EMA teachers, and teachers sharing student parameters.Source: §2 · p. 3
  • Earlier optimization-state methods aggregate or merge checkpoints, while later states guiding earlier learning remain less explored in OPSD.B-OPSD uses a future teacher for both privileged trajectories and improved token-level targets.Source: §2 · p. 3

3 PRELIMINARY

OPSD trains a model to serve as both student and self-teacher, using privileged information to supervise the student’s own trajectories.

3 PRELIMINARY

  • The student samples trajectories, while the same model conditions on privileged information and provides token-level supervision along those trajectories.The student’s next-token distribution is aligned with the teacher’s using forward KL divergence.Source: §3 · p. 3
  • The trajectory prefix marks the tokens generated so far, and stop-gradient prevents gradients from passing through the teacher branch.This clarifies which context the token-level supervision uses and which branch does not receive gradients.Source: §3 · p. 3

4 METHODOLOGY

B-OPSD trains a future self with lookahead OPSD updates, restarts the student from the initial policy, and selects supervision pathways according to answer availability and reliability.

4.1 LEARNING FROM A FUTURE SELF

  • K OPSD lookahead updates from the initial policy produce a future self, after which the student restarts from its original policy state.The restarted student generates the on-policy trajectories used for distillation.Source: §4.1 · p. 3
  • The future-self pathway conditions the future teacher on a privileged trajectory, while standard OPSD uses the initial policy as teacher; both distill along student trajectories.The objective uses a frozen teacher selected by the pathway.Source: §4.1 · pp. 3–4

4.2 ANSWER-AVAILABLE SETTING.

  • With gold answers available, lookahead training uses correct student rollouts when available and gold reasoning traces as fallback; restart-stage future supervision requires a correct future-generated trajectory.If that trajectory is incorrect, the method uses the initial policy as teacher, conditioned on a correct student rollout or the gold trace.Source: §4.2 · p. 4

4.3 ANSWER-FREE SETTING

  • Without gold answers or traces, majority-vote answers from model rollouts provide pseudo-labels for lookahead and restarted training.The future-self pathway is selected when student agreement is insufficient and future-policy answers meet the agreement threshold; otherwise, standard OPSD is used.Source: §4.3 · pp. 4–5

5 EXPERIMENTS

B-OPSD consistently improves OPSD across answer-available and answer-free mathematical reasoning settings. Ablations and analyses examine how teacher construction, privileged information, supervision timing, restarting, and training budget relate to performance.

5.1 EXPERIMENTAL SETTINGS

  • Experiments use Qwen3-4B and Qwen3-8B, with evaluation on AIME 2024, AIME 2025, and HMMT February 2025.Training uses the OPSD split of OpenThoughts-Math-30K; the evaluation reports AVG@12, PASS@12, and MAJ@12.Source: §5.1 · p. 5

5.2 MAIN RESULTS

  • 30.00 vs. 26.76 for Qwen3-4B and 57.41 vs. 48.67 for Qwen3-8B: B-OPSD improves answer-free average scores over OPSD.Gains occur on all three benchmarks using model-generated training signals.Source: Table 2 · p. 6, §5.2 · p. 6
  • 41.30 vs. 27.50 on Qwen3-4B and 64.44 vs. 48.80 on Qwen3-8B: B-OPSD raises rollout-based average scores over OPSD.With reference-solution privileged information, scores rise from 24.62 to 27.41 for Qwen3-4B and 37.22 to 44.17 for Qwen3-8B.Source: §5.2 · pp. 5–6

5.3 ABLATION STUDIES FOR B-OPSD

  • Replacing the base teacher with the future teacher raises scores from 27.50 to 34.07 with student PI and from 23.34 to 41.30 with future PI.Future rollouts score 41.30 with the future teacher, versus 34.07 with student rollouts; direct OPD scores 19.17 and 24.72, below B-OPSD’s 41.30.Source: §5.3 · p. 6, Table 4 · p. 7

5.4 IN-DEPTH ANALYSIS OF FUTURE-TEACHER SUPERVISION

  • Future (EMA) achieves 52.50 versus 41.30 for Future (Base), while applying future supervision at Steps 0–25 scores 41.30, compared with 30.19 and 27.96 later.Base, Self, and EMA teacher configurations score 27.50, 25.09, and 25.83, respectively.Source: §5.4.1 · p. 7, §5.4.2 · p. 7

5.5.1 RESTARTING THE STUDENT FROM THE BASE POLICY

  • In the shared parameter-plane projection, continuation trajectories remain near πF while the restarted student moves toward a higher-reward region.The analysis measures mean verifier reward on held-out problems.Source: §5.5.1 · p. 8, Figure 2 · p. 9

5.5.2 SELECTION OF PRIVILEGED INFORMATION

  • A correct student rollout scores 27.50, versus 20.83 with a reference solution; removing its boxed answer retains 93.7% of reasoning-token KL and scores 25.83.The final answer alone yields the lowest downstream score, 18.98, despite the largest KL on answer tokens.Source: §5.5.2 · p. 8

5.6 TRAINING EFFICIENCY

  • With a matched cumulative update budget, B-OPSD initially performs worse than uninterrupted OPSD but later surpasses it and reaches higher peaks on all three metrics.These results indicate that the gains cannot be explained solely by the extra updates used to construct the future teacher.Source: §5.6 · pp. 8–9

6 CONCLUSION

The conclusion frames OPSD effectiveness as jointly dependent on teacher capability and the privileged information it interprets. The algorithmic details describe answer-available and answer-free procedures that first construct a future teacher, then restart the student from its initial state.

6 CONCLUSION

  • B-OPSD treats optimization progress as reusable supervision for an earlier policy, with future-generated trajectories providing the greatest benefit when paired with the future teacher.The conclusion also reports that this supervision is most effective early after restart and can yield better learning trajectories than continuing from the future checkpoint.Source: §6 · p. 9

A ALGORITHMIC DETAILS

  • Both settings temporarily update the initial policy to construct a future teacher, then restore the student to its initial state for training with that teacher.The settings differ in how they select privileged information and decide when to use the future teacher.Source: A ALGORITHMIC DETAILS · p. 12
  • Answer-available training uses known correct answers, while answer-free training relies on agreement among generated answers to select privileged information.Source: A ALGORITHMIC DETAILS · p. 12

B IMPLEMENTATION DETAILS

The implementation uses Qwen3-4B and Qwen3-8B and compares OPSD variants on mathematical reasoning benchmarks. Its procedures specify trajectory selection, answer-free fallback rules, and evaluation metrics.

B IMPLEMENTATION DETAILS

  • The reproduced OPSD baseline uses n = 1 student rollout per problem and provides the dataset reference solution to the teacher as privileged information.Source: B IMPLEMENTATION DETAILS · p. 12
  • Answer-available n = 8 training selects the longest parseable gold-correct rollout as privileged information and the longest gold-wrong rollout as the student trajectory.If all student rollouts are incorrect, the dataset reference solution supplies privileged information.Source: B IMPLEMENTATION DETAILS · pp. 12, 14
  • The answer-available future pathway uses a verified correct future-policy rollout as privileged information; otherwise, the example follows the Base OPSD pathway.Future-policy sampling is greedy during the supervision window, while student training rollouts are stochastic.Source: B IMPLEMENTATION DETAILS · p. 13
  • Figure 4 compares PASS@12, MAJ@12, and AVG@12 across AIME24, AIME25, and HMMT25 at shared cumulative budgets of 50 through 150 updates.The dashed line marks the end of the 50 lookahead updates.Source: Figure 4 · p. 15
  • The answer-free protocol uses plurality-voted answers and queries the future policy only when the student lacks a strict majority; failed future checks fall back to standard OPSD.It marks generations that reach the maximum length without an end-of-sequence token as invalid.Source: B IMPLEMENTATION DETAILS · p. 15
  • Training uses the OPSD split of OpenThoughts-Math-30K (29,434 problems), evaluated on AIME 2024, AIME 2025, and HMMT February 2025, reporting average accuracy over sampled responses, pass rate, and majority-vote accuracy.Source: B IMPLEMENTATION DETAILS · p. 15

C ADDITIONAL RESULTS

The additional-results section points to a matched-budget comparison across benchmarks and metrics. The prompt templates distinguish answer-only, reference-solution, and model-generated candidate-solution privileged information.

C ADDITIONAL RESULTS

D PRIVILEGED-INFORMATION PROMPT TEMPLATES

E TEACHER-SOURCE PROTOCOL

The teacher-source comparison holds the training data, rollout-based privileged-information selection, optimizer, and evaluation protocol constant. Future variants restart from the base checkpoint and apply future supervision conditionally during an initial window.

E TEACHER-SOURCE PROTOCOL

  • All compared Qwen3-4B variants share training data, rollout-based privileged-information selection, optimizer, and evaluation protocol, while differing in target-producing policy and future-supervision duration.Source: E TEACHER-SOURCE PROTOCOL · pp. 16–17
  • Future variants restart the student from the base checkpoint, freeze the future policy, and apply future supervision conditionally for the first 25 restarted-student updates.Later updates use the corresponding Base or EMA teacher.Source: Table 8 · p. 17, E TEACHER-SOURCE PROTOCOL · p. 17
  • Table 9 covers standalone performance of the Qwen3-4B initialization and the two teachers used for direct OPD; its future model follows 50 lookahead OPSD updates.Source: Table 9 · p. 17
  • Future (EMA) differs from EMA only during those first 25 restarted-student updates; afterward both use the same EMA teacher construction.Source: E TEACHER-SOURCE PROTOCOL · p. 17

F DIRECT OPD WITH QWEN3-14B AND THE FUTURE TEACHER

Direct OPD from the future Qwen3-4B teacher outperforms OPD from Qwen3-14B, but B-OPSD achieves the strongest distillation result. The two frozen teachers have similar standalone average scores.

OPD protocol

  • The direct OPD variants train Qwen3-4B on the corresponding OPSD responses, with teachers providing token-level targets without privileged information or selective routing.The teachers are conditioned only on the original problem, and training optimizes full-vocabulary forward KL at each response-token position.Source: F DIRECT OPD WITH QWEN3-14B AND THE… · p. 17

Teacher capability

Distillation results

  • 41.30 for B-OPSD exceeds 24.72 for direct OPD from Future Qwen3-4B and 19.17 for direct OPD from Qwen3-14B.The comparison highlights B-OPSD's improvement over direct distillation from the same future teacher and from the larger teacher.Source: F DIRECT OPD WITH QWEN3-14B AND THE… · p. 18

G FUTURE-SELF MAJORITY ACCURACY ON ROUTED PROBLEMS

On answer-free training problems routed to the future-self pathway, future-policy majority-voted answers are more accurate than the restarted student's for both model sizes.

Future-self majority accuracy on routed problems

  • For both model sizes, the future policy's majority-voted answer is more accurate than the restarted student's on routed answer-free problems.Gold answers are used only for this post-hoc analysis.Source: G FUTURE-SELF MAJORITY ACCURACY ON… · p. 18

H VERIFIER-REWARD LANDSCAPE CONSTRUCTION

The verifier-reward landscape visualizes LoRA checkpoints and training trajectories in a two-dimensional optimization plane, using evaluated rewards and interpolated surface estimates.

Parameter plane

  • The parameter plane represents each LoRA checkpoint by coordinates along the future update and an orthogonal restart update rescaled to match its norm.The base model is the origin, and each coordinate represents an effective update xu + yv.Source: H VERIFIER-REWARD LANDSCAPE CONSTRUCTION · p. 18

Compared trajectories

  • The compared paths include Stage I, Restart, and two continuation branches, with future supervision applied during specified update windows.Stage I runs for 50 updates; Restart uses future supervision during updates 1–25, while Future-Continue uses it during updates 51–75.Source: H VERIFIER-REWARD LANDSCAPE CONSTRUCTION · p. 18

Reward surface and trajectories

  • Mean verifier reward is the fraction of 192 responses with a correct final answer across a 7 × 7 grid of evaluated coordinates.The grid uses 48 math problems and four non-thinking rollouts per problem.Source: H VERIFIER-REWARD LANDSCAPE CONSTRUCTION · p. 18
  • The visualization plots 28 directly evaluated checkpoints and interpolates the reward surface between evaluated locations using a smoothed thin-plate spline.The grid includes 49 evaluated points; interpolation is used only for the background surface.Source: H VERIFIER-REWARD LANDSCAPE CONSTRUCTION · p. 18
Loading 2609.37132v1…