PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridged Self-Distillation for On-Policy Learning

Runsheng Chen, Yihong Liu, Raoyuan Zhao, Michael A. Hedderich, Hinrich Schütze, Matthias Grabmair

arXiv:2610.26397v1cs.CL

TL;DR

On-policy distillation seeks to transfer strong teachers’ reasoning to smaller students, but teacher probabilities may be inaccessible or incompatible across tokenizers, and direct imitation can be difficult. BSD uses an external reasoning trace as privileged context for a student-copy self-teacher that scores the student’s own rollouts. Across teacher-access settings, BSD improves base models and generally outperforms baselines tailored to those settings.

  • Problem

    On-policy distillation faces incompatible or inaccessible teacher probabilities across tokenizers and access settings, while directly imitating a stronger teacher’s reasoning can be difficult for smaller students.

    Source: §1 · pp. 1–2

  • Method

    BSD uses an external teacher’s reasoning trace as privileged context for a copy of the student, which scores the student’s own rollouts to provide token-level guidance.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    BSD improves every base model and achieves the best or tied average performance among distillation methods in eleven of twelve teacher–student pairs across three teacher-access settings.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    Reasoning traces provide a common interface for on-policy distillation across shared-tokenizer, cross-tokenizer, and closed API teachers.

    Source: §8 · p. 9

    Experiments are limited to students up to 4B parameters from one model family and English competition mathematics; BSD also requires automatically checkable answers.

    Source: Limitations · p. 9

Abstract

from arXiv · show

On-policy distillation can transfer the reasoning capabilities of a strong teacher to a smaller student. However, distillation from a stronger teacher model faces two mismatches: its solutions may be difficult for a smaller student to imitate, and its token-level probabilities may be incompatible with, or even unavailable to, the student. Existing methods handle these settings through different forms of supervision. To this end, we propose Bridged Self-Distillation (BSD), a unified framework for distilling from any teacher that provides reasoning traces. The model to be distilled from, termed the external teacher, first provides a reasoning trace as privileged context, which can be generated in an offline fashion. The student generates rollouts without this context, while a copy of the student conditioned on it, serving as the self-teacher, scores the same rollouts in the student's own token space. BSD uses the resulting token-level guidance to adjust updates on unsuccessful rollouts, with a verifier providing the outcome signal. As the external teacher is queried only to produce text, the same training procedure applies to all teacher scenarios: shared-tokenizer, cross-tokenizer, and closed API teachers. Across these scenarios, BSD improves over the base model and outperforms baselines tailored to each scenario, offering a novel and unified pathway for distillation. Code is available: https://github.com/TUMLegalTech/Bridged-Self-Distillation

1 Introduction

On-policy distillation aims to transfer a strong teacher’s reasoning to smaller students, but teacher probabilities may be inaccessible or incompatible across tokenizers, while directly imitating difficult reasoning can challenge students. BSD uses external reasoning as privileged context for a student-copy self-teacher to guide the student’s own rollouts, and is reported to outperform baselines across three teacher-access settings.

1 Introduction

  • BSD turns an external teacher’s reasoning trace into token-level guidance by conditioning a copy of the student on that trace to score the student’s own rollouts.A verifier determines update direction, while the guidance adjusts penalties on unsuccessful rollouts.Source: §1 · p. 2
  • Because the external teacher supplies text rather than probabilities, BSD applies the same training procedure to shared-tokenizer, cross-tokenizer, and closed API teachers.Figure 1 contrasts this unified approach with scenario-specific training signals.Source: Figure 1 · p. 1, §1 · p. 2
  • BSD generally outperforms baselines across shared-tokenizer, cross-tokenizer, and closed API settings in evaluations spanning twelve teacher–student pairs.The evaluation covers three math reasoning benchmarks.Source: §1 · p. 2
  • Analyses report stable training, benefits from more complete reasoning traces, and guidance concentrated on uncertain tokens, with stronger guidance associated with larger performance gains.The reported association does not establish causation.Source: §1 · p. 2

2 Preliminaries

GRPO uses verifier-assigned binary rewards to form group-normalized advantages for policy updates. Privileged self-distillation adds a context-conditioned copy that scores the student’s rollouts, and Figure 2 depicts BSD’s use of token-level guidance with verifier-directed updates.

2 Preliminaries

  • GRPO normalizes verifier-assigned binary rewards within a rollout group and applies each rollout’s advantage uniformly to its tokens in a clipped policy-gradient update.When a group has both successful and unsuccessful rollouts, their advantages are positive and negative respectively; uniform-outcome groups contribute no gradient.Source: §2 · p. 2
  • In privileged self-distillation, the student generates rollouts without privileged context, while a context-conditioned copy scores the same tokens and refines token-level advantages.The verifier sets update direction; the privileged context changes how strongly individual tokens are updated.Source: §2 · pp. 2–3
  • BSD’s overview shows an external teacher providing reasoning trace z, a self-teacher scoring the student’s context-free rollouts, and token-level guidance reweighting unsuccessful-rollout penalties.The verifier determines the direction of every update.Source: Figure 2 · p. 3

3 Bridged Self-Distillation

BSD uses an external teacher’s reasoning trace as privileged context, then applies a student-conditioned self-teacher to guide the student’s own rollouts. Verifier outcomes determine how token-level guidance changes updates, and the reweighting applies only to unsuccessful rollouts.

3.1 External Teacher

  • BSD needs only an external teacher’s reasoning trace, used as privileged context whether it contains full reasoning or only a final response.The teacher supplies text rather than token probabilities, so the setup applies to shared-tokenizer, cross-tokenizer, and closed API teachers.Source: §3.1 · p. 3

3.2 Self-Teacher

  • A frozen copy of the student conditioned on the question and trace scores the student’s same rollout token by token.Both scores therefore come from the student model and share its token space, avoiding external-teacher logits and vocabulary alignment.Source: §3.2 · p. 3
  • BSD defines token-level guidance as the self-teacher’s per-token log-probability difference from the student.Positive guidance means the conditioned self-teacher assigns more probability to the chosen token; negative guidance means less.Source: §3.2 · p. 3

3.3 Learning

  • For unsuccessful rollouts, guidance reweights the verifier-derived negative advantage, while successful rollouts retain their original advantage.The positive weight changes penalty magnitude without changing the advantage’s sign; its strength decays over training and clipping bounds its effect.Source: §3.3 · pp. 3–4

4 Experimental Setup

The experiments use Qwen3 students and three external-teacher access settings, train on sampled mathematical reasoning problems, and evaluate across three competition-level mathematics benchmarks. Baselines include SFT and setting-specific OPD methods.

4.1 Models

  • Four students are evaluated: Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, and Qwen3-4B-Instruct, the last fine-tuned on 1,000 reasoning traces to enable reasoning blocks.Teachers are Qwen3-32B, Nemotron-3-30B-A3B, and GPT-5.6 Luna for shared-tokenizer, cross-tokenizer, and closed API access, respectively.Source: §4.1 · p. 4

4.2 Training

  • Each teacher–student pair uses 2,400 sampled OpenThoughts mathematical-reasoning training problems with one external-teacher trace per problem.The Qwen3-32B and Nemotron traces average about 4,000 tokens, while GPT-5.6 Luna returns final responses averaging 247 tokens.Source: §4.2 · p. 4

4.3 Evaluation

  • Evaluation uses AIME 2024, AIME 2025, and HMMT 2025, selecting checkpoints on 128 separate OpenThoughts development problems.Four samples are generated per benchmark problem; Avg@4 is average correctness and Pass@4 is the share solved by at least one sample.Source: §4.3 · p. 4

4.4 Baselines

  • Baselines use the same backbones and training problems: SFT imitates teacher traces, while vanilla OPD, xOPD, and ROPD match the three teacher-access settings.xOPD aligns tokenizations and maps teacher probabilities; ROPD asks the closed API teacher to create a rubric and score rollouts.Source: §4.4 · p. 4

5 Main Results

Across teacher-access settings, BSD improves over the base model and generally leads the distillation baselines. Compared with SFT, BSD also preserves response length and avoids the training-performance collapse reported for the closed API setting.

5.1 Comparing Teacher Access

  • BSD improves Avg@4 over the base model in all 12 teacher–student pairs and achieves the best average score in 11.Table 1 reports Avg@4 across three mathematical reasoning benchmarks.Source: Table 1 · p. 5, §5.1 · p. 5
  • With shared-tokenizer teachers, BSD matches or beats vanilla OPD for all four students and exceeds SFT by 3.3–8.6 points.SFT falls below the base model for three students; BSD ties OPD for Qwen3-1.7B and scores up to 3.6 points higher for the other three.Source: §5.1 · p. 5
  • With cross-tokenizer teachers, BSD improves all four students by 1.2–4.4 points and beats xOPD for three, while xOPD’s gains are inconsistent.xOPD’s changes range from a 3.4-point gain for Qwen3-0.6B to a 0.8-point decrease for Qwen3-1.7B.Source: §5.1 · p. 5
  • With closed API teachers, BSD beats SFT by 3.1–9.7 points and ROPD by 0.6–3.6 points across all four students.The API exposes neither token probabilities nor its tokenizer; SFT falls below the base model for three students, including Qwen3-0.6B dropping from 10.8 to 2.8.Source: §5.1 · pp. 5–6

5.2 Training Progression

  • Across settings, SFT response-length changes depend on the teacher, while BSD and the corresponding OPD baselines generally keep lengths stable.With the closed API teacher’s 247-token responses, SFT progressively shortens all four students’ outputs; BSD and ROPD remain relatively stable.Source: §5.2 · p. 6, Figure 3 · p. 6
  • For closed API distillation, SFT falls to nearly zero performance by training’s end, whereas BSD and ROPD remain stable or improve slightly.Figure 3d shows Qwen3-4B-Instruct distilled from GPT-5.6 Luna; Appendix H reports similar SFT declines or collapses in other pairs.Source: §5.2 · p. 6, Figure 3 · p. 6

6 Further Analysis

Ablations find higher downstream performance with privileged context and more complete traces, while applying guidance to successful rollouts does not help. Additional analyses associate stronger BSD guidance with uncertain tokens and larger gains, and illustrate self-correction on one problem.

6.1 Ablation Study

  • Avg@4 rises from 59.2 for the base model to 63.9 with trace-conditioned BSD, versus 60.6 with GRPO and 61.7 with answer-conditioned RLSD.The three methods have nearly identical training-accuracy curves, with average differences below 0.2.Source: §6.1 · p. 7
  • More complete privileged context corresponds to higher Avg@4: BSD-hint scores 61.7, BSD-guide 62.5, and BSD-trace 63.9.The three variants follow similar training-accuracy curves.Source: §6.1 · p. 7
  • Weighting successful rollouts does not help: BSD with successful-rollout weighting scores 61.4 versus 63.9 for BSD-trace.OPSD scores 59.7 and declines from 63.3 at step 25 to 55.8 at step 150, while its benchmark responses shorten from 16.8k to 12.5k tokens.Source: §6.1 · p. 7

6.2 What the Token-Level Guidance Captures

  • Guidance magnitude correlates 0.83–0.90 with student token entropy on unsuccessful rollouts, and stronger guidance is associated with larger BSD gains.Qwen3-4B-Instruct has the weakest guidance and smallest gain, while Qwen3-1.7B and Qwen3-4B have stronger guidance and larger gains.Source: §6.2 · pp. 7–8
  • On one AIME24 example, the BSD-trained student rechecks an erroneous division and solves six of eight continuations, versus four for the base model.The base Qwen3-4B initially finds the correct integer but misreads its quotient and remainder.Source: §6.2 · p. 8

7 Related Work

Prior work studies knowledge distillation, on-policy distillation, and privileged-context self-distillation; BSD uses external-teacher reasoning as privileged context within a unified framework.

Knowledge distillation

  • BSD differs from prior heterogeneous-tokenizer and black-box OPD variants by offering one framework across these settings.Those variants typically require setting-specific supervision.Source: §7 · p. 8

Case study

  • In the AIME 2024 case study, the BSD-trained student checks a quotient and remainder after the base model continues from the same erroneous prefix.Across eight continuations, the BSD student solves the problem six times versus four for the base model.Source: Figure 5 · p. 8

Privileged information and self-distillation

  • BSD follows privileged-context self-distillation but uses reasoning from an external teacher as the privileged context.Earlier approaches condition the model on answers, solutions, feedback, or successful trajectories.Source: §7 · p. 8

8 Conclusion

BSD uses an external teacher’s reasoning trace as privileged context for a student-copy self-teacher, yielding guidance in the student’s token space. Across twelve teacher–student pairs, it improves every base model and achieves the best or tied average performance among distillation methods in eleven pairs.

Conclusion

  • Across twelve teacher–student pairs, BSD improves every base model and achieves the best or tied average performance among distillation methods in eleven pairs.Pairs span shared-tokenizer, cross-tokenizer, and closed API settings.Source: §8 · p. 9
  • BSD uses an external teacher’s reasoning trace as privileged context for a self-teacher, which produces guidance in the student’s own token space without teacher probabilities or vocabulary mappings.The student generates the rollouts that the conditioned copy scores.Source: §8 · p. 9
  • BSD’s analyses find stable training, benefits from more complete reasoning, and guidance focused on uncertain tokens.Stronger guidance is associated with larger performance gains.Source: §8 · p. 9

Limitations

The evaluation covers twelve teacher–student pairs and three teacher-access settings, but experiments and analyses have stated scope limits.

Evaluation scope

  • The evaluation spans twelve teacher–student pairs and shared-tokenizer, cross-tokenizer, and closed API teachers.Source: Limitations · p. 9

Limitations

  • Experiments cover students up to 4B parameters from one model family and English competition mathematics.Whether gains hold for larger students, other model families, or other domains remains future work.Source: Limitations · p. 9
  • BSD relies on a verifier to set update direction, limiting its application to automatically checkable tasks rather than open-ended ones.Using student–teacher agreement as a verifier for open-ended tasks remains unexplored.Source: Limitations · p. 9
  • The analysis links token-level guidance to BSD gains only through correlations across students and does not establish which tokens help or why.Source: Limitations · p. 9

Ethics Statement

The appendix documents the study’s ethics and experimental procedures, then interprets BSD’s small-gap update and its treatment of successful rollouts. Its derivation is qualified as an interpretation rather than a guarantee of training stability or credit assignment.

Ethics Statement

  • The study uses public models and mathematical reasoning datasets with automatic verifiers, and reports no human subjects, personal data, or newly collected data.The authors also state that they release code but no model weights, datasets, or teacher traces, and disclose AI-assistant use for development and writing.Source: Ethics Statement · p. 9

A Experimental Details

  • Teacher traces are generated before training; BSD adds a self-teacher forward pass taking 7–18 seconds per update, under 10% of update time.The three benchmarks each contain 30 problems, and the reported training, evaluation, and ablation runs use about 1,450 GPU-hours.Source: A Experimental Details · p. 13

B Prompt Templates

  • Prompts compute guidance on the student’s original response tokens, while context-form ablations compare traces, answers, guides, and hints.For closed teachers that hide reasoning, the returned derivation is formatted as a reasoning block with the final answer after it.Source: A Experimental Details · p. 13, B Prompt Templates · p. 15

C Proofs

  • The proof uses rollouts from the old policy and a frozen student copy as self-teacher; guidance and advantages are treated as stop-gradient quantities.The stated expectations are over prompts, traces, and rollouts; the argument does not depend on how often the self-teacher is synchronized.Source: C Proofs · p. 16

C.1 Small-Gap Decomposition

  • For small |δt|, the BSD gradient is decomposed into the GRPO gradient plus a token-level correction on unsuccessful rollouts.The authors explicitly describe this decomposition as interpretive; it does not guarantee stability or good credit assignment.Source: §C.1 · p. 16

C.2 Proof of Proposition 1

  • For A < 0, expanding e^-δt yields an outcome-weighted OPD correction restricted to unsuccessful rollouts.The derivation evaluates the update at the rollout policy with clipping inactive for sufficiently small |δt|.Source: §C.2 · p. 17

C.3 Weighting Successful Rollouts

  • Weighting successful rollouts adds a correction toward the self-teacher; BSD instead leaves their GRPO update unchanged.The paper does not evaluate the alternative weight that would push successful rollouts away from the self-teacher.Source: §C.3 · p. 17
  • BSD (w/ succ.) scores 61.4 against 63.9 for BSD.The variant applies a self-teacher correction to successful rollouts, while BSD reinforces successful solutions with the GRPO advantage alone.Source: §C.3 · p. 17

D Ablation Details

The ablations compare context forms, update variants, and response patterns, while later sections describe checkpoint selection and Pass@4 reporting. The additional training-progress results show SFT declining while BSD stays stable in three further settings.

D Ablation Details

  • OPSD’s Avg@4 falls from 63.3 to 55.8 as its response length drops from 16.8k to 12.5k tokens; the base model produces 17.5k tokens.The other variants’ response lengths stay between 16.5k and 17.7k tokens.Source: D Ablation Details · p. 17, Figure 6 · p. 18
  • On an incorrect interval step, trace-conditioned guidance strongly downweights the wrong numerator and downstream tokens, while answer-only guidance is close to zero.The full trace contains the correct derivation, whereas the answer alone does not identify which intermediate step is wrong.Source: D Ablation Details · pp. 17–18, Figure 7 · p. 18

E Wording of Incorrect and Correct Responses

  • Incorrect base-model responses contain more negation and doubt, while correct BSD-student responses contain more verification and average fewer tokens.The groups differ in both model and correctness, so the wording differences cannot be attributed to BSD alone.Source: E Wording of Incorrect and Correct… · p. 18

F Selected Checkpoints

  • Each run’s selected checkpoint is identified by its training step; checkpoints are saved every 25 steps and selected on the development set.The earliest checkpoint is selected in case of a tie.Source: F Selected Checkpoints · p. 18, Table 13 · p. 19

G Pass@4 Results

  • Pass@4 reports the share of problems solved by at least one of four samples, per benchmark and averaged across the three benchmarks.The table marks the best score among the three methods for each teacher–student pair.Source: G Pass@4 Results · p. 18

H Additional Training Progression Results

I Out-of-Domain Evaluation

The out-of-domain evaluation tests whether distillation affects the Qwen3-4B student’s performance on general-knowledge and legal questions. BSD and OPD remain near the base model across teachers, while SFT falls below it in several cases.

I Out-of-Domain Evaluation

  • The evaluation uses two 200-question sets: non-STEM MMLU-Pro questions and English and German LEXam questions sampled in benchmark proportions.The base model scores 63.5 on MMLU-Pro and 45.9 on LEXam.Source: I Out-of-Domain Evaluation · p. 19
  • With GPT-5.6 Luna, SFT shortens responses on both question sets, whereas BSD and OPD stay within 300 tokens of base-model length at every checkpoint.With Nemotron-3-30B-A3B, SFT lengthens responses on MMLU-Pro; the figure reports mean response length across checkpoints.Source: I Out-of-Domain Evaluation · pp. 19–20, Figure 10 · p. 21
  • BSD averages 63.8–64.5 on MMLU-Pro and 43.9–44.9 on LEXam across checkpoints, with standard deviation at most 1.1 points; base-model LEXam variation is 4.9 points.The reported gap between the base model and trained students on LEXam is comparable to that sampling noise.Source: I Out-of-Domain Evaluation · p. 20, Table 15 · p. 20
  • BSD and OPD stay within 1.1 points of the base model on MMLU-Pro and 3.5 points on LEXam; SFT is significantly below base in three cases.The reported SFT declines include GPT-5.6 Luna on MMLU-Pro and LEXam, and Nemotron-3-30B-A3B on MMLU-Pro.Source: I Out-of-Domain Evaluation · p. 19, Figure 9 · p. 21

Related papers

Loading 2610.26397v1…