PaperlayerView exact revision on arXiv

Source-linked AI summary

Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?

Yiping Li, Fan Zhao, Zhibo Hou, Zhiyu An, Wan Du

arXiv:2610.19145v1cs.LG

TL;DR

On-policy distillation is a cheaper alternative to reinforcement learning, but the teacher’s contribution and the student’s ability to learn it are difficult to distinguish. This paper evaluates teacher corrections through controlled interventions and checkpoint analysis, finding that selected repeated corrections improve accuracy while fork resolution alone does not ensure better answers.

  • Problem

    OPD outcomes do not distinguish the teacher’s available advantage from the student’s ability to acquire it, leaving unclear where teacher advantage appears along student-generated trajectories.

    Source: §1 · p. 1

  • Method

    Controlled interventions measure accuracy recoverable through teacher corrections, while checkpoint analysis tracks which teacher preferences OPD acquires.

    Source: §1 · p. 1, §6 · p. 8

  • Results

    Mass- and KL-based repeated-flip selection recover more accuracy than random flips at similar intervention counts, while fork resolution can increase without accuracy improving.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Evaluating OPD requires measuring both teacher preferences acquired during training and accuracy still recoverable through controlled corrections.

    Source: §6 · p. 8

    Experiments cover several Qwen, Llama, and Gemma teacher-student pairs, but only mathematics and instruction-following tasks; token interventions require shared tokenizers.

    Source: Limitations · p. 9

Abstract

from arXiv · show

On-policy distillation (OPD) is increasingly studied as a cheaper alternative to reinforcement learning, yet it remains a form of knowledge distillation (KD) that relies on teacher supervision. We revisit OPD from the KD perspective by asking where the teacher's advantage lies along student-generated trajectories. Controlled token flips at teacher-student disagreements directly measure the accuracy gained from teacher corrections. Single-token flips yield a modest average gain on selected failed trajectories, whereas repeated flips raise accuracy from 28.4% to 74.5%, close to the teacher's 76.9%. Gains continue to accumulate with more flips, but mass- and KL-based selection outperform random flips by 15-18 percentage points at counts of similar magnitude. Selective intervention remains useful after OPD, but greater argmax agreement does not consistently improve accuracy across model pairs. We advocate training schedules that learn more valuable teacher corrections while maintaining distributional stability. Controlled flips and fork-resolution analysis evaluate OPD beyond accuracy by measuring which teacher preferences the student acquires and what further correction can recover.

1 Introduction

On-policy distillation offers a cheaper alternative to reinforcement learning, but its teacher-supervised nature leaves unclear where the teacher’s advantage appears along student-generated trajectories. This paper separates the value of teacher corrections from the student’s ability to learn them, using controlled flips and checkpoint analysis.

1 Introduction

  • OPD trains students on their own rollouts toward a teacher’s next-token distribution, offering a cheaper route than reinforcement learning.Reported results reach RL-level accuracy at about a tenth of the cost.Source: §1 · p. 1
  • It remains unclear what in the teacher’s next-token distributions contributes to better reasoning and where that contribution appears along the student’s trajectory.OPD remains knowledge distillation because it depends on a teacher.Source: §1 · p. 1
  • Distillation outcomes combine what the teacher offers with what the student can learn, motivating interventions that hold student parameters fixed.Checkpoint analysis then tracks how OPD changes disagreement.Source: §1 · p. 1
  • The paper evaluates both accuracy recoverable through controlled teacher corrections and which teacher preferences students acquire.This separates the value of supervision from its learnability.Source: §1 · pp. 1–2
  • Teacher corrections accumulate gains, and mass- and KL-based selection recover more accuracy than random flips at similar intervention counts.Teacher guidance is broadly useful, but not all disagreements are equally valuable.Source: §1 · p. 2
  • Across model pairs, fork resolution can increase while accuracy stagnates or declines, and selective intervention remains useful after training.Resolving disagreements alone does not establish that OPD turns teacher guidance into better answers.Source: §1 · p. 2

2 Related Work

Prior OPD work modifies supervision, rollout generation, or training analysis, while guided-decoding methods combine model predictions for generation or speed. This paper uses teacher substitutions as a measurement of where teacher advantage enters student trajectories.

2 Related Work

  • On-policy distillation trains on student-generated trajectories with teacher supervision, and related methods select teachable, low-entropy, high-divergence, or prefix tokens.TA-OPD uses a 10% budget in its main experiments.Source: §2 · p. 2
  • Prior methods alter OPD supervision through advantage regulation, support-aware matching, asymmetric objectives, reward extrapolation, or position-dependent divergence choices.TGPO supervises teacher-preferred tokens on student-generated prefixes without intervening in generation.Source: §2 · p. 2
  • Other OPD methods change rollouts through reflection-triggered handoffs, backtracking, prefix refinement, or trust-region behavior blending.These approaches encode assumptions about where teacher advantage lies, but do not directly test whether it is localized there.Source: §2 · p. 2
  • OPD analyses track training runs, combining the teacher’s contribution with what the student learns; this paper isolates teacher contribution with fixed student parameters and separately tracks checkpoint disagreement.Prior analyses examine teacher-student distribution movement, training curves, or per-token advantage dynamics.Source: §2 · p. 2
  • Speculative, contrastive, and proxy decoding combine models for speed or sequence improvement, whereas this paper uses lockstep pairing as a measurement.The student keeps its sampled token unless an intervention selects the position.Source: §2 · p. 2

3 Preliminaries

On-policy distillation trains on student-generated trajectories while penalizing differences from the teacher’s next-token distribution. The implementation approximates the per-position reverse-KL objective using only the student’s top-ranked tokens.

3 Preliminaries

  • At each position, OPD applies a weighted reverse-KL penalty between student and teacher next-token distributions conditioned on the same prefix.The student generates the training trajectory; the teacher supplies the comparison distribution.Source: §3 · p. 2
  • The top-k approximation keeps the tokens the student ranks highest, with k = 16, rather than summing over the whole vocabulary.The full-vocabulary sum is described as expensive.Source: §3 · p. 3
  • The resulting gradient reaches only tokens the student already considers plausible; teacher tokens outside the student’s top-k are not pushed up directly.This constrains which teacher-preferred tokens receive explicit supervision.Source: §3 · p. 3
  • The experiments use reverse KL throughout, a mode-seeking divergence that charges probability placed where the teacher places little.The paper also notes forward KL and Jensen–Shannon as alternatives considered by GKD, and a position-dependent mix used by TrOPD.Source: §3 · p. 3

4 Setup

The setup uses a shared-tokenizer Qwen teacher-student pair on mathematical problems, then defines disagreements and interventions for evaluating teacher corrections. Single-flip and repeated-flip procedures change selected student tokens while leaving subsequent student generation in control.

4.1 Models, Data, and Grading

  • The experiments pair Qwen3-1.7B in non-thinking mode with Qwen3-4B-Instruct-2507 on DAPO-Math-17k, using shared-tokenizer comparisons and final-answer equivalence grading.Unless stated otherwise, both models sample at temperature 0.7 and top-p 0.95, with fixed seed 0.Source: §4.1 · p. 3

4.2 Fork Definitions and Scores

  • A fork is a position where student and teacher prefer different next tokens under the same prefix, though the sampled student token may still equal the teacher’s argmax.Disagreement strength is measured separately; blind forks place the teacher’s preferred token outside the student’s top-16 candidates, while reverse-blind forks place the student token outside the teacher’s top 16.Source: §4.2 · p. 3
  • Token mass distinguishes the teacher’s probability on the student token from the student’s probability on the teacher’s choice.The token set can be a single token or a top-k set; the paper’s reflection-word list contains 13 tokens and may include words used outside reflection.Source: §4.2 · p. 3

4.3 Interventions

  • A single flip replaces one fork token, lets the teacher supply a sampled segment of L tokens, then returns continuation to the student; at L = ∞, the teacher finishes the trajectory.The baseline independently resamples the student from the same prefix, including the fork token.Source: §4.3 · p. 3
  • Dual-track decoding replaces only selected student next tokens with the teacher’s argmax, then returns control to the student; replacements stop at an answer marker.Both models condition on the current prefix, and repeated replacements measure their combined effect along the resulting trajectory.Source: §4.3 · pp. 3–4

5 Empirical Analysis

Repeated teacher corrections substantially recover student accuracy, and selecting forks by teacher mass or KL improves recovery over random flips. After OPD, fork resolution and accuracy can diverge, while selective intervention remains useful and distributional stability matters.

5.4.1 How Does Fork Resolution Change During Training?

  • Forks accounted for 13.93% of rollout positions; in the main Qwen pair, Solved and New forks settled near 35.5% and 12.9% of the original fork count, while accuracy did not consistently track resolution.Fork proportions generally changed smoothly during training, with few large reversals.Source: §5.1 · p. 4, §5.4.1 · p. 6, Figure 2 · p. 6

5.2 Single Flip

  • Single-token replacement raised success from 9.91% to 11.31%, a trajectory-weighted gain of 1.41 pp over resampling at the same position.The study selected problems the teacher solved 4/4 and the student 0/4; subgroup gains varied, and tested features weakly predicted individual flip outcomes (AUC 0.554).Source: §5.2.1 · p. 4, §5.2 · p. 4, §5.2.2 · pp. 4–5

5.2.3 Does Intervention Timing Matter?

  • Later prefixes were harder to recover even when the teacher completed the continuation, while paired single-flip gains did not distinguish timing groups.On teacher trajectories, 73.4% of fork runs contained only one fork and fewer than 1% reached five tokens, suggesting mostly local disagreement.Source: §5.2.3 · p. 5, §5.2.4 · p. 5

5.3 Dual-Track Decoding

  • 74.5% accuracy after about 408 flips approached the teacher’s 76.9%, up from the student’s 28.4%; mass- and KL-based selection beat random by 15.4 and 18.1 pp at about 23–26 flips.The gains remained roughly proportional to preceding flips when intervention stopped early, supporting accumulated correction rather than late answer hints.Source: §5.3.1 · p. 5, §5.3.2 · p. 5, Figure 1 · p. 5

5.3.3 Do Reflection Tokens Provide a Particular Advantage?

  • Reflection-based selection increased reflection-word use and trajectory length but did not significantly improve accuracy over random flips; intervention also hurt on some student-solved problems.In a subgroup of 16 trajectories where the student solved problems the teacher did not, intervention reduced student accuracy from 31.2% to 18.8%.Source: §5.3.3 · p. 5, Table 2 · p. 5, §5.3.4 · p. 6

5.4.2 Which Forks Are More Learnable?

  • Fork-resolution patterns varied by teacher mass and reverse KL across model pairs, and blind forks resolved less often than non-blind forks: 14.6% versus 37.0%.The authors relate the lower blind-fork resolution to the top-16 loss lacking an explicit supervision term for the teacher-preferred token.Source: §5.4.2 · p. 6, Figure 3 · p. 7

5.4.3 How Much Does Teacher Intervention Help after OPD?

  • After OPD, mass-based selection remained more effective than random flips at low intervention counts, with curves that rose steeply and then flattened.Applying the same mass threshold brought checkpoints with different starting accuracies to similar performance.Source: §5.4.3 · p. 6, Figure 4 · p. 7

5.4.4 Which Remaining Forks Are Useful Intervention Targets?

  • Prioritizing New or Moved forks yielded no consistent accuracy advantage across model pairs, making those preference changes weak signals for further correction.In the main Qwen pair, New selection underperformed random controls and Moved showed no detectable advantage; neither group significantly improved accuracy over unassisted decoding in Gemma and Llama.Source: §5.4.4 · p. 7, Table 3 · p. 7

6 Conclusion

The paper separates the accuracy value of teacher corrections from the student’s ability to learn them. Its findings support evaluating both acquired preferences and remaining recoverable accuracy, while training directions aim to balance useful corrections with distributional stability.

6 Conclusion

  • OPD evaluation should measure both teacher preferences acquired during training and accuracy still recoverable through controlled corrections.This separates the value of teacher supervision from its learnability by the student.Source: §6 · p. 8
  • Repeated flips recover more accuracy than a single intervention, and mass- and KL-based selection outperform random flips at similar intervention counts.Gains continue to accumulate as more corrections are applied, despite weak prediction of individual single-flip gains.Source: §6 · p. 8
  • Across model pairs, greater fork resolution does not consistently improve accuracy, and only 35.5% of original forks are resolved in the main pair’s final checkpoint.Selective intervention remains useful after OPD, so saturated training accuracy need not mean useful teacher guidance is exhausted.Source: §6 · p. 8
  • A proposed training direction gives more supervision to Unchanged forks and less to New and Moved forks while maintaining distributional stability.The authors frame this as a hypothesis and propose organizing corrections by value and learnability.Source: §6 · p. 8

Limitations

The experiments are limited to mathematics and instruction following, use shared-tokenizer model pairs, and include a single-flip study selected to make teacher corrections easier to observe. The appendices specify the evaluation settings and analysis procedures across the experiments.

Limitations

  • The study covers several Qwen, Llama, and Gemma teacher–student pairs, but only mathematics and instruction-following tasks.The single-flip study selects problems the teacher solves and the student fails, so its average gains characterize that selected setting rather than random problems.Source: Limitations · p. 9

A.3 Single Flip

  • Single-flip evaluations use teacher segments of lengths L ∈ {1, 16, 128, ∞} and student continuations, with 5,901 eligible forks at L = 1.Eligibility requires that the teacher’s sampled token changes the original student token; at longer segment lengths, all teacher paths contribute.Source: §A.3 · p. 12

A.4 Dual-Track Decoding

  • Dual-track decoding accepts student tokens until a selected fork, replaces that token, discards the rest of the chunk, and resumes from the corrected prefix.Chunk size controls how many student tokens are generated and scored per round, not the teacher-intervention length.Source: §A.4 · p. 12

A.4.1 Reflection-Word List

  • Reflection-word matching uses a fixed list and retains words with non-reflective mathematical uses, including “no solution” and “well-defined”.Matching ignores case, leading whitespace, and attached punctuation.Source: §A.4.1 · p. 12

A.4.3 IFEval

  • The IFEval evaluation uses all 541 official prompts and strict prompt accuracy, requiring every instruction in a prompt to be satisfied.The reported interventions use teacher argmax replacements without an answer-zone freeze.Source: §A.4.3 · p. 13

A.6.1 Fork Resolution and Accuracy

  • After OPD, checkpoint fork changes are measured on fixed pretraining prefixes, whereas accuracy uses each checkpoint’s own rollouts.The fixed-prefix analysis includes all 729,574 original argmax forks, even when the sampled student token equals the teacher’s argmax.Source: §A.6.1 · p. 13

A.6.2 Fork Resolution by Pre-Training Scores

  • Fork-resolution groups are fixed using pretraining teacher mass or reverse KL, and reported rates are pooled position-level rather than problem-weighted.Teacher mass is the teacher’s probability of the base student’s argmax.Source: §A.6.2 · p. 13

A.6.4 Online Fork-Group Selection

  • Online New and Moved policies intervene at every eligible fork, while Old, Unchanged, and Random policies are calibrated toward 100 flips per trajectory.The realized counts are reported in Table 17.Source: §A.6.4 · p. 14

A.6.5 Additional Checkpoint Evaluations

The supplementary evaluations extend checkpoint analyses to additional model pairs and IFEval, while the statistical procedures compare paired outcomes at the problem or trajectory level. The tests account for correlated observations and treat fork-group comparisons as exploratory.

A.6.5 Additional Checkpoint Evaluations

  • Additional checkpoint evaluations cover Qwen, Llama, and Gemma on 1,000 DAPO-Math problems and IFEval’s 541 official prompts.Fork changes use each pair’s fixed pretraining trajectories and original fork count, with mass and reverse-KL group membership retained across checkpoints.Source: §A.6.5 · p. 14
  • The additional Gemma and Llama fork-group interventions use 200 DAPO-Math problems at checkpoints 16, 64, and 256.Each policy intervenes at every eligible fork in its selected group, and these runs report realized counts without budget calibration.Source: §A.6.5 · p. 14
  • The IFEval fork-group interventions use all 541 prompts, and the completed runs verify that each recorded teacher insertion changes the student’s preferred token.Thus the reported counts are actual flips.Source: §A.6.5 · p. 14

B Statistical Tests

  • Statistical tests aggregate differences before testing because positions within trajectories and trajectories from the same problem can be correlated.All reported t-tests are two-sided, and group-comparison p-values are unadjusted for multiple testing and interpreted as exploratory.Source: B Statistical Tests · p. 14

B.1 Single-Flip Tests

  • Single-flip gain tests compare teacher-intervention success with 16 independent student resamples from the same prefix.The paired-prefix comparison controls for how recoverable a position is without the teacher; at L = 1, 5,901 of 5,926 positions are eligible.Source: §B.1 · p. 15
  • Fork-group tests compare a group’s mean paired gain with the remaining positions, requiring at least three eligible positions on each side per included unit.A positive significant contrast indicates greater average teacher-intervention gain in the group, not merely higher absolute recovery.Source: §B.1 · p. 15

B.2 Dual-Track Tests

  • Dual-track policy comparisons use paired tests on problem-level correctness differences, rather than pairing individual trajectories.Policies generate different continuations, and random controls approximate intervention coverage or counts rather than matching exact realized flips.Source: §B.2 · p. 15

B.3 Handoff Tests

  • Handoff tests compare observed recovery with a constant-gain-per-flip prediction and test whether departures vary with teacher exit position.The prediction interpolates between student-alone accuracy and trajectory accuracy at f = 1, weighted by the fraction of flips made before f.Source: §B.3 · pp. 15–16

C Population Statistics

The population analysis profiles disagreements at positions where the sampled student token differs from the teacher’s argmax. At these forks, the other model’s preferred token is often highly ranked, though some disagreements are blind forks.

C Population Statistics

  • 12.23% of all positions are profiled forks where the sampled student token also differs from the teacher’s argmax.A sampled-token mismatch alone need not be a fork, because sampling can differ even when both models’ argmaxes agree.Source: C Population Statistics · p. 16
  • At profiled forks, the teacher’s argmax is the student’s second choice at 46% and its third choice at 17%.Most probability mass lies on the top two or three candidates of either model.Source: C Population Statistics · p. 16
  • At 7–8% of profiled forks, the other model’s argmax falls outside the top 16 in either ranking direction.These profiles distinguish disagreements over highly ranked alternatives from blind forks.Source: C Population Statistics · p. 16

D.1 Overall Gains by Intervention Length

Single-token teacher replacements improve outcomes over resampling, but their value varies by fork group and intervention length. Group comparisons depend on eligibility and aggregation, and replacement outperforms keeping the original token.

D.1 Overall Gains by Intervention Length

  • 0.01407 trajectory-weighted gain (95% CI [0.00662, 0.02153], p = 0.00047) for single-token flips over resampling; the problem-weighted gain is 0.01220 (p = 0.0152).Equally averaging trajectory means within each problem gives 0.01407 (p = 0.0207).Source: §D.1 · p. 16, Table 4 · p. 16

D.2 Gains by Fork Group

  • Group-versus-rest contrasts answer whether a fork group benefits more than other forks, and some significance depends on trajectory- versus problem-level aggregation.For longer segments, outcomes include additional teacher tokens and cannot be attributed to the first replacement alone.Source: §D.2 · p. 16, Table 5 · p. 17, Table 6 · p. 18

D.3 Group Eligibility and Success Rates

  • Larger gains do not necessarily mean higher replacement success: blind forks’ larger gains mainly reflect lower student-resampling success.The group comparisons use different eligibility rules and therefore different positions and trajectory weights.Source: §D.3 · p. 16, Table 5 · p. 17

D.4 Predicting Single-Flip Gains over Resampling

  • Single-flip gain prediction uses 2,598 non-tied forks; AUC is reported only for L = 1 because longer continuations do not isolate the initial token’s predictive value.The labels compare estimated true-flip success against the resample rate at the same fork.Source: §D.4 · p. 18

D.5 Keep-Token Baseline

  • Replacement improves over keeping the original token, while resampling and keeping do not differ detectably.The keep arm preserves the student token and samples 16 continuations at each fork prefix.Source: §D.5 · p. 18

E Intervention Timing and Recovery

Position alone does not reliably identify single-token interventions with larger gains, although gains decline from the penultimate to final decile. Fork disagreements are usually locally clustered, and student responses to a correction persist beyond the replacement token.

E Intervention Timing and Recovery

  • Gains decline from the penultimate to final decile across tested intervention lengths, but no single-token position group reliably yields larger gains after accounting for resampling.Absolute recovery declines later for both teacher intervention and student resampling.Source: E Intervention Timing and Recovery · pp. 18, 20, Table 7 · p. 19

F.1 Fork Spacing on Teacher Trajectories

  • The median distance between successive forks is three tokens, and 27.9% are adjacent; most consecutive fork runs contain only one or two tokens.Although 47.0% of forks belong to runs of at least two tokens, the runs are generally short.Source: §F.1 · p. 20, Figure 8 · p. 20

F.2 Student Responses after a Teacher Correction

  • After a replacement, student entropy stays elevated for about 40 tokens; on the teacher’s continuation, the student’s probability for teacher tokens rises from 0.61 to 0.83.The student’s entropy tracks the teacher’s within 0.05 nats on that continuation, while its teacher-token probability remains 0.06 to 0.13 below the teacher’s.Source: §F.2 · p. 20

G.1 Complete Results for the Main Model Pair

The appendix reports complete intervention results and robustness checks across teacher-exit positions, model sizes, datasets, and model families. Selective mass- and KL-based policies often outperform random flips at comparable intervention counts, while the effects vary by setting and subgroup.

G.1 Complete Results for the Main Model Pair

  • Against random controls, µT t < 10−3 gains 15.375 pp (p = 1.18 × 10−10), and KL > 5 gains 18.125 pp (p = 8.02 × 10−14); against the 0.22 control, KL > 4 gains 12.75 pp (p = 1.36 × 10−9).These comparisons use the same problem-paired test.Source: §G.1 · p. 20

G.2.1 Gains across Teacher Exit Positions

  • Teacher-exit tests show gains accumulating roughly in proportion to preceding flips, averaging 0.10–0.11 pp per flip across exit positions.The student completes reasoning and answering alone after intervention stops.Source: §G.2.1 · p. 22, §G.2 · p. 22, Figure 12 · p. 23

G.3 Ablation across Model Sizes

  • Across the two additional Qwen pairs, mass- and KL-based selection achieve higher accuracy than random flips at similar intervention counts, while more random flips also improve accuracy.This reproduces the overall pattern reported for the original pair.Source: §G.3 · p. 23, Figure 13 · p. 24

G.4 Ablation on a Different Dataset

  • On IFEval, mass- and KL-based selection remain beneficial, especially at low flip budgets, but random selection gradually approaches their curves as coverage increases.Even near 90% fork coverage, accuracy is around 76%, below the teacher’s 83.9%.Source: §G.4 · p. 23, Figure 14 · p. 24

G.5 Ablation across Model Families

  • At about 20–25 flips per trajectory on DAPO-Math, mass selection reaches 8.4% accuracy, KL selection 6.5%, and random selection 4.4%.The same KL > 1 threshold covers 28.8% of Llama forks, versus 49.5–74.5% in the additional Qwen pairs.Source: §G.5 · p. 23

G.6 Results by Problem Group

  • For 16 trajectories on problems the student solves but the teacher does not, accuracy falls from 31.2% student-alone to 18.8% under either tested mass threshold.The result illustrates that teacher replacement is not uniformly beneficial.Source: §G.6 · p. 23, Table 14 · p. 24

G.7 Teacher Sampling versus Argmax

  • Teacher sampling improves accuracy at µT t < 0.05, while the other tested thresholds show no significant difference from argmax replacement.The comparisons concern complete online policies because sampled replacements can change subsequent prefixes.Source: §G.7 · p. 23, Table 15 · p. 24

H.1.2 Additional Model Pairs and IFEval

Across model pairs, changes in fork resolution and accuracy do not follow one common trajectory. Additional analyses show selective post-training recovery, but also distributional changes at positions where the student and teacher already agreed.

H.1.2 Additional Model Pairs and IFEval

  • By the final checkpoint, 22.1% of original Llama forks, 32.5% of Gemma forks, and 15.1% of IFEval forks are solved, while accuracy does not follow the same increase.Source: §H.1.2 · p. 24

H.1.3 Composition of Resolved Forks

  • At step 256, forks with µT t < 0.05 make up 39.7% of original forks and 50.7% of resolved forks, while the most extreme mass and low reverse-KL groups remain small.The µT t < 10−5 group is 2.3% of original forks and 4.0% of resolved forks; reverse KL below 0.1 is 1.0% and 1.4%, respectively.Source: §H.1.3 · p. 25, Figure 17 · p. 26

H.1.4 Fork Resolution by Mass and KL across Pairs

  • Mass resolution is strongly ordered in the 0.6B Qwen student, from 56.7% at µT t < 10−5 to 22.8% at µT t ≥0.3, but the 8B-to-4B pair lacks the same monotonic pattern.Lower-mass groups are also more frequently resolved in Llama and IFEval, though less strongly than in the 0.6B student.Source: §H.1.4 · p. 25, Figure 18 · p. 27, Figure 19 · p. 27

H.2.1 Additional Qwen Model Size

  • At the 4B-to-0.6B Qwen final checkpoint, KL-based intervention with KL > 3 raises accuracy from 27.88% to 58.75% with about 105 flips per trajectory.This shows further teacher correction can recover accuracy after OPD in this pair.Source: §H.2.1 · p. 28, Figure 21 · p. 29

H.2.2 Instruction Following

  • On IFEval, final-checkpoint accuracy rises from 66.27% without intervention to 72.04% with KL > 3 and about 18 flips per trajectory; KL > 1 reaches 74.03% with about 84 flips.Source: §H.2.2 · p. 28, Figure 21 · p. 29

H.3.2 Gemma and Llama

  • At final, Gemma Old- and Unchanged-fork interventions reach 43.0% accuracy, versus 32.0% without intervention; New and Moved reach 32.5% and 30.5%.Llama’s New and Moved policies reach 3.5% versus 1.6% without intervention, while Unchanged reaches 6.0%; the group ranking varies across checkpoints.Source: §H.3.2 · p. 28, Table 18 · p. 29, Table 19 · p. 30

H.3.3 IFEval

  • On IFEval, none of the four fork-group policies consistently improves accuracy across checkpoints at the tested low flip counts; differences range from −1.06 to +0.65 pp.The policies use 0.44 to 0.58 flips per trajectory.Source: §H.3.3 · p. 28, Table 20 · p. 30

H.4.2 Probability Changes at Originally Agreed Positions

  • On positions where the student and teacher still agree, entropy rises from Base to Final in Llama, Gemma, and IFEval, while the shared token’s probability falls by 2.20, 3.54, and 4.20 pp, respectively.Fork resolution tracks preferred-token changes, so these distribution changes are not captured by resolution alone; Qwen and the 8B-to-4B pair show different trajectories.Source: §H.4.2 · p. 31, §H.4.1 · p. 31, Table 21 · p. 32

Related papers

Loading 2610.19145v1…