PaperlayerView exact revision on arXiv

Source-linked AI summary

On-Policy Distillation Teaches New Skills but Not New Knowledge

Yixuan Tang, Yi Yang

arXiv:2610.09639v1cs.CL

TL;DR

The paper asks whether OPD gives students new factual knowledge, compositional skill, or both, a distinction obscured by end-to-end benchmarks. It independently controls teacher additions and student rollouts, finding strong compositional transfer but weak factual transfer under reverse KL, with forward KL more effective for facts.

  • Problem

    End-to-end reasoning scores cannot distinguish newly acquired facts from improved coordination of intermediate results, though these capabilities make different demands.

    Source: §1 · p. 1

  • Method

    The authors independently control the student’s initial capabilities and teacher additions, then evaluate factual recall and compositional skill separately across synthetic and real-world tasks.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across synthetic and real-world benchmarks, reverse-KL OPD transfers compositional skill strongly but factual knowledge weakly; forward KL improves factual transfer, while student rollouts improve multi-step execution.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    Reverse-KL OPD is better suited to organizing and using existing knowledge than updating factual knowledge; forward-KL supervision is more effective when factual transfer is the goal.

    Source: §7 · p. 12

Abstract

from arXiv · show

On-policy distillation (OPD) strengthens language-model reasoning, yet whether students acquire new factual knowledge or compositional skill for multi-step reasoning remains unknown. We separate these capabilities using a controlled synthetic framework that measures the student's initial capabilities and independently controls the teacher's additional facts, compositional skill, or both. Across four models from three families, reverse-KL OPD reliably transfers compositional skill across unseen reasoning structures, but transfers minimal factual knowledge. Decoupling the distillation recipe reveals the source of this asymmetry: replacing reverse KL with forward KL restores factual transfer, whereas student rollouts specifically improve the execution of multi-step reasoning. Experiments on recent factual QA and competition mathematics show a similar asymmetry under reverse-KL OPD, yielding notable reasoning gains without factual memory expansion. Together, these results demonstrate that on-policy distillation does not expand a model's parametric knowledge, but instead teaches it to organize and compose the knowledge it already possesses.

1. Introduction

The paper asks whether on-policy distillation (OPD) transfers factual knowledge, compositional skill, or both, because benchmark gains can conflate fact recall with coordination of intermediate results. It independently controls student capabilities and teacher additions, finding a consistent transfer asymmetry under reverse-KL OPD.

1. Introduction

  • Factual knowledge and compositional skill are distinct demands: one relies on stored associations, while the other coordinates intermediate steps.End-to-end scores cannot separate these capabilities; a model may know the facts but fail to compose them, or reason well but fail on unfamiliar facts.Source: §1 · p. 1
  • The study independently controls the student’s initial capabilities and the teacher’s added facts or compositional skill to test what OPD transfers.The synthetic setup compares teachers trained with held-out facts, branching trees over initial facts, or both against the unchanged student.Source: §1 · p. 2
  • T_s yields 79.10% greedy accuracy on unseen trees versus 0.72% under T∅, while T_f yields 13.12% held-out single-step accuracy and 0.54% on held-out chains.Across four models from three families, reverse-KL OPD transfers compositional skill much more reliably than factual knowledge.Source: §1 · p. 2
  • Reverse-KL OPD is better suited to strengthening compositional reasoning than updating factual knowledge; forward-KL objectives are more effective for factual transfer.The paper frames the distinction as using existing knowledge versus adding facts to parametric memory.Source: §1 · p. 2

2. Related Work

Prior work contrasts fixed teacher-output training with OPD on student-generated trajectories, while reporting mixed evidence about what students acquire. This paper uses controlled synthetic tasks to separate factual and compositional transfer.

2. Related Work

  • Sequence-level distillation trains on teacher-generated sequences, whereas OPD evaluates teacher supervision on student-generated trajectories to address exposure bias.Generalized Knowledge Distillation spans divergence objectives, and MiniLLM uses reverse KL.Source: §2 · pp. 2–3
  • Prior OPD studies report capability expansion and factual acquisition, while others attribute reasoning gains to more efficient sampling of existing capabilities.The studies use different sources of teacher information and measures of capability, motivating independent control of added knowledge and skill.Source: §2 · p. 3
  • Earlier synthetic tasks study memorization, function composition, and transfer of memorized associations; this paper varies teacher knowledge and skill independently.Random lookup tables represent facts, while chains and branching trees specify how those facts are combined.Source: §2 · p. 3

3. Experimental Setting: Separating Knowledge and Skill

The experimental setting separates arbitrary facts from the structures used to compose them, then controls the student’s starting abilities and the teacher’s added capabilities. Distillation and evaluation test factual recall and generalization to branching-tree structures separately.

3.1. Synthetic Tasks for Knowledge and Reasoning

  • Random lookup tables encode facts, while chains and branching trees test how facts are combined; trees require retaining intermediate results across branches.The structures use the same facts and number of operations, isolating their organization and reuse.Source: §3.1 · p. 4

3.2. Student Initialization and Teacher Design

  • The initial student learns initial facts and sequential chains, while specialized teachers add held-out facts, branching trees, or both.The frozen initial student is the baseline teacher; the teacher design crosses exposure to held-out facts with exposure to branching trees.Source: §3.2 · pp. 4–5

3.3. On-Policy Distillation and Evaluation

  • Reverse-KL distillation trains on student rollouts, with the teacher’s next-token distribution conditioned on the student’s preceding tokens.Factual transfer is tested using chains containing held-out facts; compositional transfer is tested on trees over initial facts, including shared, teacher-only, and unseen structures.Source: §3.3 · pp. 5–6
  • Evaluation separates held-out-fact recall from tree reasoning over initial facts, and tests whether tree skill generalizes beyond training structures.Figure 2 reports greedy final-answer accuracy and Pass@32 as percentages, averaged across four models.Source: §3.3 · p. 6, Figure 2 · p. 6

4. Synthetic Experiment Results

Reverse-KL on-policy distillation transfers compositional skill to unseen tree structures but transfers little factual knowledge. Separating prefix source from KL direction shows that forward KL improves factual transfer, while student rollouts strengthen multi-step execution.

4.1. On-Policy Distillation Teaches New Skills but Not New Knowledge

  • 79.10% greedy accuracy on unseen tree structures after distillation from T_s, versus 0.72% under T_∅, shows compositional skill generalizes beyond training structures.Both tests use facts learned during student initialization.Source: §4.1 · p. 6
  • 13.12% mean greedy accuracy on single-step held-out-fact queries falls to 0.54% on chains, showing factual transfer is limited and varies across models.Gemma-2-2B reaches 32.95% on single-step queries.Source: §4.1 · p. 6
  • 88.11% mean Pass@32 on unseen trees contrasts with 15.74% on single-step held-out-fact queries and 2.58% on held-out-fact chains.The joint T_fs condition shows the same disparity: robust compositional skill but unreliable recall or chaining of new facts.Source: §4.1 · pp. 6–7

4.2. Distilling the Distillation Recipe

  • Forward KL substantially raises single-step factual accuracy under student prefixes, from 10.95% to 54.10% in Qwen3-4B and from 32.95% to 94.10% in Gemma-2-2B.Teacher-prefix forward KL reaches similar accuracy, indicating student rollouts provide no distinct advantage for learning facts.Source: Figure 3 · p. 7, §4.2 · p. 8
  • Reverse KL weakly penalizes omitted facts when the student assigns the correct token negligible probability, whereas forward KL penalizes misses under the teacher distribution.This explains why factual transfer depends on the divergence objective rather than on-policy exploration.Source: §4.2 · p. 8
  • Student prefixes reduce reference-completion NLL on unseen tree structures under both KL directions, with greedy accuracy improving in three of four comparisons.For Qwen3-4B, NLL falls from 2.40 to 1.77 bits under reverse KL and from 2.13 to 1.15 bits under forward KL.Source: §4.2 · p. 8

5. Real-World Validation

On real-world benchmarks, reverse-KL on-policy distillation improves mathematical reasoning but does not transfer recent factual knowledge. The evaluation contrasts a factual QA suite with held-out competition problems and compares performance before and after distillation.

5. Real-World Validation

  • Knowledge-2026 contains 809 recent factual QA pairs, while Math-2026 contains 201 competition problems held out from distillation prompts.The initial student scores 11.25% and 17.41% greedy accuracy on the respective suites.Source: §5 · p. 8
  • The knowledge teacher reaches 84.55% on Knowledge-2026, while the reasoning teacher reaches 43.78% on Math-2026.The knowledge teacher is evaluated on the same 809 factual QA pairs used for its fine-tuning.Source: §5 · pp. 8–9
  • 11.25% to 9.64% on Knowledge-2026 contrasts with 17.41% to 23.88% on Math-2026, reproducing the weak factual-transfer and stronger reasoning pattern.These are greedy accuracies before and after distillation; Table 2 reports the real-world benchmark comparison.Source: §5 · p. 9, Table 2 · p. 9
  • On-policy distillation from T_R raises greedy Math-2026 accuracy from 17.41% to 23.88%, a 6.47-percentage-point gain; acquired reasoning generalizes effectively to competition problems excluded from distillation.Distillation from T_R leaves factual recall at 10.75%, matching S_0’s initial 11.25%; distillation from T_K shifts math accuracy only to 19.40%.Source: §5 · p. 9

6. Additional Analysis

Execution traces distinguish wrong lookup values from wrong reasoning order, while parameter analyses show factual and compositional updates follow different directions and depths. Hybrid objectives improve both transfer targets, with ToDi performing best on each.

6.1. Execution Traces of Factual and Compositional Failures

  • Reverse-KL students preserve execution format but miss held-out lookup values, whereas forward KL restores lookups; on unseen trees, forward KL flattens traversal while reverse KL from T_s executes valid postorder.The traces distinguish factual failures as wrong values from compositional failures as wrong reasoning order.Source: §6.1 · p. 9, Figure 4 · p. 10

6.2. Weight Updates under Factual and Compositional Distillation

  • Factual and compositional updates occupy near-orthogonal parameter directions, with cross-task cosine similarities of 0.004–0.046, unlike 0.64–0.82 when the divergence objective changes within a task.The authors interpret the patterns as distinct parameter directions for storing facts and acquiring compositional skills.Source: §6.2 · p. 10, Table 3 · p. 11
  • In Qwen3-4B, factual updates concentrate in final layers and peak at layer 35, while compositional updates peak in early and middle layers 3 and 11.The paper relates late layers to token-level factual associations and earlier layers to routing and assembling intermediate representations.Source: §6.2 · p. 10, Table 3 · p. 11

6.3. Transfer under Alternative Distillation Objectives

  • ToDi reaches 64.3% factual recall after distillation from T_f and 86.4% on unseen tree structures after distillation from T_s, the highest accuracy on both tests.These results exceed forward KL by 10.2 and 10.0 percentage points, respectively; across objectives, T_f leaves compositional accuracy near zero and T_s leaves held-out factual recall at its initial level.Source: Table 4 · p. 11, §6.3 · p. 11

7. Conclusion

The conclusion characterizes reverse-KL OPD as stronger for organizing existing knowledge than updating factual knowledge, with forward KL a more effective alternative for factual transfer. The execution examples contrast left-to-right chains with branching postorder trees, using independently sampled lookup values.

7. Conclusion

  • Standard reverse-KL OPD transfers compositional skill strongly but factual knowledge weakly, while forward-KL supervision is more effective when factual transfer is the goal.The paper frames OPD as improving how models organize and use existing knowledge rather than updating factual knowledge.Source: §7 · pp. 11–12

A. Execution Examples

  • The chain proceeds left to right, whereas the branching tree evaluates each subtree before combining their results, requiring retrieval of the left result across intervening right-subtree steps.Both examples use the same inputs and query the same seven initial facts; their intermediate results are combined differently.Source: A. Execution Examples · p. 16

B.1. Lookup Tables and Data Construction

  • Lookup-table entries are sampled uniformly from the ten digits and represented as four-digit strings with leading zeros.The tokenizer adds 25 fact identifiers, an identity marker, and two answer delimiters; the identity marker is excluded from the initial/held-out fact partition.Source: §B.1 · p. 16

B.2. Training Corpora and Budgets

The appendix describes estimated 10-million-token budgets for initialization and teacher training, alongside 20,000-prompt distillation corpora. Budget shares differ by student initialization, teacher capability, and query type.

B.2. Training Corpora and Budgets

  • Each student-initialization or teacher-training corpus has an estimated budget of 10 million tokens, counting special tokens and non-whitespace characters.The estimate sums these units across prompts and reference completions.Source: §B.2 · p. 17
  • The S0 corpus allocates 45% of its budget to initial-fact single-step queries, 25% to chains, and 30% to groups of seven independent queries, totaling 166,365 records.The chains have two, three, five, or seven steps.Source: §B.2 · p. 17
  • Each trained teacher assigns 40% of its budget to initialization tasks; the remaining 60% covers held-out facts for T_f, initial-fact trees for T_s, or a mixture for T_fs.For T_fs, the remaining budget is divided among held-out-fact queries, trees over initial facts, and chains and trees querying held-out facts.Source: §B.2 · p. 17
  • Each distillation corpus contains 20,000 prompts, and held-out-fact steps in chains or trees are sampled in counts of one, three, or seven with probabilities 0.5, 0.3, and 0.2.Queries are excluded from student initialization and teacher training; remaining steps use initial facts.Source: §B.2 · p. 17

B.3. Tree Structures, Test Sets, and Metrics

The appendix partitions seven-internal-node tree structures into shared, teacher-only, and unseen pools, then defines held-out-fact and tree-structure tests. It reports greedy exact-answer accuracy separately from sampled Pass@32.

B.3. Tree Structures, Test Sets, and Metrics

  • Of 429 ordered full binary-tree structures with seven internal nodes, 64 belong to each of the shared and teacher-only pools, 128 are unseen, and 173 are unused.The balanced height-three tree belongs to the unseen pool; teacher-only means excluded from distillation prompts, not necessarily from earlier student training.Source: §B.3 · p. 17
  • Factual tests query held-out facts, while compositional tests use initial facts and vary the tree-structure pool; each test has 2,000 queries excluded from training.All multi-step tests use seven computational steps.Source: §B.3 · p. 17
  • Greedy accuracy is the percentage of queries whose greedily decoded answer exactly matches the four-digit reference, including leading zeros.Missing or unparseable answers count as incorrect.Source: §B.3 · p. 17
  • Pass@32 measures whether at least one of 32 independently sampled completions has the correct final answer, and does not require a fully correct execution trace.Sampling uses temperature 1.0; this metric differs from greedy final-answer accuracy.Source: §B.3 · p. 17

B.4. Student and Teacher Baselines

The baseline section contrasts strong performance on initialized facts with weak performance on shared compositional evaluation, then specifies the optimization setup used for initialization, teacher training, and OPD.

B.4. Student and Teacher Baselines

  • Table 5 reports greedy final-answer accuracy for S0 and specialized teachers across four models from three model families.The tests use exact match of the final four-digit answer.Source: §B.4 · p. 17, Table 5 · p. 18
  • Qwen3-4B S0 scores 100% on seven-step chains over initial facts, versus 0.45% greedy accuracy and 1.45% Pass@32 on Comp-Shared.The seven-step chain task was used during initialization.Source: §B.4 · p. 18

B.5. Optimization and Hyperparameters

  • Initialization and teacher training use completion cross-entropy, while OPD trains for three epochs on 20,000 prompts per condition with one student rollout per prompt visit.The OPD learning rate is 3 × 10^-6, and reverse KL is computed over the full vocabulary.Source: §B.5 · p. 18
  • All stages use AdamW with cosine scheduling and LoRA adapters on attention and feed-forward projections; embedding updates are masked to newly added token rows.LoRA uses rank 16, α = 32, and dropout 0.05.Source: §B.5 · p. 18

C. Training Dynamics

Training dynamics differ by teacher condition: skill-focused distillation drives rollout accuracy upward and reverse KL toward zero, while conditions involving held-out facts retain a gap.

C. Training Dynamics

  • Under T_s, rollout exact match rises from near zero to above 95% in all four models as reverse KL falls toward zero.Once the student produces correct executions, its trajectories coincide with the teacher’s supervision.Source: C. Training Dynamics · p. 18, Figure 5 · p. 19
  • Under T_f, rollout exact match stays at or below about 30% in every model, while reverse KL plateaus above zero.The residual gap occurs at answers depending on held-out facts, where the teacher is confident but the student distributes probability across indistinguishable candidates.Source: C. Training Dynamics · p. 18, Figure 5 · p. 19
  • The joint T_fs condition shows the same reverse-KL plateau as T_f rather than the near-zero endpoint under T_s.Figure 5 tracks rollout accuracy in its top row and reverse KL in nats in its bottom row across three epochs.Source: C. Training Dynamics · p. 18, Figure 5 · p. 19

D. Real-World Benchmarks and Teacher Profiles

The section introduces real-world factual and mathematics benchmarks with specialized teachers, then defines measurements of parameter updates relative to S0.

D. Real-World Benchmarks and Teacher Profiles

  • Knowledge-2026 tests 809 post-cutoff factual QA pairs, while Math-2026 tests 201 held-out competition problems; teachers receive factual QA or verified math supervision.The factual items postdate Qwen3-4B-Base’s pretraining cutoff.Source: D. Real-World Benchmarks and Teacher… · p. 19

E. Weight Update Measurements

  • Weight-update measurements use merged LoRA increments relative to S0, including global and module-specific norms across layers and projection types.For Qwen3-4B, layer norms are averaged across four depth quartiles.Source: E. Weight Update Measurements · pp. 19–20, Table 6 · p. 21
  • Directional similarity compares concatenated projection-update vectors from two training conditions within the same model family.The comparison is cosine similarity between the vectors.Source: E. Weight Update Measurements · p. 20, Table 6 · p. 21
  • Effective rank summarizes the singular-direction distribution of each projection increment, ranging from 1 for rank-1 updates to 16 for a uniform spectrum.The metric is the exponential Shannon entropy of the normalized spectral distribution, averaged across adapted projections.Source: E. Weight Update Measurements · p. 20, Table 6 · p. 21
  • Embedding drift measures displacement of fact-identifier tokens from S0, reported separately for held-out and sampled initial tokens.Table 6 reports mean embedding drift for held-out and initial fact identifiers.Source: E. Weight Update Measurements · p. 20, Table 6 · p. 21

Related papers

Loading 2610.09639v1…