Source-linked AI summary
Every Coin Has Two Sides: On the Dual Nature of Generalization in On-Policy Distillation of Large Language Models
Zhaoyi Li, Deyang Kong, Yuan Wei, Evan Yang, Ranran Shen, Mahardika Krisna Ihsani, Ming Yang, Wei Zhang, Chuan Hao, Jian Yang, Ran Tao, Bryan Dai, Shikun Zhang, Wei Ye, Ying Wei, Defu Lian
TL;DR
OPD’s broader generalization remains unclear because prior evaluations largely stay within one domain and near-training benchmarks. This controlled study finds that OPD transfers reasoning behavior broadly for same-origin teacher–student pairs, while multi-teacher routing produces mixture- and origin-dependent capability tradeoffs.
Problem
Prior OPD evaluations mostly use one domain and nearby benchmarks, leaving broader policy transfer insufficiently distinguished from local fitting.
Method
The study varies one generalization factor at a time across in-domain shifts, cross-domain transfer, and multi-teacher OPD.
Results
OPD transfers reasoning behavior beyond trained problems and domains mainly for same-origin pairs, while multi-teacher combinations produce capability seesaws shaped by mixture and origin.
Takeaways & Limitations
Prompt routing cannot isolate domain experts’ influence, so diagnosing multi-teacher OPD requires accounting for cross-domain transfer and teacher–student origin.
Takeaways & Limitations
The experiments focus on reasoning-oriented models in four domains, so the observed patterns may not extend directly to multimodal, tool-using, or interactive-agent settings.
Abstract
from arXiv · showhide
On-policy distillation (OPD) transfers teacher capabilities by supervising trajectories sampled from the student's own policy, yet its generalization behavior remains poorly understood, as most studies evaluate OPD on a single domain and on benchmarks close to the training data. We present a controlled study that varies one generalization factor at a time, from in-domain distribution shifts to cross-domain transfer and the multi-teacher setting. We find that OPD transfers a teacher's reasoning behavior rather than its answers to particular problems: training difficulty barely matters, and even problems the teacher never solves are useful. Transfer depends strongly on the origin relationship between teacher and student: same-origin pairs bring the student close to the teacher across languages, reasoning horizons, and even other domains, whereas cross-origin pairs mostly fit the trained distribution. This broad reach is a double-edged sword: since routing prompts to domain experts cannot confine each teacher's influence, combining them yields a mixture-dependent seesaw among their capabilities. These results clarify when OPD generalizes and offer a useful perspective for diagnosing multi-teacher OPD.
1. Introduction
This controlled study shows that OPD transfers teachers’ reasoning behavior rather than answers, remaining robust to training difficulty and several evaluation shifts. It also finds that transfer reach depends on the teacher–student origin relationship, with same-origin pairs generalizing broadly across languages, horizons, and domains.
- Study setup: OPD samples trajectories from the student’s current policy and queries the teacher on states the student actually visits.This distinguishes OPD from conventional offline distillation and reduces exposure bias between training trajectories and the student’s distribution.
- Study setup: The controlled study varies one generalization factor at a time while holding remaining conditions fixed, spanning in-domain shifts, cross-domain transfer, and multi-teacher settings.Within math, it examines training-problem difficulty, language shifts, and reasoning-horizon shifts.
- In-domain generalization: Training only on English short-horizon math still improves the student on Chinese and long-horizon math.The transferred ability therefore holds up under shifts in evaluation language and reasoning horizon.
- In-domain generalization: OPD is largely insensitive to training-problem difficulty: problems the teacher never solves are as useful as those it always solves.This suggests OPD conveys the teacher’s reasoning patterns rather than answers to particular problems.
- Cross-origin transfer: Same-origin teachers move students close to their own level across languages, reasoning horizons, and even other domains, whereas cross-origin pairs mostly fit the trained distribution.The origin relationship between teacher and student affects the reach of transferred behavior.
2. Preliminaries and Experiment Settings
The study defines OPD as token-level teacher supervision on student-generated trajectories and extends it to routed multi-teacher training. Experiments compare same- versus cross-origin teachers across in-domain shifts and cross-domain transfer in math, code, science, and instruction following.
- OPD and MOPD: OPD samples trajectories from the student policy and minimizes reverse KL using dense token-level supervision from the teacher.The student generates responses, while the teacher supervises each token conditioned on the prompt and preceding response context.
- OPD and MOPD: PG-style OPD reinforces tokens the teacher rates more probable than the student and suppresses tokens it rates less probable.The reverse KL is estimated with a sampled-token k1 approximation and optimized as a reinforcement-learning objective.
- OPD and MOPD: MOPD routes each prompt to its corresponding domain teacher while keeping the single-teacher OPD optimization procedure unchanged.The student continues sampling from its own rollouts, enabling integration of capabilities from multiple domains.
- Same/Cross-Origin OPD: The experiments distinguish same-origin teachers and students sharing a base model from cross-origin pairs derived from different base models.The study reports significantly different generalization behaviors between these two settings.
- In/Cross-Domain Generalization: Generalization is evaluated through in-domain shifts in problem difficulty, language, and reasoning horizon, plus cross-domain transfer from one task domain to another.The four studied domains are math, code, science, and instruction following; math evaluations include English, Chinese, and long-horizon distributions.
3. In-Domain Generalization
Within math, OPD generalizes broadly across training-problem difficulty and shifted evaluation distributions. Same-origin teacher–student pairs transfer more reliably across languages and reasoning horizons than cross-origin pairs, while dynamic filtering provides a small consistent gain.
- 3.1 Training-problem difficulty: Easy, hard, and randomly sampled BigMath subsets converge to nearly identical final accuracy across all teacher–student pairs.The subsets contain 25K problems each; easy has teacher pass-rate = 1 and hard has pass-rate = 0.
- 3.1 Training-problem difficulty: OPD transfers a teacher’s reasoning behavior rather than answers to particular problems, making problems the teacher never solves as useful as problems it always solves.Even grade-school problems recover most of the OPD gain.
- 3.1 Training-problem difficulty: Dynamically discarding only problems the student already solves gives a small but consistent gain for both tested teacher–student pairs.The tested pairs are Polaris-7B →DS-distill-1.5B and Light-R1-14B →DSdistill-7B; restricting training to fully solved or fully unsolved problems does not help.
- 3.2 Distribution shifts and model origin: Training only on English, short-horizon math improves performance on Chinese and long-horizon math, showing generalization across in-domain distribution shifts.Long-horizon evaluation problems are formed by composing multiple atomic math problems.
- 3.2 Distribution shifts and model origin: Same-origin teachers are much more effective than cross-origin teachers across language and horizon shifts, even when the cross-origin teacher has higher standalone performance.Same-origin students approach their teacher’s level on Chinese and long-horizon math, whereas cross-origin teachers yield smaller gains.
4. Cross-Domain Generalization and Its Implication for MOPD
Same-origin OPD transfers a teacher’s capabilities across domains, while cross-origin transfer is concentrated in the training distribution. In MOPD, this broad influence prevents domain routing from isolating teachers and creates a mixture-dependent seesaw among capabilities.
- Cross-Domain Transfer: Same-origin OPD transfers teacher capabilities beyond the training domain in both directions, including between math, code, and science.Math training improves code and science, while code, science, or instruction-following training transfers back to math.
- Origin Dependence: Cross-domain transfer depends on model origin: same-origin students approach teacher performance across domains, whereas cross-origin students generalize mainly within the trained distribution.For same-origin OPD, training-domain gaps are largely closed; cross-origin cross-domain performance remains clearly below in-domain generalization.
- Implication for MOPD: A teacher’s cross-domain influence means routing prompts to domain experts cannot confine each teacher’s effect to its assigned domain.MOPD diagnosis must consider the full teacher set and each teacher’s capabilities outside its target domain.
- MOPD Training Dynamics: During training, MOPD students can first follow the stronger teacher’s curve and later drift toward the weaker teacher’s level, revealing a shifting balance between teachers.On AIME24-Horizon-2, students track the stronger-math JustRL-only curve before moving toward the lower Nemotron-only level.
- MOPD Seesaw: Changing teacher mixture ratios pulls performance on each benchmark toward the teacher receiving the larger share, producing a mixture-dependent seesaw rather than a clean sum of expert skills.The effect is driven by mixture share rather than by which domain a teacher is assigned to teach.
5. More Discussion on Same/Cross-Origin OPD Experiments
Same-origin OPD aligns the student’s policy more broadly than cross-origin OPD, producing stronger teacher influence in multi-teacher distillation. This origin-dependent asymmetry explains why MOPD mixtures can tilt toward same-origin teachers.
- Policy alignment: For K=16, teacher–student top-K overlap starts higher with same-origin teachers than with cross-origin teachers across 1.5B and 7B students.The overlap ratio tracks how closely the teacher’s and student’s next-token distributions align during training.
- Multi-teacher influence: In MOPD, the same-origin Light-R1-7B math teacher pulls DS-distill-7B’s GPQA-Diamond science accuracy toward itself even at a balanced 1:1 mixture.The competing cross-origin teacher is Light-R1-14B, serving as the science/IF teacher.
- Policy alignment: Same-origin OPD raises teacher–student top-K overlap over training, whereas cross-origin OPD does not.This supports broad generalization through whole-policy alignment rather than merely reducing KL on the training distribution.
- Multi-teacher influence: A same-origin teacher exerts stronger influence than a cross-origin teacher, tilting the MOPD seesaw according to origin as well as mixture ratio.The asymmetry appears when same-origin and cross-origin teachers are combined.
6. Conclusion
The study finds that OPD generalizes beyond training prompts mainly for same-origin teacher–student pairs, while multi-teacher routing produces a capability seesaw rather than isolated expert composition.
- 6. Conclusion: OPD generalization was evaluated under controlled distribution changes spanning in-domain shifts and cross-domain transfer.The study varied training and evaluation distributions systematically.
- 6. Conclusion: OPD is largely insensitive to training problem difficulty and transfers a teacher’s ability beyond the training-prompt domain, mainly for same-origin pairs.The conclusion contrasts broad transfer with the stronger dependence on teacher–student origin relationships.
- 6. Conclusion: Because routing does not confine a teacher’s influence to its assigned domain, MOPD combines domain experts into a seesaw among capabilities rather than isolated expert skills.This provides a perspective for diagnosing multi-teacher OPD.
Limitations
The experiments are limited to reasoning-oriented models across four domains, so the observed generalization may not extend to multimodal, tool-using, or interactive-agent settings. The multi-teacher study also uses two complementary teachers with fixed domain-based routing, unlike larger, more heterogeneous systems with adaptive routing or non-uniform sampling.
- Scope of evaluation: The experiments cover reasoning-oriented models in Math, Code, Science, and instruction-following, leaving generalization to multimodal, tool-using, and interactive-agent settings unresolved.Such tasks may require supervision based on external observations or actions.
- Multi-teacher setting: MOPD evaluates two teachers with complementary capabilities under fixed domain-based prompt routing, whereas practical systems may use larger heterogeneous expert pools, adaptive routers, and non-uniform teacher sampling.The controlled setup makes individual teachers’ cross-domain effects easier to analyze.
Ethical Considerations … A.2. Understanding OPD mechanisms
The paper positions OPD within large-scale post-training and prior mechanism studies, then examines whether its capability transfer extends across multiple generalization factors. It also cautions that broad transfer can propagate teacher biases beyond monitored domains.
- Ethical Considerations: OPD can transfer teacher behavior beyond routed training domains, potentially propagating undesirable behaviors or biases into unmonitored areas.Prompt routing in MOPD should not be treated as a strict capability boundary.
- A.1. OPD in Large-Scale Post-Training: Conventional distillation trains on teacher-generated responses, whereas GKD and MiniLLM reduce training–inference mismatch through student-generated sequences and on-policy optimization.GKD obtains teacher feedback on student-generated sequences; MiniLLM uses reverse KL optimization and an on-policy policy-gradient estimator.
- A.1. OPD in Large-Scale Post-Training: OPD and multi-teacher variants are important components of large-scale LLM post-training for consolidating domain experts and integrating diverse abilities.Recent technical reports support their practical effectiveness and scalability, but mainly evaluate final integrated models.
- A.2. Understanding OPD mechanisms: Prior work identifies compatible teacher–student thinking patterns and genuinely novel teacher capabilities as factors supporting effective OPD.This work builds on studies of the conditions under which OPD succeeds.
- A.2. Understanding OPD mechanisms: Other studies identify unreliable guidance on student-generated prefixes, tokenizer mismatch, student–teacher mismatch, and length exploitation as OPD failure modes.These works analyze sampled-token optimization bias and variance and characterize OPD as an exploration mechanism.
- A.2. Understanding OPD mechanisms: Methods such as Early-Stopping OPD and Asymmetric OPD modify rollout or optimization procedures to mitigate local training failures.Early-Stopping OPD limits supervision to earlier positions, while Asymmetric OPD treats positive and non-positive token advantages differently.
- A.2. Understanding OPD mechanisms: This paper asks whether OPD transfers across difficulty, language, reasoning horizon, task domain, and model origin.The question complements prior work focused on explaining and mitigating local OPD training failures.
A.3. Data Selection in OPD … B.3. Training Configuration
The paper controls OPD data selection and studies how teacher capabilities interact across domains and model origins. Its experiments specify datasets, evaluation benchmarks, lineage criteria, and standardized training settings.
- A.3. Data Selection in OPD: OPD data selection is examined through controlled teacher-side filtering and student-side dynamic sampling, including teacher-unsolved queries and student-mastered queries.These factors are compared across different model-origin settings.
- A.4. Capability Interaction in MOPD: The study evaluates whether each teacher transfers a broader capability profile beyond its nominal expert skill across primary and non-primary domains.This cross-domain perspective complements prior approaches that integrate domain-specialized teachers through routed token-level supervision.
- B.1.1. Training Datasets: The experiments cover mathematical, code, scientific, and instruction-following reasoning using domain-specific training pools.Mathematics uses Big-Math-RL-Verified with a matched 8K-query subset for absolute-difficulty comparisons; code uses approximately 24K DeepCoder-Preview-Dataset problems; science uses filtered TextbookReasoning or SCP-116K data; and the science/IF pool splits evenly between science and instruction-following examples.
- B.1.2. Evaluation Benchmarks: Evaluation spans English, Chinese, and long-horizon mathematical reasoning, code generation, scientific reasoning, and instruction following.The benchmarks include AMC 2023, MATH-500, AIME 2025/2026, BeyondAIME, OlymMATH-Hard, OlymMATH-ZH, LiveMathBench-ZH, R-HORIZON, LiveCodeBench v5, GPQA-Diamond, and IFEval.
- B.2.1. Model Lineage: Model lineage distinguishes same-origin from cross-origin OPD: models are same-origin when they share a concrete initialization checkpoint and one or both receive further post-training.DeepSeek-R1-Distill-Qwen-1.5B is the lineage root for JustRL-DeepSeek-1.5B, Nemotron-Research-Reasoning-Qwen-1.5B, and Dev-1.5B.
- B. Experiment Settings: The study uses model lineage to organize experiments involving post-training relationships among the evaluated models.Table 3 summarizes the post-training lineages, while Table 4 summarizes maximum sequence lengths for OPD rollout and evaluation.
- B.3. Training Configuration: Most configurations converge within 100–200 steps, so each run allows at most 200 steps with a prompt batch size of 128.This gives a maximum training budget of 200×128 = 25.6K prompt instances per run.
- B.3. Training Configuration: Each prompt produces four independent on-policy responses, with temperature 1.0, top-𝑝1.0, and unrestricted top-𝑘sampling implemented as top-𝑘= −1.Maximum sequence length varies with the teacher–student configuration and is summarized in Table 4.
B.4. Evaluation Configuration … C.2. Additional Generalization Results across Model Origin
The evaluation uses independently sampled responses scored with Avg@K and standardized answer extraction, while additional experiments show that OPD remains effective across broad training difficulties and model-origin configurations. Same-origin training transfers across distributions, and cross-origin training improves several mathematical domains.
- B.4. Evaluation Configuration: Evaluation samples responses at temperature 1.0 with top-p 0.95 and top-k = −1, reporting Avg@K over independently sampled generations.Avg@K evaluates every sampled response independently rather than selecting the best of K generations.
- B.4. Evaluation Configuration: Visualization curves use centered moving averages: the first point is unsmoothed, the second averages three checkpoints, and later points use five-point windows.Raw training trajectories extend beyond the final plotted step, ensuring complete windows from the third displayed point onward.
- B.4.1. Answer Extraction Instruction: Mathematical and scientific benchmarks extract the final answer from \boxed{} and use Math-Verify3 to parse and compare it with the reference.GPQA-Diamond expects a boxed option A, B, C, or D, while math tasks generally expect boxed numbers or mathematical expressions.
- C.1. Additional Results on the Training Data Difficulty: OPD produces substantial improvements from both grade-school GSM8K and highly challenging DeepMath-103K training queries, whose final average scores differ by fewer than two points.This indicates that effective on-policy supervision does not require a narrow absolute difficulty range.
- C. Additional Experiment Results: Across English Math, Chinese Math, Long-Horizon Math, and Code, Figure 9 evaluates in-domain distribution shifts and cross-domain transfer for three teacher–student configurations.The figure jointly examines these benchmarks across additional model-origin settings.
- C.2. Additional Generalization Results across Model Origin: Same-origin Polaris-4B → Qwen3-4B training on either Big-Math-RL-Verified or Polaris-53K yields stable improvements and converges to similar performance across evaluated benchmarks.Effective OPD therefore does not require access to the teacher’s original post-training data.
- C.2. Additional Generalization Results across Model Origin: Cross-origin Qwen3-32B → Qwen3-8B-SFT training on Big-Math-RL-Verified clearly improves English Math, Chinese Math, and Long-Horizon Math.Qwen3-8B-SFT starts from Qwen3-8B-Base and was supervised fine-tuned on OpenThoughts3-1.2M before OPD.
C.3. Complete Cross-domain Transfer Results
OPD transfers mathematical reasoning across domains: Code- or Science-supervised runs improve math evaluations without Math queries, while teacher–student origin often matters more than the nominal training domain. Instruction-following prompts can likewise carry capabilities absent from the training domain.
- Cross-domain transfer: OPD on Code or Science queries generally improves English, Chinese, and Long-Horizon Math reasoning without using Math queries.These results cover both DS-distill-1.5B and DS-distill-7B students.
- Teacher origin: Same-origin teachers typically yield stronger and more stable gains across the three evaluation distributions.Within a teacher lineage, Math-, Code-, and Science-supervised runs differ comparatively little.
- Teacher origin: Changing teacher origin produces a larger performance gap than changing the nominal domain of the training queries.This indicates that the teacher–student post-training relationship can outweigh the nominal supervision domain for cross-domain transfer.
- Instruction-following transfer: Instruction-following prompts can carry teacher capabilities that are not explicitly represented by the nominal training domain.The study evaluates Nemotron-1.5B →DS-distill-1.5B and Polaris-7B →DS-distill-7B on English Math, Chinese Math, and additional reported evaluations.
C.4. Full Mathematical Results for MOPD
MOPD’s mathematical results show that mixtures with more JustRL supervision generally perform better, across both teacher–domain assignments. The strongest reported configuration reaches 19.6%, while the reversed assignment reaches 17.5% for JustRL-only supervision.
- First teacher–domain assignment: At training step 200, JustRL-dominant mixtures generally maintain stronger mathematical performance than balanced or Nemotron-dominant mixtures.The first assignment uses JustRL for Math and Nemotron for science/IF supervision.
- First teacher–domain assignment: 19.6%: The 25/8 configuration achieves the highest average mathematical score, slightly exceeding both JustRL-only and balanced 1/1 configurations.These results are reported across five mathematical reasoning benchmarks.
- Reversed teacher–domain assignment: 17.5%: Under the reversed assignment, JustRL-only supervision reaches a higher average mathematical score than both balanced and Nemotron-only configurations at 15.8%.Here, Nemotron supervises Math queries and JustRL supervises science/IF queries, yet increasing JustRL supervision still improves mathematical performance.