PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Free On-Policy Distillation: How Far Can We Go Without External Data?

Gengsheng Li, Mao Zheng, Mingyang Song, Jie Sun, Zeyuan Liu, Ruiqi Liu, Tianyu Yang, Qiyong Zhong, Haiyun Guo, Junfeng Fang, Shiming Xiang, Jinqiao Wang, Tat-Seng Chua

arXiv:2609.14193v4cs.LGcs.AI

TL;DR

The paper asks how much OPD depends on external training questions and whether that dependence can be reduced. It studies fewer real questions and teacher-generated questions, finding comparable performance with generated pools while empty prompts work only in limited cases.

  • Problem

    Prior work emphasized OPD algorithms, while how much OPD depends on training questions and how far that dependence can be reduced remained unclear.

    Source: §1 · p. 1

  • Method

    DF-OPD uses domain and topic instructions to have the teacher generate a fixed question pool without seed examples, then distills student responses with the standard OPD objective.

    Source: §4.1 · p. 6

  • Results

    64 generated questions yield performance comparable to full-data single-teacher OPD, while 1k generated questions score 31.31 versus 31.18 using approximately 7k real examples across three domains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings invite a reassessment of the role of training data in on-policy distillation.

    Source: §1 · p. 2

    Experiments cover limited teacher–student combinations, model scales, task domains, and budgets; question efficiency does not imply computational efficiency, as OPD still requires repeated rollouts and teacher feedback.

    Source: §7 · p. 10

Abstract

from arXiv · show

On-policy distillation (OPD) is increasingly applied to frontier foundation model post-training. Prior work in this area has largely focused on algorithmic advances, yet it remains unclear how much OPD depends on its training questions and, in particular, how far this dependence can be reduced. Across two representative single-teacher OPD settings, we find that training on 8 real prompts yields performance comparable to training on 17k problems, while datasets differing substantially in measured difficulty and initial distillation gap yield similar outcomes. Our analyses suggest two complementary explanations: repeated sampling could allow even a few prompts to expose substantial teacher supervision, while OPD transfers generalizable reasoning capabilities beyond dataset-specific knowledge. Building on these observations, we next propose a data-free on-policy distillation (DF-OPD) setting to investigate whether the system can supply the training questions itself, eliminating the need for external data. With 64 self-generated questions obtained without seed examples, DF-OPD yields performance comparable to full-data OPD in both single-teacher settings. This finding also holds in multi-teacher OPD: across mathematics, code, and instruction following, 1k generated questions achieve performance comparable to training on approximately 7k real post-training examples. We further explore whether OPD can operate even without explicit training questions. The experiments show that this is effective only in limited cases, where the student unexpectedly generates and answers its own questions, thereby reducing the process to an implicit form of DF-OPD. Together, these findings invite a reassessment of the role of training data in on-policy distillation. Code is available at https://github.com/Ryuki661/DF-OPD

1 INTRODUCTION

The introduction asks how much OPD depends on external training questions and tests progressively reducing that dependence. It reports comparable performance with few real questions and with teacher-generated questions, while empty prompts work only in limited cases.

1 INTRODUCTION

  • OPD combines student-generated trajectories with dense, token-level teacher feedback, addressing limitations of SFT and sparse-reward RLVR.SFT trains on reference prefixes, while RLVR learns from model-generated trajectories but typically uses sparse outcome-level rewards.Source: §1 · p. 1
  • Across two single-teacher settings, 8 questions yield performance comparable to approximately 17k problems, and datasets with different difficulty and initial distillation gaps produce similar outcomes.Repeated sampling can expose substantial teacher supervision; programming-question training recovers over 90% of in-domain gains on mathematics benchmarks.Source: §1 · p. 2
  • DF-OPD replaces external questions with teacher-generated questions created without seed examples, achieving performance comparable to full-data OPD.The introduction reports single-teacher results with 64 generated questions and multi-teacher results with 1k generated questions.Source: §1 · p. 2
  • Empty-prompt OPD works only in limited settings, when the student spontaneously poses and answers a question, making the process implicit DF-OPD.The introduction links effectiveness to student Self-QA behavior and notes that this reliance limits generality.Source: §1 · p. 2

2 PRELIMINARIES AND EXPERIMENTAL SETUP

The paper defines OPD as teacher supervision on student-sampled trajectories and describes the models, datasets, and benchmark protocols used to evaluate it.

2.1 ON-POLICY DISTILLATION

  • OPD applies teacher supervision to student-generated trajectories; its question supplies visited states for supervision rather than a fixed reference answer.The method uses a top-k surrogate, while MOPD routes each question to its domain-specific teacher.Source: §2.1 · p. 3

2.2 EXPERIMENTAL SETUP

  • Experiments use two single-teacher model pairings and a three-teacher MOPD setting, with external-question datasets and benchmark-specific evaluation protocols.Single-teacher experiments evaluate six mathematics benchmarks; MOPD evaluates mathematics, code, and instruction following, reporting peak macro averages across benchmarks.Source: §2.2 · p. 3

3 HOW SENSITIVE IS OPD TO ITS TRAINING DATA?

Across two single-teacher settings, few training questions and datasets with different measured difficulty and initial distillation gaps yield comparable outcomes. Analyses point to repeated-sampling signal coverage and cross-domain transfer as complementary explanations.

3.1 SENSITIVITY TO THE NUMBER OF TRAINING QUESTIONS

  • 56.3 versus 56.4 in the RL-tuned teacher setting and 47.7 versus 47.4 in the larger teacher setting show comparable performance with 8 versus 17k questions.The training curves use matched update and trajectory budgets; Figure 2 reports six-benchmark macro accuracy and peak scores.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2 SENSITIVITY TO DATA DIFFICULTY AND INITIAL DISTILLATION GAP

  • Despite initial correctness of 46.1%, 60.7%, and 68.9% and gaps of 0.268, 0.093, and 0.132, sources yield 57.4–58.3% and 48.0–49.7% peaks across settings.The diagnostics are for DAPO, DeepMath, and ORZ, respectively; the peak ranges correspond to the RL-tuned and larger teacher settings.Source: Figure 3 · p. 4, §3.2 · p. 5

3.3 ANALYSIS

  • At 4,096 trajectories, coverage rises from 0.114 with one question to 0.507; expanding the pool from 1 to 8 adds 0.141, while 8 to 1,024 adds about 0.08.This pattern suggests repeated sampling exposes substantial supervision from few questions, with diminishing coverage gains from enlarging the pool.Source: §3.3 · p. 5, Figure 4 · p. 5
  • Training on programming questions retains 96% and 92% of DAPO’s gains over the initial students in the RL-tuned and larger teacher settings, respectively.Peak mathematics accuracy is 0.44 and 1.11 percentage points below DAPO in those settings, suggesting transfer beyond dataset-specific knowledge.Source: §3.3 · pp. 5–6, Figure 4 · p. 5

4 CAN SELF-GENERATED QUESTIONS REPLACE EXTERNAL DATA?

The section introduces DF-OPD, where a teacher generates questions without seed examples and then supervises student responses. Generated questions support performance comparable to real-data OPD in both single- and multi-teacher settings, though the multi-teacher setting benefits more from larger pools.

4.1 DATA-FREE ON-POLICY DISTILLATION

  • DF-OPD replaces external questions with a fixed teacher-generated pool, after which the student samples responses and the frozen teacher provides token-level feedback on student-generated prefixes.Only question extraction, format checks, and deduplication are applied; questions are not filtered for correctness, difficulty, or solution quality.Source: §4.1 · p. 6, Figure 5 · p. 6
  • 64 generated questions yield performance comparable to full-data OPD in both single-teacher settings: 56.61 versus 56.41 and 49.16 versus 47.36.The question pools are generated from simple instructions without external seed examples; the standard OPD objective is unchanged.Source: §4.1 · p. 6, Figure 6 · p. 7

4.2 EXTENDING DF-OPD TO MULTIPLE TEACHERS

  • The multi-teacher setup has mathematics, code, and instruction-following teachers generate domain-specific questions and supervise the student’s trajectories.The real-data MOPD baseline uses 7,458 questions; generated pools preserve the real-data domain proportions as closely as possible.Source: §4.2 · p. 7
  • 1k generated questions achieve an overall score of 31.31 versus 31.18 for real-data MOPD, with recovery rates of 98.5% versus 96.6%.Self-generated questions score slightly higher in mathematics and instruction following, and slightly lower in code.Source: §4.2 · pp. 7–8, Table 1 · p. 7
  • MOPD recovery rises from 75.6% with 8 generated questions to 98.5% with 1k, while single-teacher DF-OPD matches its full-data baseline with 64.The authors suggest that dividing questions among three domains and possible cross-domain interference may increase coverage demands.Source: §4.2 · p. 8
  • Removing explicit topic guidance changes scores by −0.60 to +1.50 percentage points across the two single-teacher settings and MOPD.The study also reports that 64 generated questions support effective single-teacher distillation from each of three domain experts.Source: §4.2 · p. 8

5 CAN OPD WORK WITHOUT EXPLICIT TRAINING QUESTIONS?

The section tests OPD with empty prompts, removing explicit question content from student inputs. This works well in the RL-tuned teacher setting but less consistently elsewhere, and the stronger outcomes coincide with student self-questioning and answering.

5.1 DISTILLATION WITH EMPTY PROMPTS

  • Empty-prompt OPD reaches 56.00 in the RL-tuned teacher setting, retaining approximately 95.6% of full-data OPD’s gain, but retains only approximately 7.4% and 26.7% in the larger-teacher and multi-teacher settings.The corresponding scores in those settings are 35.44 and 26.07.Source: §5.1 · p. 8, Figure 7 · p. 9

5.2 EMPTY-PROMPT ROLLOUTS AS IMPLICIT DF-OPD

  • Without a user question, some students spontaneously pose a question and attempt to answer it within the same rollout, a behavior termed Self-QA.In these rollouts, the student’s question supplies task content for the subsequent solution attempt and elicits teacher supervision, making the process an implicit form of DF-OPD.Source: §5.2 · pp. 8–9
  • Across students, Self-QA rates of 62.0%, 11.8%, and 0.1% coincide with retention of 95.6%, 26.7%, and 7.4% of full-data gains, respectively.Rates were measured over 1,024 empty-prompt rollouts from each initial student.Source: §5.2 · p. 8
  • Teacher-generated questions allow control over question counts, domains, and their mixture, whereas empty-prompt performance varies across the settings studied.DF-OPD does not rely on spontaneous student behavior to supply questions.Source: §5.2 · p. 9

6 RELATED WORK

Prior OPD work has studied supervision, optimization, token importance, and data-centric strategies, while concurrent studies report limited sensitivity to training data. This paper extends that inquiry across data dimensions and investigates self-generated questions and question-free OPD.

6 RELATED WORK

  • Prior OPD research examines supervision and optimization, token importance, alignment over high-probability token supports, difficulty-aware balancing, and reliability-aware prompt ordering.The passage cites Song & Zheng (2026), Xu et al. (2026b), Li et al. (2026b), Hou et al. (2026a), and Zhu et al. (2026) for these lines of work.Source: §6 · p. 9
  • Concurrent studies find limited sensitivity to training data, but the scope and limits of that insensitivity remain underexplored.The passage cites Fu et al. (2026b), Hou et al. (2026b), and Li et al. (2026c).Source: §6 · p. 9
  • This work examines OPD’s sensitivity across data quantity, dataset-level difficulty, and initial distillation gap, then studies self-generated questions and empty-prompt distillation.It also relates its focus to small-dataset alignment and reasoning through SFT and RL.Source: §6 · p. 9

7 CONCLUSION

The conclusion reports that OPD remains effective with few or self-generated questions, while question-free performance varies by setting. The paper also describes its OPD objective, model configurations, and training datasets.

7 CONCLUSION

  • Across two single-teacher settings, few questions and datasets with different difficulty and initial distillation gaps yield comparable outcomes; self-generated questions support single- and multi-teacher distillation without external question corpora.Omitting explicit questions is less consistent and can coincide with student self-questioning; the experiments cover a limited range of models, scales, domains, and budgets, and still require repeated rollouts and teacher feedback.Source: §7 · pp. 9–10

A OPD TRAINING OBJECTIVE

  • The OPD training objective uses a stop-gradient surrogate over the student’s top-k support to provide token-level supervision at each response state.During updates, support sets, sampled trajectories, and bracketed weights are held fixed while gradients flow through the student’s log-probabilities.Source: A OPD TRAINING OBJECTIVE · p. 14

B.1 MODELS

  • The experiments use five teacher–student relationships across two single-teacher settings and a three-domain multi-teacher setting, including publicly released checkpoints.The larger Qwen3 teacher has approximately 30B total parameters and 3B activated parameters; the three MOPD teachers are domain-specific RL variants of MixSFT.Source: §B.1 · pp. 14–15, Table 2 · p. 15

B.2 DATASETS

  • The multi-teacher real-data pool contains 7,458 examples: 1,540 mathematics, 1,870 code, and 4,048 instruction-following records.The single-teacher datasets include DAPO-Math-17k with 17,917 records, and the MOPD questions include domain labels for teacher routing.Source: §B.2 · p. 15

B.3 TRAINING AND EVALUATION PROTOCOL

The section specifies training and evaluation protocols, defines diagnostics for initial distillation gaps and signal coverage, and points to benchmark-level training curves.

B.3 TRAINING AND EVALUATION PROTOCOL

  • Single-teacher OPD uses student support size k = 16, while MOPD uses k = 128; evaluation uses distinct sampling and response-count settings across domains.Checkpoints are evaluated periodically, and each experiment reports its highest macro-average score; MOPD domain scores come from the checkpoint with the highest Overall score.Source: §B.3 · pp. 15–16

C.1 INITIAL DISTILLATION GAP

  • The initial distillation gap measures teacher–student prediction discrepancy on trajectories from the initial student, averaged across valid response tokens.The metric compares different data sources under a given teacher–student setting, with the student fixed at initialization.Source: §C.1 · pp. 16–17

C.2 SIGNAL COVERAGE

  • Signal coverage measures the fraction of held-out reference signal energy retained by projecting onto the span of sampled-state supervision signals.The unit signal feature combines the student state representation with the correction direction associated with teacher–student discrepancy; coverage is assessed as question-pool size and repeated sampling vary.Source: §C.2 · pp. 17–18

D.1 PER-BENCHMARK RESULTS

  • Figure 8 provides complete training curves for each benchmark in the RL-tuned teacher setting, comparing DAPO question-pool sizes with full-data training to show benchmark-level sensitivity to training-question count.Source: §D.1 · p. 18

D.2 DATA SENSITIVITY ON ADDITIONAL DATASETS

Additional-dataset experiments test whether few-question OPD results extend beyond DAPO, while appendices describe self-generated question processing and benchmark-level curves.

D.2 DATA SENSITIVITY ON ADDITIONAL DATASETS

  • In both teacher settings, training on 64 DeepMath or ORZ questions yields peak macro-average mathematics accuracy close to 1k-question training; even one question produces performance gains.Source: §D.2 · p. 18

E.1 QUESTION GENERATION AND DATA PROCESSING

  • Teachers generate question pools without external seed examples or retrieved content, then filter invalid, short, instruction-restating, duplicate, and overlong outputs.Each teacher receives 1,024 requests, samples four outputs per request, and uses temperature 1.0 and top-p = 0.95; processing does not verify correctness or select by difficulty or solution quality.Source: §E.1 · pp. 20–21

E.2 EXAMPLES OF GENERATED QUESTIONS

  • Tables 7–11 show five retained generated-question examples per teacher alongside their generation topics.The examples underwent the extraction and processing steps described for question generation.Source: §E.2 · p. 22, Table 7 · p. 22, Table 8 · p. 22

E.3 DETAILED RESULTS IN SINGLE-TEACHER SETTINGS

  • Figures 11 and 12 provide per-benchmark training curves for the two single-teacher settings, comparing generated-question pool sizes with full DAPO training.Each figure covers six mathematics benchmarks under unchanged training and evaluation protocols.Source: §E.3 · p. 22

E.4 DETAILED RESULTS IN THE MULTI-TEACHER SETTING

The supplementary results compare generated-question MOPD with real-data training, test domain experts separately, and assess sensitivity to topic guidance.

E.4 DETAILED RESULTS IN THE MULTI-TEACHER SETTING

  • 31.31 Overall with 1k generated questions is comparable to 31.18 with real-data training on 7,458 questions.Figure 13 compares domain-average curves, and Figure 14 shows benchmark-level curves.Source: §E.4 · p. 23, Figure 13 · p. 26

E.5 SINGLE-TEACHER DISTILLATION WITH DOMAIN EXPERTS

  • With 64 generated questions, domain scores were 23.12 in mathematics, 22.05 in code, and 47.77 in instruction following, versus real-data scores of 24.27, 21.88, and 49.52.Relative to the initial student, generated questions improved the three scores by 7.29, 6.28, and 6.76 points, respectively.Source: §E.5 · p. 25, Figure 15 · p. 27

E.6 SENSITIVITY OF DF-OPD TO TOPIC GUIDANCE

  • Across ten comparisons, removing topic guidance changed scores by −0.60 to +1.50 points; only the largest increase exceeded 1 point, and DF-OPD remained effective in the studied settings.For 64 questions, the two single-teacher score changes were +0.02 and +0.29 points; for 1k MOPD questions, the change was −0.52 points.Source: Table 12 · p. 28, §E.6 · p. 28

F EMPTY-PROMPT DISTILLATION

Empty-prompt OPD can take the form of implicit DF-OPD when students generate and solve questions within their rollouts, but its effectiveness is inconsistent across settings. The section describes how empty inputs and Self-QA behavior were constructed and measured, and illustrates contrasting student responses.

F.1 INPUT TEMPLATES AND TRAINING SETUP

  • All settings retained an empty user message and generation prefix; only MOPD retained an empty system message, and its domain proportions matched the full-data setting.The same empty-input template was used for training samples within each setting; other training and evaluation parameters remained unchanged.Source: §F.1 · p. 28, Figure 16 · p. 29

F.2 SELF-QA MEASUREMENT

  • The Self-QA measurement sampled 1,024 pre-training responses per student at temperature 1.0, with a 4,096-token maximum.Generation stopped at EOS or the length limit.Source: §F.2 · p. 28
  • Self-QA classification required question-generation, solution-attempt, and domain-content cues, while excluding requests for a user question and refusals.The reported Self-QA rate is the fraction of sampled responses satisfying the complete rule; it does not measure answer correctness.Source: §F.2 · p. 29, Table 13 · p. 29
  • Empty-prompt OPD acts as implicit DF-OPD when a student generates a question and attempts to answer it, but the behavior varies across students and results are not consistent across settings.The question supplies task content for the solution attempt, while the teacher supervises the student-generated trajectory.Source: §F.2 · p. 29

F.3 REPRESENTATIVE ROLLOUTS

  • Representative rollouts contrast DeepSeek-R1-Distill-Qwen-1.5B generating and solving a mathematics question with Qwen3-1.7B asking the user for one.The examples are illustrative; the Self-QA frequencies are measured over the full sample sets, and the example solutions are not verified for correctness.Source: §F.3 · pp. 29–30, Table 14 · p. 30, Table 15 · p. 31

G LIMITATIONS AND FUTURE WORK

The study covers a limited range of teacher–student combinations, scales, domains, and training budgets, and reducing distinct questions does not necessarily reduce computational cost. Related work includes synthetic instruction generation and multi-teacher distillation.

G LIMITATIONS AND FUTURE WORK

  • The experiments cover a limited range of teacher–student combinations, model scales, task domains, and training budgets, so broader and longer evaluations are needed to establish applicability.The number of questions needed may vary with the teacher–student relationship, target capabilities, and training horizon.Source: G LIMITATIONS AND FUTURE WORK · p. 31
  • Reducing distinct questions does not necessarily reduce computation because the experiments kept trajectories per update and total updates unchanged, while retaining repeated student rollouts and teacher feedback.Related work includes synthetic instruction generation from seed examples, template-elicited instruction–response pairs, and self-play task generation; MOPD combines domain-associated questions with domain experts generating questions and routing labels.Source: G LIMITATIONS AND FUTURE WORK · p. 31, H ADDITIONAL RELATED WORK · p. 31
Loading 2609.14193v4…