Source-linked AI summary
What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection
Zhinan Hou, Jiaqi Zhang, Xunliang Cai, Keyou You
TL;DR
OPD’s data requirements and selection mechanisms are underexplored, motivating a study of how much data is needed and which examples drive improvement. The paper evaluates one-example distillation, analyzes example difficulty and CoT length, and proposes hard-example selection. Training on 8 selected hard examples matches the 17K full-dataset baseline, with findings spanning models from 1.5B to 7B parameters.
Problem
OPD’s data efficiency, effective training examples, and drivers of student improvement remain underexplored, despite the practical scarcity and cost of high-quality data.
Method
The paper studies 1-shot OPD, compares examples by difficulty and reasoning length, analyzes token-level changes, and proposes selecting hard examples for training.
Results
8 selected hard examples achieve 53.6% validation accuracy, matching the 53.7% full-dataset baseline, with extreme data efficiency generalizing across 1.5B–7B models.
Takeaways & Limitations
Hard examples can improve OPD even when teacher and student accuracy is 0%, because longer CoTs support teacher alignment and reasoning patterns such as self-reflection.
Takeaways & Limitations
Difficulty-driven selection is an empirical heuristic rather than a mathematically optimal training-set selection method.
Abstract
from arXiv · showhide
On-Policy Distillation (OPD) has emerged as a widely adopted post-training paradigm for enhancing large language models in reasoning domains. However, the data-centric mechanisms in OPD remain relatively underexplored. This paper presents a empirical study of data efficiency and data selection in OPD. We begin by investigating an extreme setting: training OPD on only one example, namely 1-shot OPD. Surprisingly, we find that 1-shot OPD is consistently effective across all sampled training examples and harder examples often yield superior performance gain. We next investigate what actually drives the student model's improvement in the training data. Our analysis reveals that the improvement is not driven by high token entropy, but the longer CoT paths which hard problems naturally generate. Training on longer CoT can help maintain closer alignment with the teacher over a long reasoning horizon, and learn critical thinking patterns usually missing in short CoTs, such as reflection (e.g., ``Alternatively''). Based on these insights, we propose a simple data selection method that selects only hard examples for training, where even ``unsolvable'' examples that completely exceed the teacher's capability can be successfully used. Our experiments conducted on four models ranging from 1.5B to 7B show that training the student model on only 8 selected hard examples matches the performance of the 17K dataset baseline.
1 Introduction
The paper studies the underexplored data requirements and selection mechanisms of on-policy distillation. It finds that one-example training can be effective, especially with harder examples and longer reasoning trajectories.
- Research gap: OPD data requirements and selection remain underexplored despite the method’s growing use for transferring reasoning capabilities.The paper frames data necessity, effective data, and the drivers of student improvement as practically important questions when high-quality data are scarce or expensive.
- Main findings: 1-shot OPD is consistently effective across sampled training examples.The student’s validation performance remains stable for up to 2,000 steps without overfitting or catastrophic policy collapse.
- Main findings: Harder examples often produce better performance, even when they exceed both teacher and student capabilities and have 0% training accuracy.The validation accuracy can continue improving despite the training accuracy remaining at 0%.
- Mechanism: Longer CoT paths, rather than high token entropy, drive improvement by supporting long-horizon teacher alignment and learning reflection patterns.The paper highlights self-reflection patterns such as “Alternatively” as rarely learned from short CoTs.
- Data efficiency: Training on 8 selected hard examples matches the 17K full-dataset baseline for a 1.5B student, with the pattern generalizing across 1.5B–7B models.The reported average accuracies are 53.6% for 8 examples and 53.7% for the full dataset.
2 Preliminaries
OPD trains a student to match a teacher on trajectories generated by the student itself. The paper describes token-level KL objectives and a top-k variant that focuses supervision on the student’s most probable tokens.
- On-policy supervision: The student samples response trajectories, and both student and teacher distributions are evaluated on the student-generated prefixes.At each step, the two next-token distributions are compared over the vocabulary.
- Objective: Standard OPD minimizes sequence-level reverse KL divergence over student-generated trajectories.The sequence objective is subsequently decomposed into token-level terms using autoregressive factorization.
- Supervision granularity: OPD implementations differ by whether token-level supervision uses sampled tokens, the full vocabulary, or a top-k subset.The experiments primarily use top-k OPD to reduce memory overhead while retaining dense multi-token supervision.
- Top-k OPD: Top-k OPD selects the k tokens with highest student probabilities and renormalizes both distributions over that subset before computing KL divergence.The subset is defined as S_t = TopK(p_t, k).
- Top-k OPD: The final training objective sums subset-KL losses across positions on student-generated trajectories.This combines on-policy generation with restricted token-level teacher supervision.
3 Pilot Experiments: 1-Shot OPD
The pilot experiments test whether OPD can align a student using only one example and examine how example difficulty, training dynamics, and reasoning-token changes relate to performance. The setup ranks examples by joint student–teacher accuracy and evaluates validation accuracy alongside policy alignment.
- 3 Pilot Experiments: 1-Shot OPD: 1-shot OPD trains on a single example to test policy alignment without data variety and guide later data-efficiency analyses.The pilot setting is explicitly designed as a clean baseline for studying suitable data and sufficient training-set size.
- 3.1 Experimental Setup: Examples are ranked by the average of student and teacher rollout accuracies, with deterministic tie-breaking by dataset index.The pool contains 1,000 sampled DAPO-Math-17K examples.
- 3.1 Experimental Setup: The study categorizes examples as Easy, Medium, or Hard according to the ranked average accuracy.The supplied setup defines thresholds of A_i > 0.9 for Easy and A_i < 0.1 for Hard.
- 3.1 Experimental Setup: Training uses top-16 per-token reverse KL supervision, eight responses per prompt, and a maximum response length of 7,168 tokens.The experiments use the verl framework with batch size and mini-batch size 64.
- 3.2 1-Shot OPD is Effective for Many Examples: Validation tracks average accuracy on AMC 2023, AIME 2024, and AIME 2025, while overlap ratio measures student–teacher policy alignment.Figure 1 visualizes these trajectories for two examples at each difficulty level against the full-set baseline.
- 3.2 1-Shot OPD is Effective for Many Examples: 1-shot OPD validation performance plateaus after 300 steps and remains stable through 2,000 steps without decay or policy collapse.The overlap ratio continuously increases, indicating progressive alignment with the teacher.
- 3.2 1-Shot OPD is Effective for Many Examples: The best one-shot hard example reaches 51.7% average accuracy, close to the 53.7% full-set baseline.Table 1 compares the base student, teacher, full-set distillation, and one-shot models across mathematical reasoning benchmarks.
4 Data Selection for OPD
Harder examples often improve OPD more effectively, and controlled experiments attribute this advantage to longer CoT trajectories rather than token entropy. Longer reasoning paths also help students maintain teacher alignment and acquire patterns such as self-reflection.
- Harder Problems Often Lead to Better Performance: Harder training examples often improve validation accuracy even when their training accuracy remains at 0%.Easy examples can also improve validation accuracy after reaching 100% training accuracy.
- Deeper Analysis: Length-controlled comparisons reduce the advantage of hard examples to 0.5% and −0.9%, while unrestricted hard examples outperform easier ones by 3.9% and 2.4%.These results indicate that higher token entropy alone does not explain the performance difference.
- Deeper Analysis: Longer CoT trajectories help the student sustain closer policy alignment with the teacher across extended reasoning horizons.The study evaluates token-level KL loss in 100-token position intervals.
- Deeper Analysis: Harder problems teach additional reasoning patterns, including self-reflection and backtracking, that are absent from shorter easy-problem trajectories.The token “Alternatively” is absent for the easy model, emerges for the medium model, and ranks higher for the hard model.
5 How Much Data is Sufficient for OPD?
A small number of selected hard examples can provide most of the benefit of full-set OPD. Performance improves through eight hard examples, then plateaus, and this efficiency transfers across model pairings and scales.
- Few-Shot OPD Can Match Full-Set Performance: Hard-example performance improves steadily from N = 1 to N = 8, while N = 16 and N = 64 yield no additional gains.At a fixed size of eight, hard examples outperform medium and easy examples.
- Few-Shot OPD Can Match Full-Set Performance: 8 hard examples achieve 53.6% average accuracy, nearly matching the Full-Set 17K baseline at 53.7%.Increasing the hard-example set beyond eight provides no additional gains.
- Few-Shot OPD on Other Models: Across three additional model pairings, few-shot OPD on hard examples consistently produces substantial validation gains.The pairings vary model scale, capability, and backbone architecture.
- Few-Shot OPD on Other Models: For the DeepSeek-7B student, 1-shot OPD reaches 58.4% average accuracy and 8-shot OPD reaches 59.5%, near the 59.6% full-set result.For Qwen3-1.7B-Base and Qwen3-4B-Base, 8-shot accuracies are 21.3% and 29.2%, versus full-set results of 22.5% and 30.8%.
6 Related Work
Prior work positions OPD as an on-policy alternative to off-policy distillation and explores algorithmic and practical improvements. Data selection is established in other post-training settings, but remains underexplored for OPD.
- On-Policy Distillation: OPD trains on student-generated trajectories under a reverse KL objective, mitigating the exposure bias associated with static teacher-generated data.The related-work discussion contrasts OPD with off-policy approaches such as SFT.
- On-Policy Distillation: Recent research studies OPD variants, practical training recipes, and integration into large-scale post-training pipelines.The cited examples include algorithmic and systems-oriented efforts.
- Data Selection for LLM Post-Training: Data selection research for LLM post-training has largely focused on SFT and RLVR, using quality assessment and gradient-based methods.The passage identifies several established selection directions.
- Data Selection for LLM Post-Training: Data selection for OPD remains underexplored relative to these other post-training settings.This gap motivates the paper’s focus on efficient OPD data selection.
7 Conclusions
The paper finds that OPD can achieve strong mathematical-reasoning gains with very little data, especially when selecting hard examples. Longer CoT trajectories, rather than token entropy, account for the improvement, including on tasks both models cannot solve.
- Conclusions: 1-shot OPD can substantially improve mathematical reasoning, while 8 selected examples nearly match full-dataset distillation.The conclusion summarizes the paper’s data-efficiency result.
- Conclusions: Harder tasks consistently outperform easier ones, and unsolvable tasks can still produce strong gains despite teacher and student failure.The conclusion reports this pattern across the studied setting.
- Conclusions: The improvement is driven by longer CoT trajectories rather than high token-level entropy.The paper identifies longer reasoning paths as the central explanatory factor.
A More Training Details
The OPD experiments use the default hyperparameters summarized in Table 4, including rollout, length, optimization, temperature, regularization, and loss-aggregation settings.
- Training setup: Each prompt produces n = 8 responses during OPD training.The maximum prompt length is 1,024 tokens and maximum response length is 7,168 tokens.
- Optimization: Training spans one epoch across 8 H800 80G GPUs with a learning rate of 1 × 10−6.The setup disables KL regularization and uses token-mean loss aggregation.
- Sampling: Student and teacher sampling temperatures are both set to 1.0.These are the default temperatures unless otherwise noted.
- Reference: Table 4 provides the default hyperparameters for OPD.The supplied table passage identifies the table but does not enumerate additional values.
B.1 1-shot OPD over Extended Training
Extended 1-shot OPD training remains stable: validation accuracy rises early and then stays steady through 2,000 steps, without overfitting or policy collapse. A temperature study separately varies rollout entropy on an easy example.
- Extended training: Validation accuracy rises during the first 300 steps and remains stable through 2,000 steps for easy, medium, and hard examples.The experiment uses π105, π794, and π973 as representative examples across difficulty levels.
- Extended training: The prolonged 1-shot OPD runs show no performance decay or catastrophic policy collapse.The figure caption likewise reports exceptionally stable validation performance across all three models.
- Temperature study: Figure 7 tracks rollout entropy and Average Validation Accuracy across AMC 2023, AIME 2024, and AIME 2025.The two subfigures compare entropy trajectories with corresponding validation accuracy under the different temperatures.
- Temperature study: The auxiliary temperature experiment varies student rollout temperature T over {0.6, 0.8, 1.0, 1.1} for easy problem π178.Changing T artificially increases token-level rollout entropy without changing the example’s inherent difficulty.
C Limitations
The paper identifies practical and scope limitations: difficulty-based selection is empirical rather than mathematically optimal, while longer CoT can encounter exposure bias and compounded errors.
- Data selection: Difficulty-driven data selection is a simple empirical choice rather than a mathematically optimal training-set selection method.The authors note that finding theoretically optimal OPD training sets may be computationally intractable.
- Verifier-free domains: In verifier-free domains, CoT length is proposed as a practical proxy for difficulty-based selection.Longer CoTs are linked in the paper to sustained alignment and activation of advanced reasoning patterns such as reflection.
- Rollout horizon: Excessively long on-policy rollouts can suffer severe exposure bias as student reasoning errors compound over long horizons.The resulting drift into out-of-distribution states degrades the quality of the teacher’s policy supervision.
- Rollout horizon: The maximum rollout length 7,618 is an empirical choice, and advanced algorithms for longer CoTs are left for future work.The paper points to Prune-OPD as one promising direction.
E.1 Accuracy of Sampled Examples
Example difficulty is estimated from baseline rollout accuracies of both student and teacher models. The resulting average accuracy ranks examples into easy, medium, and hard groups for selection and analysis.
- Accuracy measurement: For each problem, student and teacher baseline accuracies are measured using 16 independent rollouts per model.The evaluated models are DeepSeek-R1-Distill-Qwen-1.5B and JustRL-DeepSeek-1.5B.
- Accuracy measurement: Task difficulty is represented by the average accuracy Ai = (Si + Ti)/2 of the student and teacher.Si and Ti denote the respective rollout accuracies for problem i.
- Difficulty grouping: Examples are grouped as Easy when Ai > 0.9, Medium when 0.1 ≤ Ai ≤ 0.9, and Hard when Ai < 0.1.The categories are used to rank training examples and select representative problems.