Source-linked AI summary
One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation
Justin Robert, Raheel Qader
TL;DR
RL-based reasoning faces sparse, expensive supervision and tends to concentrate probability on existing reasoning, while dense self-distillation risks collapse: progressively fewer reasoning paths. This review organizes OPSD's failure modes around signal weighting, privileged information, and teacher dynamics, synthesizing established findings and open limitations rather than reporting new experiments.
Problem
RLVR provides sparse end-of-trajectory rewards, requires many long rollouts, and tends to concentrate probability on reasoning the base model already produces.
Method
The review unifies OPSD findings by analyzing collapse through signal geometry, privileged information, and temporal teacher dynamics.
Results
Privileged information helps only when students can reconstruct it at test time; collapse may remain invisible to mean score and entropy, with pass@k revealing fewer distinct reasoning paths.
Takeaways & Limitations
The review provides a shared vocabulary for scattered findings and distinguishes established results from disputed or unresolved questions.
Takeaways & Limitations
A criterion that is both available during training and correlated with a token's actual usefulness remains to be built.
Abstract
from arXiv · showhide
On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.
Introduction
OPSD removes the need for a larger teacher by conditioning the model itself on privileged information, but its dense signal introduces unresolved risks including collapse, capability degradation, and test-time information dependence. This review focuses on mathematical reasoning and organizes those risks around signal placement, privileged information, and training dynamics.
- On-policy distillation provides token-level supervision on the student’s own rollouts, resolving credit assignment but requiring a larger teacher.The teacher scores each generated token while the student remains on-policy, at the cost of running a second model.
- OPSD and SDPO replace the larger teacher with the model itself conditioned on privileged information unavailable to the student.Examples include a reference solution for OPSD and environment feedback for SDPO.
- Combining dense distillation with on-policy reinforcement learning promises cheaper training without dependence on a larger model.The review identifies this promise alongside unresolved tensions in how the signal affects the student.
- Dense supervision can narrow reasoning diversity and entropy, degrade earlier capabilities, and encourage reliance on information unavailable at test time.These tensions motivate the review’s focus on collapse and its controlling factors.
- The review restricts its scope to mathematical reasoning and examines how the field controls OPSD’s signal through three levers.The levers concern where the signal is applied, what the teacher sees, and when guidance changes.
1 OPSD: Where It Comes From, How It Works, What It Is For
OPSD emerged by combining reinforcement learning’s on-policy sampling with distillation’s dense token-level signal, then replacing the larger teacher with a better-informed copy of the student. The method can match or exceed GRPO with fewer generated tokens, but its dense supervision risks collapse and its reported gains require careful evaluation.
- Genealogy: OPSD answers how to provide a dense, cheap learning signal without depending on a larger model.It follows a progression from SFT, RL, and on-policy distillation, each addressing a limitation of the preceding approach.
- Genealogy: SFT is dense but off-policy, whereas RL is on-policy but sparse and expensive for long rollouts.RLVR assigns one terminal reward across hundreds of generated tokens, making credit assignment difficult.
- Genealogy: On-policy distillation restores token-level density by comparing the student’s rollout distributions with a larger teacher’s distributions.Forward KL is mode-covering, while reverse KL encourages concentration on one teacher mode and can impoverish diversity.
- Mechanism: OPSD uses the model itself as teacher, conditioning it on privileged information such as a reference solution, hint, or environment feedback.The teacher scores the student’s rollout token by token rather than generating a separate rollout.
- Mechanism: A student rollout is sampled autoregressively, then teacher and student distributions are collected at each position for distillation.The founding implementation caps rollouts at 1,024 tokens, and the distributions can be computed in parallel across positions after generation.
- Strengths and weaknesses: OPSD matches or exceeds GRPO on mathematical reasoning while using far fewer generated tokens, with a reported AIME25 rise from 36.7 to 43.9 by step 50.Its optimization step is nevertheless roughly twice as costly as GRPO’s: 20.6 s versus 11.2 s on Qwen3-8B with 8×H100.
- Strengths and weaknesses: The approach transfers beyond mathematics to code and agentic tasks, where SDPO reports efficiency gains of the same order.This extends the family’s reported efficiency pattern beyond the review’s main mathematical scope.
- Strengths and weaknesses: Dense supervision can produce collapse, defined here as progressive narrowing of the reasoning paths the model can generate.The review treats collapse as a symptom rather than a lever and evaluates it through pass@k.
2 Developments Since the Founding Paper
Recent work reframes OPSD’s central failure as collapse, shaped by signal placement, privileged information, and teacher dynamics. Evidence favors selective, information-aware guidance, but key criteria and interventions remain unresolved.
- 2.1 Lever A — Signal Geometry: Which Divergence, Which Density?: Selective density is proposed as a remedy: weight tokens by importance rather than treating every token in the rollout equally.One analysis estimates that distilling only positively aligned tokens could improve the signal by a factor of ten to fifteen, but this remains an oracle measurement.
- 2.1 Lever A — Signal Geometry: Which Divergence, Which Density?: Uniform token-level guidance is most unstable on high-entropy decisions, where reasoning branches and multiple continuations remain plausible.The student’s most probable token changes 84 times on high-entropy tokens versus 7 times on low-entropy tokens, while retaining 6.8 % versus the teacher’s 18.5 % high-entropy tokens.
- 2.1 Lever A — Signal Geometry: Which Divergence, Which Density?: A hybrid divergence keeps reverse KL broadly while adding forward KL on teacher high-entropy tokens to preserve plausible continuations where uncertainty signals branching.The reverse KL supplies concentrated imitation on confident tokens; forward KL supplies mode coverage on uncertain tokens.
- 2.1 Lever A — Signal Geometry: Which Divergence, Which Density?: The field lacks a weighting criterion that is both available during training and demonstrably aligned with token usefulness, and it has not established the optimal granularity of selection.Alignment with the ideal gradient is measured after the trajectory, whereas teacher entropy is online but not guaranteed to approximate usefulness; current methods operate token by token or problem by problem.
- 2.2 The Symptom — Collapse: Two Families of Causes: Collapse narrows the reasoning paths a model can produce and can occur alongside higher token entropy but lower functional diversity.On Qwen3-8B, self-distillation raises pass@1 from 71.9 to 73.4 while pass@16 falls from 83.6 to 78.5.
- 2.3 Lever B — The Nature of the Privileged Information: Privileged information can aggravate collapse through PMI: solution-conditioned teachers amplify tokens entailed by the answer and suppress deliberation tokens needed for exploration.This makes the information given to the teacher an upstream control variable, while existing remedies intervene only after conditioning.
- 2.3 Lever B — The Nature of the Privileged Information: Privileged-information design matters: random correct demonstrations can make frequent solutions still more probable while causing rare correct solutions to disappear.The review therefore distinguishes correctness from diversity when constructing demonstrations.
3 Synthesis: Three Levers, One Symptom
The review organizes collapse around three levers: where the signal is applied, what privileged information the teacher receives, and when teacher guidance changes. Across these levers, the central unresolved problem is improving accuracy without sacrificing diversity.
- The three levers control signal placement, privileged information, and teacher dynamics, while collapse serves as the symptom used to judge their effects.Lever C has two independent mechanisms: the teacher’s exposure to privileged information and updates to its weights.
- Axis A. Where? Weighting the tokens: Uniform token weighting dilutes reasoning supervision because formatting occupies much of the gradient in long rollouts.The founding paper weighted all tokens equally, although only a small fraction of tokens contains genuinely committing reasoning decisions.
- Axis A. Where? Weighting the tokens: No token-weighting method has reached consensus because accuracy gains remain in tension with diversity loss.The review identifies this performance–diversity trade-off as the central unresolved issue for the signal’s shape.
- Axis B. What? Choosing the privileged information: Reference solutions can make the teacher an oracle, whereas hints, plans, critiques, and environment feedback may convey skills without entailing the answer.These alternatives are distinguished from reference solutions by whether the student can reconstruct the privileged information rather than simply presuppose the answer.
- Axis B. What? Choosing the privileged information: No privileged-information type has reached consensus, and adapting both its nature and quantity to context remains little explored.This axis is still unsettled despite reported gains from several forms of information.
- Axis C. When? Making the teacher dynamic: Dynamic teachers can use moving-average weights, performance-triggered refreshes, or decaying privileged information, but this timing axis remains least explored.Documented collapses may appear only after several hundred training steps, complicating evaluation with short runs.
Conclusion
The review finds that OPSD’s information asymmetry can enable self-guidance but also bias the learning signal toward unavailable shortcuts. It therefore presents OPSD as a promising but still research-stage technique whose conclusions are limited mainly to recent mathematical-reasoning studies.
- Privileged information helps only when the student can reconstruct it at test time; otherwise, it can harm performance by imposing unavailable information.The review identifies this transferability condition as an established result.
- Collapse may be hidden by mean score or entropy: only pass@k reveals when a model produces fewer distinct reasoning lines.A self-distilled model can have higher entropy than an RL-trained model while still exhibiting fewer reasoning paths.
- OPSD should not be used naively in production because poorly chosen privileged information can degrade reasoning performance, and dense self-distillation can worsen forgetting and collapse.The authors characterize OPSD as a post-SFT fine-tuning method rather than a turnkey training method.
- The review covers mathematical reasoning but excludes multimodal and agentic settings, while most cited studies use recent preprints, one model family, and models no larger than a few billion parameters.These scope and evidence constraints limit how broadly the conclusions should be read.
- The review’s qualified conclusion is that self-guidance remains viable when the teaching side has information the answering side lacks and the asymmetry is controlled.The paper frames six months of work as qualifying, rather than overturning, the core principle.