Source-linked AI summary
On-Policy Self-Distillation without Any Supervision
Yijiang Li, Bingyang Wang, Yijun Liang, Yunjie Tian, Di Fu, Nuno Vasconcelos
TL;DR
Existing on-policy self-distillation methods still depend on external supervision, limiting genuine self-distillation on unlabeled problems. U-OPSD uses majority-vote consensus from the model’s own rollouts as privileged context and distills on disagreeing completions, improving over base models and matching or surpassing supervised methods across the evaluated settings.
Problem
Existing OPD and OPSD methods rely on ground-truth solutions, environmental feedback, or stronger-model guidance, leaving the privileged information external.
Method
U-OPSD samples multiple on-policy rollouts, forms a pseudo-solution by majority vote under a self-consistency threshold, and distills the conditioned distribution on disagreeing completions.
Results
Across five mathematical benchmarks and six Qwen3 configurations, U-OPSD improves over base models and matches or exceeds supervised SFT, GRPO, and OPSD, with gains up to 10.9%, 8.9%, and 3.2%, respectively.
Takeaways & Limitations
The results support replacing ground-truth solutions with model-generated consensus for on-policy self-distillation in the evaluated mathematical-reasoning settings.
Takeaways & Limitations
Experiments cover only Qwen3 models and competition mathematics with automatically checkable final answers; open-ended generation would require softer consensus than exact-match voting.
Abstract
from arXiv · showhide
On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at [https://github.com/williamium3000/u-opsd](https://github.com/williamium3000/u-opsd).
1 Introduction
Existing on-policy distillation methods still depend on external supervision, while U-OPSD constructs privileged context from the model’s own consensus and distills on disagreeing rollouts. Across Qwen3 configurations and mathematical benchmarks, it improves over base models and matches or exceeds supervised baselines.
- Motivation: Existing OPD and OPSD methods rely on external supervision from ground-truth solutions, environmental feedback, or stronger models.
- Method: U-OPSD samples multiple independent rollouts, extracts their answers, and uses majority voting under a self-consistency threshold to form a pseudo-solution.
- Method: The longest agreeing rollout becomes the teacher reference, while disagreeing rollouts serve as student trajectories for token-level self-distillation.
- Results: 8.5–10.7% gains over base models occur in non-thinking mode, while thinking-mode gains are 1.9–2.2% across the evaluated Qwen3 configurations.
- Results: U-OPSD remains on par with or better than supervised SFT, GRPO, and OPSD with ground-truth labels, with gains up to 10.9%, 8.9%, and 3.2%, respectively.
2 Related work
Related work develops on-policy distillation, self-rewarding reinforcement learning, and self-training from model-generated signals. These approaches reduce reliance on external teachers or labels but differ in whether supervision is token-level, reward-based, or consistency-derived.
- Self-rewarding reinforcement learning: Self-rewarding reinforcement learning replaces external verification with reward signals derived from the model’s behavior on unlabeled data.
- On-policy (self-) distillation: On-policy distillation trains on student-generated trajectories while retaining dense token-level teacher supervision and reducing train–inference mismatch.
- Self-training, self-distillation and consistency: Self-training and self-distillation derive supervision from model generations using verification, confidence, consistency, and iterative self-improvement.
3 Method
U-OPSD replaces externally supervised on-policy distillation with pseudo-solutions derived from the model’s own rollout consensus. It filters for sufficiently consistent votes, then distills solution-conditioned teacher distributions into disagreeing on-policy trajectories.
- Preliminaries: Existing OPD and OPSD methods provide dense on-policy supervision but rely on external teachers, ground-truth solutions, demonstrations, or environmental feedback.OPD uses a richer teacher context, while OPSD uses the same model conditioned on a ground-truth solution.
- U-OPSD: U-OPSD constructs its privileged context from internal consistency, using majority agreement among the model’s own on-policy rollouts instead of external supervision.The pseudo-solution replaces the ground-truth solution in the teacher context.
- Sampling and voting: For each prompt, U-OPSD samples G rollouts, parses their final answers, and separates valid generations according to agreement with the majority-vote pseudo-answer.Invalid generations are excluded from both agreeing and disagreeing sets.
- Sampling and voting: U-OPSD trusts a vote only when its self-consistency score reaches the threshold, using τ = 1/2 for an absolute majority.Prompts below the threshold contribute no gradient, and truncated generations reduce confidence because the score is normalized by all G rollouts.
- Distillation: After selecting an agreeing rollout as the teacher reference, U-OPSD distills its solution-conditioned next-token distribution along disagreeing rollouts.This provides dense corrective supervision on prefixes leading toward answers inconsistent with the pseudo-solution.
- Distillation: The method skips prompts with unreliable votes or no disagreement, concentrating training on cases where the model identifies a plausible solution but still produces conflicting trajectories.This competence-frontier curriculum emerges from the model’s voting statistics without external difficulty labels or scheduling.
4 Experiments
Experiments show that U-OPSD improves mathematical reasoning across model modes and instruction-tuned models, while ablations identify how consensus, rollout selection, teacher updates, and divergence choices affect performance.
- Main results: U-OPSD improves Qwen3-4B and Qwen3-8B over their base models by 8.5% and 10.7% in non-thinking mode.It also exceeds OPSD by 3.2% and 2.3% at the two model scales.
- Main results: U-OPSD improves Qwen3-4B and Qwen3-8B over their base models by 2.2% and 1.9% in thinking mode, matching or slightly exceeding OPSD and outperforming GRPO.The gains are smaller than in non-thinking mode.
- Main results: U-OPSD achieves the highest average performance on both tested instruction-tuned models and surpasses supervised OPSD by 1.1% and 1.7%.The recipe also transfers to Qwen3-30B-A3B-Instruct-2507 without model-specific hyperparameter tuning.
- Pseudo-label quality: Pseudo-labels are available for 94.0% of prompts, and 86.7% of those pseudo-labels match the gold answer.Under G=8 and τ=0.5, 96.3% of rollouts yield parsable boxed answers and fewer than 10% of valid rollouts disagree with the vote.
- Configuration ablations: The lowest tested self-consistency threshold performs best, while increasing rollouts beyond G=8 yields initially higher but eventually saturating returns.τ=0.3 reaches 58.59 versus 57.10 at τ=0.5; G=12 gains 4.7% over the default, while G=16 gives back half of that gain.
- Configuration ablations: EMA teacher updates outperform the frozen default, with decay 0.995 gaining 2.4% at the best checkpoint and 4.1% at step 150.The reported experiments otherwise use the frozen teacher for cost-matched comparison with OPSD.
- Teacher reference and target: Conditioning the teacher on the longest agreeing reasoning trace and distilling the longest disagreeing rollout gives the best setting at 59.00.Replacing the full reference with only the boxed pseudo-label lowers performance by 10.3% to 15.8% and falls below the base model.
- Divergence computation: Full-vocabulary divergence outperforms sampled-token distillation under pseudo-label supervision, while top-k truncation preserves comparable performance with lower computation.The full-distribution objective leads by 17.8% on AIME25 and 7.2% on HMMT25 in the cited comparison; top-100 ties it on AIME24 and trails elsewhere.
5 Limitations
The evaluation is limited to competition mathematics within the Qwen3 family, and the voting procedure assumes answers can be extracted and canonicalized. Results also vary by reasoning regime, while majority-vote supervision depends on base-model competence and uncertainty remains about training variance.
- Scope: The experiments cover one model family, Qwen3, at 4B and 8B scales, and one domain, competition mathematics with automatically checkable final answers.
- Scope: Extending U-OPSD to open-ended generation would require replacing exact-match voting with a softer consensus.
- Regime dependence: The advantage over supervised OPSD is larger in non-thinking mode than in thinking mode, where U-OPSD matches OPSD in the measured regime.U-OPSD exceeds OPSD by 3.2% and 2.3% at 4B and 8B in Table 1, versus 0.9% and 0.02% in Table 2.
- Base-model competence: Majority-vote supervision is bounded by the answer the base model produces most often, with 13.3% wrong pseudo-labels measured in-domain.Training dynamics under deliberately corrupted votes were not measured.
- Variance and repeats: Seed-replicated error bars for every training setup are pending, despite evaluating each checkpoint with 12 samples across five benchmarks.
6 Conclusion
U-OPSD replaces ground-truth solutions in on-policy self-distillation with majority-vote consensus from the model’s own rollouts, distilling only on self-inconsistent completions. It outperforms supervised OPSD on the five-benchmark average in non-thinking mode and ties it in thinking mode across the evaluated model sizes.
- U-OPSD replaces the ground-truth solution with the model’s majority-vote consensus, using the agreeing rollout as teacher reference and disagreeing rollouts for distillation.
- The label-free variant outperforms supervised OPSD on the five-benchmark average.
- Across two model sizes, the advantage reproduces at both scales in non-thinking mode and becomes a tie in thinking mode.The thinking-mode base model is already strong, leaving little headroom.