Source-linked AI summary
Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning
Junlong Ke, Zichen Wen, Weijia Li, Conghui He, Linfeng Zhang
TL;DR
Existing on-policy self-distillation applies token-level supervision despite variation in teacher uncertainty. EGRSD adds an entropy-based confidence gate to reward direction and teacher–student magnitude, while CL-EGRSD uses causal lookahead to preserve transient pivots. On Qwen3-4B and Qwen3-8B, both methods advance the accuracy–length frontier among compared trainable methods.
Problem
On-policy self-distillation typically weights token-level supervision uniformly despite substantial variation in the entropy of the teacher’s predictive distribution.
Method
EGRSD combines reward-grounded direction, teacher–student likelihood-ratio magnitude, and an entropy-based confidence gate with a nonzero lower bound; CL-EGRSD adds causal-lookahead entropy.
Results
EGRSD and CL-EGRSD advance the accuracy–length frontier among compared trainable methods on Qwen3-4B and Qwen3-8B.
Takeaways & Limitations
Entropy-gated token updates improve the compared trainable methods’ accuracy–length frontier while retaining transient pivot signals.
Takeaways & Limitations
Evaluation is restricted to mathematical reasoning, and transfer to code, agentic reasoning, or open-ended reasoning remains untested.
Abstract
from arXiv · showhide
On-policy self-distillation trains a reasoning model on its own rollouts while a teacher, often the same model conditioned on privileged context, provides dense token-level supervision. Existing objectives typically weight the teacher's token-level signal uniformly across a chain-of-thought sequence, despite substantial variation in the entropy of the teacher's predictive distribution. We propose EGRSD (Entropy-Guided Reinforced Self-Distillation), which unifies token-level updates through three signals: a reward-grounded direction, a teacher-student likelihood-ratio magnitude, and the proposed teacher-entropy confidence gate that down-weights high-entropy token positions while maintaining a nonzero lower bound on every token weight. We further introduce CL-EGRSD, a causal-lookahead variant that distinguishes sustained high-entropy spans from transient high-entropy positions whose following context rapidly becomes low entropy. Experiments with Qwen3-4B and Qwen3-8B in thinking mode show that EGRSD and CL-EGRSD advance the accuracy-length frontier among the compared trainable methods.
1 Introduction
Reasoning models often generate redundant intermediate tokens, motivating methods that preserve accuracy while reducing computation. EGRSD and CL-EGRSD add entropy-aware confidence to on-policy self-distillation and advance the accuracy–length frontier on Qwen3-4B and Qwen3-8B.
- Redundant verification loops, self-correction markers, and repeated derivations increase reasoning latency and cost.
- On-policy self-distillation trains on student-sampled trajectories while a privileged teacher supplies dense token-level supervision.
- Teacher predictive entropy distinguishes low-entropy routine computation from high-entropy branching and strategy-shift positions.
- EGRSD gates direction-aware token updates with privileged-teacher entropy, attenuating high-entropy positions while preserving a nonzero weight floor.
- CL-EGRSD uses a short causal lookahead to preserve transient high-entropy pivots while attenuating sustained high-entropy spans.
- EGRSD and CL-EGRSD advance the accuracy–length frontier among compared trainable methods on Qwen3-4B and Qwen3-8B.
2 Related work
Related work establishes on-policy self-distillation as a response to offline train–test mismatch and frames teacher uncertainty as a signal for selective token supervision. EGRSD uses entropy as a multiplicative confidence gate rather than changing divergence objectives or training a separate uncertainty estimator.
- On-policy distillation trains on student-sampled trajectories while a teacher provides dense token-level feedback, reducing offline train–test mismatch.
- Prior work motivates preserving high-entropy teacher tokens because they can signal uncertainty and represent multiple plausible continuations.
- EGRSD makes lock–fork heterogeneity explicit through a multiplicative teacher-confidence gate on the direction-aware update.
3 Background
The background decomposes direction-aware self-distillation into reward-driven direction and teacher–student magnitude, while identifying uniform token weighting as the remaining confidence gap. EGRSD adds teacher entropy to this decomposition without changing rollout sampling or reward definition.
- The teacher is a fixed privileged model, while the student samples on-policy rollouts and is the only trainable component.
- OPSD aligns the student with the teacher at every rollout position, whereas CRISP adapts the same per-token framework toward compression.
- The proposed update multiplies direction, teacher–student magnitude, and entropy-based confidence while leaving the rollout distribution and reward definition unchanged.
- RLSD combines outcome-reward direction with a teacher–student likelihood-ratio magnitude shared across token updates.
- RLSD treats all tokens under its likelihood ratio as equally informative, and EGRSD adds teacher predictive entropy to modulate that coupling.
4 Method
The method interprets token entropy through lock, fork, and pivot regimes, then uses confidence gates to attenuate sustained uncertainty without discarding transient transitions. CL-EGRSD applies causal-lookahead minimum entropy to recover weight at pivot positions.
- EGRSD: EGRSD adds an explicit teacher-confidence signal to the direction–magnitude decomposition used by earlier objectives.
- Entropy regimes: Low-entropy lock positions correspond to routine computation, while sustained high-entropy forks and transient high-entropy pivots require different treatment.
- Entropy regimes: Uniform suppression of high-entropy tokens can destroy pivot evidence, whereas uniform weighting can over-weight sustained fork evidence.
- EGRSD: The confidence gate computes privileged-teacher entropy, normalizes it across the minibatch, and applies a nonzero lower-bounded attenuation controlled by γ.
- EGRSD: The linear gate can be interpreted geometrically as a bounded shrinkage rule whose sensitivity is controlled by γ.
- CL-EGRSD: CL-EGRSD replaces instantaneous entropy with the minimum entropy in a causal future window, preserving transient pivots while continuing to attenuate sustained high-entropy spans.
5 Experiments
Experiments on Qwen3-4B and Qwen3-8B compare EGRSD and CL-EGRSD with trainable baselines under matched evaluation settings. The methods improve the accuracy–length frontier, while ablations examine entropy strength, lookahead, and teacher updates.
- Main comparison: EGRSD and CL-EGRSD outperform all trainable baselines on both Qwen3-4B and Qwen3-8B.The comparison includes SFT, GRPO, OPSD, and CRISP under the same 100-step training budget.
- Main comparison: On Qwen3-8B, CL-EGRSD achieves the highest observed Avg., including a +7.40 HMMT25 gain over the no-train baseline at 12,232 tokens.Its generation length is comparable to OPSD’s 12,097 tokens.
- Entropy strength: Moderate entropy attenuation is most stable: γ=0.1 reaches Avg. 68.32 in one snapshot, while γ=0.3 peaks at 68.23 with a smoother trajectory.Very weak γ=0.0 and strong γ≥0.7 settings trail at most snapshots.
- Token efficiency: EGRSD records Eff=6.08 and CL-EGRSD Eff=6.06 on Qwen3-4B, the only trainable methods improving over SFT’s Eff=6.05.GRPO, OPSD, and CRISP fall to Eff≤5.86, while SFT’s nominal efficiency reflects compression paired with a ten-point accuracy drop.
- Lookahead: W=5 gives the strongest Qwen3-8B lookahead result, with larger gains under stronger suppression and joint tuning of γ and W.Lookahead is less consistent on Qwen3-4B but helps more on the larger model.
- Teacher update schedule: All three online teacher-update schedules underperform the frozen teacher.Tracking collapses the teacher–student log-ratio and magnitude toward zero and one, respectively, while eroding the privileged-context reference.
- Mechanism: High-entropy positions have roughly 140× the log-prob gap of low-entropy positions, supporting entropy as a confidence signal rather than a length penalty.CL-EGRSD selectively restores transient pivots while retaining suppression for sustained high-entropy spans.
- Lookahead: CL-EGRSD restores pivot-token weight by +0.111 on average versus +0.053 for sustained forks, a 2.1× selectivity ratio at γ=0.3 and W=5.Pivots comprise 20.9% of analyzed tokens, while lock restoration is below 0.001.
6 Conclusion
The paper concludes that entropy-gated token updates improve the accuracy–length frontier across Qwen3-4B and Qwen3-8B, while connecting EGRSD to efficient reasoning and selective token-level credit assignment.
- Across Qwen3-4B and Qwen3-8B benchmarks, entropy-gated updates were the only compared intervention improving over no training on the accuracy–length frontier.
- EGRSD attenuates high-entropy positions, whereas CL-EGRSD uses short causal lookahead to preserve transient pivot tokens.
- The approach targets reasoning efficiency by allocating token-level credit according to teacher confidence rather than weighting all positions uniformly.
- Its positioning combines reasoning compression, on-policy self-distillation, and token-level credit assignment.
C Derivations for the main-text remarks
The derivations motivate the linear entropy gate as a conservative approximation to a shrinkage rule and formalize causal lookahead for recovering weight at transient pivots without inflating uncertainty at lock positions.
- The shrinkage model treats a raw update as useful signal plus zero-mean noise and uses a multiplicative gate to estimate the useful component.
- The entropy-to-noise connection relies on a monotone proxy linking normalized teacher entropy to the noise-to-signal ratio.
- The linear gate is an endpoint chord chosen as a simple conservative form, not claimed to be MSE-optimal.
- The causal smoothing family is monotone, conservative, idempotent, and restricted to current and future entropy values.
- At pivots, the minimum-based lookahead member maximizes weight recovery, while sustained forks receive essentially no recovery.
D Full experimental details
The experiments compare EGRSD and CL-EGRSD with established baselines under a shared training protocol on Qwen3-4B and Qwen3-8B, with a documented hyperparameter boundary for the smaller model.
- The comparison includes no training, SFT, GRPO, OPSD, and CRISP under a shared protocol, while RLSD lacks publicly compatible training code.
- Training uses answer-verified OpenThoughts-114k samples with a problem and concise reference solution, giving the teacher privileged context unavailable to the student.
- Evaluation uses Qwen3-4B and Qwen3-8B in thinking mode across mathematical reasoning benchmarks, with shared configurations differing mainly in loss-specific switches.
- The main configurations use γ = 0.3 for EGRSD, while CL-EGRSD uses W = 3 on 4B and (γ, W) = (1.0, 5) on 8B.
- Stronger-suppression configurations for Qwen3-4B were not evaluated, leaving that joint γ–W sweep for future work.
F Training dynamics
Training-dynamics diagnostics indicate that entropy-weighted objectives do not systematically inflate gradient magnitudes, with clipping remaining an occasional safeguard.
- EGRSD and CL-EGRSD have similar mean pre-clip gradient norms of approximately 0.035, while baseline traces mostly remain between 0.02 and 0.08.
- Only one EGRSD spike reaches approximately 0.11 above the 0.1 clipping threshold, while CL-EGRSD remains below the threshold throughout.
- The diagnostics therefore characterize gradient clipping as an occasional safeguard rather than a persistent constraint.
G Extended CL-EGRSD ablation
The extended CL-EGRSD ablation isolates the interaction between entropy coefficient γ and lookahead window W on Qwen3-8B. Strong suppression with (γ, W) = (1.0, 5) improves over the matched EGRSD reference across all four evaluated benchmarks, while qualitative analyses examine pivot rescue and lock/fork regimes.
- Experimental setup: The sweep evaluates selected configurations on AIME24, AIME25, HMMT25, and Minerva with K = 4.MATH500 and GSM8K are excluded because they saturate across the method family.
- Quantitative sweep: (γ, W) = (1.0, 5) improves over the matched W = 0 reference on all four evaluated benchmarks.The largest gains are on HMMT25 (47.50 →52.50) and AIME25 (66.67 →70.00).
- Qualitative mechanism: Figure A4 compares current entropy, five-token lookahead entropy, EGRSD weight, and CL-EGRSD weight to visualize pivot rescue.Green spans mark transient pivots whose future entropy becomes low, while red spans mark sustained forks.
- Qualitative mechanism: Figure A5 overlays local entropy peaks with four-token left context on two Minerva windows to examine transition points.The accompanying analysis states that most annotated peaks are transient pivots rather than sustained forks.
I AceReason-Nemotron-7B cross-architecture diagnostic
The AceReason-Nemotron-7B diagnostic tests the methods on a reasoning-tuned external base with a different post-training recipe and longer native reasoning style. CL-EGRSD is the only trained method that clearly exceeds the already-strong no-training baseline, while the analysis links this result to entropy-gated preservation of useful supervision.
- Scope: The diagnostic is separate from the Qwen3 main comparison because AceReason uses a different post-training recipe, a longer native reasoning style, and a competitive no-training model.The results are reported as a cross-architecture diagnostic rather than a headline comparison.
- Results: 61.45 avg@4 (+0.82 over no-train) is achieved by CL-EGRSD at γ = 0.5, exceeding the 60.63 no-training baseline.CL-EGRSD also uses shorter completions (7,943 vs. 8,382 tokens) and reaches 93.91 on GSM8K (+2.42).
- Results: Most trained baselines fail to improve over the no-training model: SFT reaches 59.60, GRPO 60.52, OPSD 60.13, and CRISP 60.75.These methods cluster around or below the already-strong no-training result.
- Mechanism analysis: The entropy trace shows strategy-pivot spikes, and at W = 5 the lookahead entropy is ≤0.08 for 13 of 14 annotated peaks.This matches the transient-transition signature targeted by CL-EGRSD.
- Interpretation: The authors state that uniform-weight distillation struggles to add value on this base, while entropy gating preserves a useful learning signal.They also caution that the entropy coefficient may need retuning across bases.
J Weak-base cross-architecture diagnostic on Olmo-3-7B Base
The Olmo-3-7B Base diagnostic examines whether the methods avoid negative transfer on a weak, non-reasoning-tuned external base. CL-EGRSD is the only trainable method reported to improve over both no training and SFT, within a broader evaluation whose scope remains limited to mathematical reasoning.
- Results: CL-EGRSD is the only trainable method that improves over both the no-training model and SFT on Olmo-3-7B Base.The diagnostic is designed to test negative transfer on a weak external base rather than provide a headline accuracy comparison.
- Interpretation: The authors interpret the entropy gate as stabilizing training under weak external bases by attenuating harmful token-level updates when privileged teacher signals are less reliable.This interpretation connects the weak-base result to the gate’s intended treatment of unreliable supervision.
- Limitations: The evaluation is restricted to mathematical reasoning, leaving transfer to code, agentic reasoning, and open-ended reasoning untested.The limitation applies to the reported evaluation scope rather than establishing failure in those settings.
- Limitations: EGRSD and CL-EGRSD add γ and, for CL-EGRSD, the lookahead window W as hyperparameters beyond existing OPSD-style objectives.The paper identifies scale-aware or schedule-based γ control as a future direction.
- Method: The training step combines on-policy student rollouts, reward-derived direction, teacher–student log-ratio magnitude, and an entropy-based confidence gate.CL-EGRSD replaces instantaneous entropy with a causal minimum over a future window W, while EGRSD uses W = 0.