PaperlayerView exact revision on arXiv

Source-linked AI summary

Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning

Junlong Ke, Zichen Wen, Weijia Li, Conghui He, Linfeng Zhang

arXiv:2605.13255v1cs.AI

TL;DR

Existing on-policy self-distillation applies token-level supervision despite variation in teacher uncertainty. EGRSD adds an entropy-based confidence gate to reward direction and teacher–student magnitude, while CL-EGRSD uses causal lookahead to preserve transient pivots. On Qwen3-4B and Qwen3-8B, both methods advance the accuracy–length frontier among compared trainable methods.

  • Problem

    On-policy self-distillation typically weights token-level supervision uniformly despite substantial variation in the entropy of the teacher’s predictive distribution.

    Source: §1 · p. 2, §6 · p. 9

  • Method

    EGRSD combines reward-grounded direction, teacher–student likelihood-ratio magnitude, and an entropy-based confidence gate with a nonzero lower bound; CL-EGRSD adds causal-lookahead entropy.

    Source: §1 · p. 2, §4.3 · p. 6

  • Results

    EGRSD and CL-EGRSD advance the accuracy–length frontier among compared trainable methods on Qwen3-4B and Qwen3-8B.

    Source: §1 · p. 2, §6 · pp. 9–10

  • Takeaways & Limitations

    Entropy-gated token updates improve the compared trainable methods’ accuracy–length frontier while retaining transient pivot signals.

    Source: §6 · pp. 9–10

    Evaluation is restricted to mathematical reasoning, and transfer to code, agentic reasoning, or open-ended reasoning remains untested.

    Source: Limitations · p. 22

Abstract

from arXiv · show

On-policy self-distillation trains a reasoning model on its own rollouts while a teacher, often the same model conditioned on privileged context, provides dense token-level supervision. Existing objectives typically weight the teacher's token-level signal uniformly across a chain-of-thought sequence, despite substantial variation in the entropy of the teacher's predictive distribution. We propose EGRSD (Entropy-Guided Reinforced Self-Distillation), which unifies token-level updates through three signals: a reward-grounded direction, a teacher-student likelihood-ratio magnitude, and the proposed teacher-entropy confidence gate that down-weights high-entropy token positions while maintaining a nonzero lower bound on every token weight. We further introduce CL-EGRSD, a causal-lookahead variant that distinguishes sustained high-entropy spans from transient high-entropy positions whose following context rapidly becomes low entropy. Experiments with Qwen3-4B and Qwen3-8B in thinking mode show that EGRSD and CL-EGRSD advance the accuracy-length frontier among the compared trainable methods.

1 Introduction

Reasoning models often generate redundant intermediate tokens, motivating methods that preserve accuracy while reducing computation. EGRSD and CL-EGRSD add entropy-aware confidence to on-policy self-distillation and advance the accuracy–length frontier on Qwen3-4B and Qwen3-8B.

  • Redundant verification loops, self-correction markers, and repeated derivations increase reasoning latency and cost.Source: §1 · p. 1
  • On-policy self-distillation trains on student-sampled trajectories while a privileged teacher supplies dense token-level supervision.Source: §1 · p. 1
  • Teacher predictive entropy distinguishes low-entropy routine computation from high-entropy branching and strategy-shift positions.Source: §1 · pp. 1–2
  • EGRSD gates direction-aware token updates with privileged-teacher entropy, attenuating high-entropy positions while preserving a nonzero weight floor.Source: §1 · p. 2
  • CL-EGRSD uses a short causal lookahead to preserve transient high-entropy pivots while attenuating sustained high-entropy spans.Source: §1 · p. 2
  • EGRSD and CL-EGRSD advance the accuracy–length frontier among compared trainable methods on Qwen3-4B and Qwen3-8B.Source: §1 · p. 2, Figure 1 · p. 2

2 Related work

Related work establishes on-policy self-distillation as a response to offline train–test mismatch and frames teacher uncertainty as a signal for selective token supervision. EGRSD uses entropy as a multiplicative confidence gate rather than changing divergence objectives or training a separate uncertainty estimator.

  • On-policy distillation trains on student-sampled trajectories while a teacher provides dense token-level feedback, reducing offline train–test mismatch.Source: §2 · p. 3
  • Prior work motivates preserving high-entropy teacher tokens because they can signal uncertainty and represent multiple plausible continuations.Source: §2 · p. 3
  • EGRSD makes lock–fork heterogeneity explicit through a multiplicative teacher-confidence gate on the direction-aware update.Source: §2 · p. 3

3 Background

The background decomposes direction-aware self-distillation into reward-driven direction and teacher–student magnitude, while identifying uniform token weighting as the remaining confidence gap. EGRSD adds teacher entropy to this decomposition without changing rollout sampling or reward definition.

  • The teacher is a fixed privileged model, while the student samples on-policy rollouts and is the only trainable component.Source: §3 · p. 3
  • OPSD aligns the student with the teacher at every rollout position, whereas CRISP adapts the same per-token framework toward compression.Source: §3 · p. 3
  • The proposed update multiplies direction, teacher–student magnitude, and entropy-based confidence while leaving the rollout distribution and reward definition unchanged.Source: Figure 2 · p. 4
  • RLSD combines outcome-reward direction with a teacher–student likelihood-ratio magnitude shared across token updates.Source: §3 · p. 4
  • RLSD treats all tokens under its likelihood ratio as equally informative, and EGRSD adds teacher predictive entropy to modulate that coupling.Source: §3 · p. 4

4 Method

The method interprets token entropy through lock, fork, and pivot regimes, then uses confidence gates to attenuate sustained uncertainty without discarding transient transitions. CL-EGRSD applies causal-lookahead minimum entropy to recover weight at pivot positions.

EGRSD

  • EGRSD adds an explicit teacher-confidence signal to the direction–magnitude decomposition used by earlier objectives.Source: §4.1 · p. 4, Table 1 · p. 5

Entropy regimes

  • Low-entropy lock positions correspond to routine computation, while sustained high-entropy forks and transient high-entropy pivots require different treatment.Source: §4.1 · p. 5, Figure 3 · p. 5
  • Uniform suppression of high-entropy tokens can destroy pivot evidence, whereas uniform weighting can over-weight sustained fork evidence.Source: §4.1 · p. 5

EGRSD

  • The confidence gate computes privileged-teacher entropy, normalizes it across the minibatch, and applies a nonzero lower-bounded attenuation controlled by γ.Source: §4.2 · pp. 5–6
  • The linear gate can be interpreted geometrically as a bounded shrinkage rule whose sensitivity is controlled by γ.Source: §4.2 · p. 6

CL-EGRSD

  • CL-EGRSD replaces instantaneous entropy with the minimum entropy in a causal future window, preserving transient pivots while continuing to attenuate sustained high-entropy spans.Source: §4.3 · p. 6

5 Experiments

Experiments on Qwen3-4B and Qwen3-8B compare EGRSD and CL-EGRSD with trainable baselines under matched evaluation settings. The methods improve the accuracy–length frontier, while ablations examine entropy strength, lookahead, and teacher updates.

Main comparison

  • EGRSD and CL-EGRSD outperform all trainable baselines on both Qwen3-4B and Qwen3-8B.The comparison includes SFT, GRPO, OPSD, and CRISP under the same 100-step training budget.Source: §5.1 · pp. 6–7, §5.2 · p. 7
  • On Qwen3-8B, CL-EGRSD achieves the highest observed Avg., including a +7.40 HMMT25 gain over the no-train baseline at 12,232 tokens.Its generation length is comparable to OPSD’s 12,097 tokens.Source: §5.2 · p. 7, Table 3 · p. 8

Entropy strength

  • Moderate entropy attenuation is most stable: γ=0.1 reaches Avg. 68.32 in one snapshot, while γ=0.3 peaks at 68.23 with a smoother trajectory.Very weak γ=0.0 and strong γ≥0.7 settings trail at most snapshots.Source: §5.3 · p. 7, Table 4 · p. 8

Token efficiency

  • EGRSD records Eff=6.08 and CL-EGRSD Eff=6.06 on Qwen3-4B, the only trainable methods improving over SFT’s Eff=6.05.GRPO, OPSD, and CRISP fall to Eff≤5.86, while SFT’s nominal efficiency reflects compression paired with a ten-point accuracy drop.Source: §5.3 · pp. 7–8, Table 5 · p. 8

Lookahead

  • W=5 gives the strongest Qwen3-8B lookahead result, with larger gains under stronger suppression and joint tuning of γ and W.Lookahead is less consistent on Qwen3-4B but helps more on the larger model.Source: §5.3 · p. 8, Table 6 · p. 9, §5.5 · p. 9

Teacher update schedule

  • All three online teacher-update schedules underperform the frozen teacher.Tracking collapses the teacher–student log-ratio and magnitude toward zero and one, respectively, while eroding the privileged-context reference.Source: §5.4 · pp. 8–9, Table 7 · p. 9

Mechanism

  • High-entropy positions have roughly 140× the log-prob gap of low-entropy positions, supporting entropy as a confidence signal rather than a length penalty.CL-EGRSD selectively restores transient pivots while retaining suppression for sustained high-entropy spans.Source: §5.5 · p. 9

Lookahead

  • CL-EGRSD restores pivot-token weight by +0.111 on average versus +0.053 for sustained forks, a 2.1× selectivity ratio at γ=0.3 and W=5.Pivots comprise 20.9% of analyzed tokens, while lock restoration is below 0.001.Source: §5.5 · p. 9

6 Conclusion

The paper concludes that entropy-gated token updates improve the accuracy–length frontier across Qwen3-4B and Qwen3-8B, while connecting EGRSD to efficient reasoning and selective token-level credit assignment.

  • Across Qwen3-4B and Qwen3-8B benchmarks, entropy-gated updates were the only compared intervention improving over no training on the accuracy–length frontier.Source: §6 · p. 9
  • EGRSD attenuates high-entropy positions, whereas CL-EGRSD uses short causal lookahead to preserve transient pivot tokens.Source: §6 · p. 10, Figure 4 · p. 10
  • The approach targets reasoning efficiency by allocating token-level credit according to teacher confidence rather than weighting all positions uniformly.Source: A Extended related work · pp. 13–14, B Background details · p. 15
  • Its positioning combines reasoning compression, on-policy self-distillation, and token-level credit assignment.Source: A Extended related work · p. 14

C Derivations for the main-text remarks

The derivations motivate the linear entropy gate as a conservative approximation to a shrinkage rule and formalize causal lookahead for recovering weight at transient pivots without inflating uncertainty at lock positions.

  • The shrinkage model treats a raw update as useful signal plus zero-mean noise and uses a multiplicative gate to estimate the useful component.Source: §C.1 · p. 15
  • The entropy-to-noise connection relies on a monotone proxy linking normalized teacher entropy to the noise-to-signal ratio.Source: §C.1 · p. 15
  • The linear gate is an endpoint chord chosen as a simple conservative form, not claimed to be MSE-optimal.Source: §C.1 · p. 15
  • The causal smoothing family is monotone, conservative, idempotent, and restricted to current and future entropy values.Source: §C.2 · p. 15
  • At pivots, the minimum-based lookahead member maximizes weight recovery, while sustained forks receive essentially no recovery.Source: §C.2 · p. 16

D Full experimental details

The experiments compare EGRSD and CL-EGRSD with established baselines under a shared training protocol on Qwen3-4B and Qwen3-8B, with a documented hyperparameter boundary for the smaller model.

F Training dynamics

Training-dynamics diagnostics indicate that entropy-weighted objectives do not systematically inflate gradient magnitudes, with clipping remaining an occasional safeguard.

  • EGRSD and CL-EGRSD have similar mean pre-clip gradient norms of approximately 0.035, while baseline traces mostly remain between 0.02 and 0.08.Source: F Training dynamics · p. 18
  • Only one EGRSD spike reaches approximately 0.11 above the 0.1 clipping threshold, while CL-EGRSD remains below the threshold throughout.Source: F Training dynamics · p. 18, Figure A1 · p. 18
  • The diagnostics therefore characterize gradient clipping as an occasional safeguard rather than a persistent constraint.Source: F Training dynamics · p. 18

G Extended CL-EGRSD ablation

The extended CL-EGRSD ablation isolates the interaction between entropy coefficient γ and lookahead window W on Qwen3-8B. Strong suppression with (γ, W) = (1.0, 5) improves over the matched EGRSD reference across all four evaluated benchmarks, while qualitative analyses examine pivot rescue and lock/fork regimes.

Experimental setup

  • The sweep evaluates selected configurations on AIME24, AIME25, HMMT25, and Minerva with K = 4.MATH500 and GSM8K are excluded because they saturate across the method family.Source: G Extended CL-EGRSD ablation · pp. 18–19

Quantitative sweep

  • (γ, W) = (1.0, 5) improves over the matched W = 0 reference on all four evaluated benchmarks.The largest gains are on HMMT25 (47.50 →52.50) and AIME25 (66.67 →70.00).Source: G Extended CL-EGRSD ablation · p. 19

Qualitative mechanism

  • Figure A4 compares current entropy, five-token lookahead entropy, EGRSD weight, and CL-EGRSD weight to visualize pivot rescue.Green spans mark transient pivots whose future entropy becomes low, while red spans mark sustained forks.Source: H Qualitative pivot examples · pp. 19–20
  • Figure A5 overlays local entropy peaks with four-token left context on two Minerva windows to examine transition points.The accompanying analysis states that most annotated peaks are transient pivots rather than sustained forks.Source: H Qualitative pivot examples · p. 20

I AceReason-Nemotron-7B cross-architecture diagnostic

The AceReason-Nemotron-7B diagnostic tests the methods on a reasoning-tuned external base with a different post-training recipe and longer native reasoning style. CL-EGRSD is the only trained method that clearly exceeds the already-strong no-training baseline, while the analysis links this result to entropy-gated preservation of useful supervision.

Scope

  • The diagnostic is separate from the Qwen3 main comparison because AceReason uses a different post-training recipe, a longer native reasoning style, and a competitive no-training model.The results are reported as a cross-architecture diagnostic rather than a headline comparison.Source: I AceReason-Nemotron-7B… · p. 20

Results

  • 61.45 avg@4 (+0.82 over no-train) is achieved by CL-EGRSD at γ = 0.5, exceeding the 60.63 no-training baseline.CL-EGRSD also uses shorter completions (7,943 vs. 8,382 tokens) and reaches 93.91 on GSM8K (+2.42).Source: I AceReason-Nemotron-7B… · p. 20
  • Most trained baselines fail to improve over the no-training model: SFT reaches 59.60, GRPO 60.52, OPSD 60.13, and CRISP 60.75.These methods cluster around or below the already-strong no-training result.Source: I AceReason-Nemotron-7B… · p. 20

Mechanism analysis

  • The entropy trace shows strategy-pivot spikes, and at W = 5 the lookahead entropy is ≤0.08 for 13 of 14 annotated peaks.This matches the transient-transition signature targeted by CL-EGRSD.Source: I AceReason-Nemotron-7B… · p. 21

Interpretation

  • The authors state that uniform-weight distillation struggles to add value on this base, while entropy gating preserves a useful learning signal.They also caution that the entropy coefficient may need retuning across bases.Source: I AceReason-Nemotron-7B… · p. 21

J Weak-base cross-architecture diagnostic on Olmo-3-7B Base

The Olmo-3-7B Base diagnostic examines whether the methods avoid negative transfer on a weak, non-reasoning-tuned external base. CL-EGRSD is the only trainable method reported to improve over both no training and SFT, within a broader evaluation whose scope remains limited to mathematical reasoning.

Results

  • CL-EGRSD is the only trainable method that improves over both the no-training model and SFT on Olmo-3-7B Base.The diagnostic is designed to test negative transfer on a weak external base rather than provide a headline accuracy comparison.Source: J Weak-base cross-architecture… · p. 22

Interpretation

  • The authors interpret the entropy gate as stabilizing training under weak external bases by attenuating harmful token-level updates when privileged teacher signals are less reliable.This interpretation connects the weak-base result to the gate’s intended treatment of unreliable supervision.Source: J Weak-base cross-architecture… · p. 22

Limitations

  • The evaluation is restricted to mathematical reasoning, leaving transfer to code, agentic reasoning, and open-ended reasoning untested.The limitation applies to the reported evaluation scope rather than establishing failure in those settings.Source: Limitations · p. 22
  • EGRSD and CL-EGRSD add γ and, for CL-EGRSD, the lookahead window W as hyperparameters beyond existing OPSD-style objectives.The paper identifies scale-aware or schedule-based γ control as a future direction.Source: Limitations · p. 23

Method

  • The training step combines on-policy student rollouts, reward-derived direction, teacher–student log-ratio magnitude, and an entropy-based confidence gate.CL-EGRSD replaces instantaneous entropy with a causal minimum over a future window W, while EGRSD uses W = 0.Source: K Algorithm · p. 23

Related papers

Loading 2605.13255v1…