E$^2$-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation

1 INTRODUCTION

• Entropy overshoot occurs when student entropy rises above the teacher’s and stays elevated, eroding decisiveness and amplifying teacher-side uncertainty into noisy behavior.Figure 1 also describes the reference-conditioned teacher as less certain on student prefixes, suggesting a reasoning-path mismatch.Source: Figure 1 · p. 1, §1 · p. 2
• The teacher’s answer-conditioned guidance transfers poorly to student prefixes, while fixed forward KL keeps diffusing an already diffuse student policy.The teacher can narrate toward a known answer rather than reason from scratch, making its guidance answer-specific rather than transferable.Source: §1 · p. 2
• Exemplar-guided teaching replaces the current answer with a solved neighboring problem, offering strategy guidance without revealing the destination answer.The teacher is conditioned on the exemplar and student prefix; the student still sees only the problem and its own prefix.Source: §1 · p. 2, Figure 2 · p. 2
• Entropy-aware distillation uses the student–teacher entropy gap to select forward KL for exploration or reverse KL for refocusing, with gap magnitude scaling correction strength.Together, the changes align teacher guidance with the student’s reasoning path and regulate when to explore versus concentrate.Source: §1 · pp. 2–3, Figure 2 · p. 2
• Up to 4.3 points in mean@16 over OPSD; out-of-domain gains reach 4.9 mean@16 points over the corresponding base model.The reported gains accompany a framework that requires no extra forward passes and has minimum overhead.Source: §1 · p. 2

2.1 KL DIVERGENCE

• Forward KL favors coverage of target-supported modes, whereas reverse KL favors concentration by penalizing model mass unsupported by the target.These complementary biases motivate using different KL directions for different student–teacher entropy states.Source: §2.1 · p. 3

2.2 ON-POLICY DISTILLATION

• On-policy distillation scores student-generated prefixes with both student and teacher, then matches their per-token distributions using a divergence with gradients through the student.The passage also describes pointwise clipping and a sparse-support approximation using teacher top-K tokens plus a tail bucket.Source: §2.2 · p. 3

2.3 RELATED WORK

• OPSD uses a student copy as teacher, conditioned on the current reference solution and matched with forward KL; E2-OPSD instead targets the solution-conditioned mismatch and relative entropy dynamics.The related-work discussion also positions E2-OPSD’s retrieved solved neighbor as reusable strategy guidance across instances.Source: §2.3 · p. 4

3 METHOD

• E2-OPSD changes teacher context and per-token divergence while retaining one frozen teacher and requiring no additional forward passes, verifier, or reward during distillation.Privileged context is removed at inference.Source: §3 · p. 4, §3.2 · p. 6

3.1 EXEMPLAR-GUIDED TEACHING

• Exemplar-Guided Teaching replaces the current solution with a solved neighboring problem, so the teacher must solve the current problem while using the exemplar as transferable strategy.Problems are embedded and nearest neighbors are selected offline; a neighbor is sampled per rollout, and the student continues to see only its own problem and prefix.Source: §3.1 · p. 5

3.2 ENTROPY-AWARE DISTILLATION

• Bidirectional Entropy Routing applies forward KL when the student is more concentrated than the teacher and reverse KL when the student is more diffuse.This replaces fixed one-way pressure with a state-dependent correction that can broaden or refocus the student policy.Source: §3.2 · p. 5
• Adaptive Gap Damping scales routed updates by entropy-gap magnitude, suppressing small gaps relative to the batch scale while retaining most weight for much larger gaps.The batch scale is recomputed each minibatch, so damping adapts to training dynamics without a tunable threshold.Source: §3.2 · p. 5

4.1 EXPERIMENTAL SETTINGS

• The evaluation uses four Qwen3 and Olmo3 configurations, DeepMath-103K training, four math benchmarks, a 100-puzzle ZebraLogic subset, and HumanEval, with three evaluation seeds.Baselines include Base, OPSD, SDPO, GRPO, RLSD, and OPSD + EOPD; trained methods share a 360-step budget.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• E2-OPSD achieves the highest average math mean@16 across four model configurations spanning two model families and thinking and instruct modes.Table 1 reports math mean@16 over three seeds.Source: §4.2 · p. 6, Table 1 · p. 7
• E2-OPSD improves over Base in logical reasoning and code generation across all four configurations, with logical reasoning gains of 4.9 and 3.1 points on two Qwen3 models.Table 2 covers out-of-domain logical reasoning and code generation on ZebraLogic and HumanEval, reporting mean@16 and pass@8.Source: §4.2 · p. 7, Table 2 · p. 7
• Under OPSD, student entropy crosses the teacher’s and keeps rising, while E2-OPSD keeps student entropy regulated around the teacher throughout training.Figure 4 also shows routing shifting from forward-KL-dominated early exploration toward increased reverse-KL concentration later.Source: §4.2 · p. 8, Figure 4 · p. 8

4.3 ABLATION STUDY

• On Qwen3-1.7B Thinking, exemplar-guided teaching and bidirectional entropy routing raise mean@16 from OPSD’s 44.6 to 46.0 and 47.6; adding adaptive gap damping reaches 48.4, and all components reach 48.9.Figure 5(a) presents the component contributions, with each component added on top of OPSD.Source: §4.3 · p. 8, Figure 5 · p. 8
• With teacher context fixed and adaptive gap damping disabled, entropy-aware routing outperforms fixed KL directions by expanding or contracting according to the student–teacher entropy gap.Fixed forward KL drives overshoot, reverse KL restricts exploration, and their mixture applies both corrections simultaneously.Source: §4.3 · p. 9, Figure 6 · p. 9
• On student prefixes, exemplar-guided teaching reduces teacher–student forward KL by 25.9%, from 0.1549 to 0.1148 nats.The teacher’s entropy is 0.261 nats with EGT, closer to the student’s 0.268 nats than the reference-conditioned teacher’s 0.190 nats.Source: Figure 5 · p. 8, §4.3 · p. 9
• Entropy-discordant positions account for 7.67% of response tokens but 20.18% of relative gradient mass without AGD; with AGD, their share rises to 28.50%.High entropy is defined as H ≥ ln 2; strongly damped positions have 0.985 mean probability-mass overlap.Source: Figure 5 · p. 8, §4.3 · p. 9

• OPSD’s answer-informed teacher supplies guidance the student cannot fully learn, while forward KL keeps pushing the student outward into persistent distributional mismatch.Source: §5 · p. 9
• E2-OPSD constructs more locally learnable teacher targets and adapts distillation’s direction and strength to the student’s training state.Source: §5 · p. 10
• Across thinking and instruct models, E2-OPSD improves reasoning, with gains transferring to out-of-domain tasks.Source: §5 · p. 10
• The analyses suggest effective self-distillation requires matching teacher guidance to student-reachable states and uncertainty structure, not merely changing entropy.Source: §5 · p. 10

• This section expands Section 4.1’s experimental protocol, while Appendix B provides complete results and standard deviations.Source: A EXPERIMENTAL DETAILS · p. 16

• Training prompts come from DeepMath-103K, filtered to the difficulty bands in Table 3; Instruct configurations use Qwen3-4B-Instruct-2507 and Olmo-3-7B-Instruct.Within each configuration, student, teacher where applicable, and evaluation use the same reasoning mode.Source: §A.1 · p. 16, Table 3 · p. 16
• Evaluation covers AMC23, AIME’24, AIME’25, and HMMT’25 Feb for math, plus ZebraLogic for logic and HumanEval for code generation.ZebraLogic uses a fixed 100-puzzle subset, sampling four puzzles per grid size, shared across configurations.Source: §A.1 · p. 16

• Methods and ablations use veRL with vLLM rollouts on a single 8×H200 node, shared Table 4 training settings, and 360 optimizer steps.Source: §A.2 · p. 16, Table 4 · p. 16

A.3 EVALUATION PROTOCOL

• Mean@16 is averaged over three evaluation seeds.Source: §A.3 · p. 16
• Evaluation uses the student alone with the original question.Source: §A.3 · p. 16
• Students use the backbone’s corresponding reasoning mode, and Table 5 summarizes evaluation settings.Source: §A.3 · p. 16, Table 5 · p. 17

A.4 METHOD-SPECIFIC IMPLEMENTATION AND EXEMPLARS

• E2-OPSD uses one retrieved solved neighbor as teacher context, while OPSD uses the current reference solution.Both methods use a frozen initial teacher; E2-OPSD combines BER and AGD, with teacher-selected top-100 support plus a tail and per-element cap.Source: §A.4 · p. 18
• The exemplar index uses 15 self-excluded neighbors within each difficulty band, with exemplars used only during training.One neighbor and its dataset reference solution are drawn for each rollout; index construction for Qwen3-1.7B took 219 seconds on eight H200 GPUs.Source: §A.4 · p. 18
• The exemplar prompt labels the solved neighbor as a different related problem and directs the model to transfer strategies while solving the original.The prompt also specifies that the student prefix is scored by the teacher and requests a boxed final answer.Source: §A.4 · p. 18

B.1 EVALUATION METRICS

• Estimated pass@1 equals mean@16, and reported entries are mean ± population standard deviation over three seeds.Best and second-best distinct means are indicated in the evaluation reporting.Source: §B.1 · p. 19

B.2 IN-DOMAIN MATHEMATICAL REASONING

• 48.89±0.32 versus 44.64±0.41 are the E2-OPSD and OPSD Avg entries, respectively, in one reported mathematical reasoning configuration.Avg equally weights the four benchmarks.Source: Table 7 · p. 19

B.3 OUT-OF-DOMAIN EVALUATION

• The out-of-domain evaluation covers logical reasoning and code generation, with ZebraLogic scored by whole-puzzle correctness.Source: Table 10 · p. 22

B.4 ABLATION RESULTS

• The component ablation uses Qwen3-1.7B Thinking and equally weights four benchmarks in its Avg; AGD means adaptive gap damping.Source: Table 11 · p. 22

C.1 ROLLOUT AND PROBE SETTINGS

• The diagnostic probes use fixed student responses; the teacher receives either the reference solution or a retrieved neighbor, while the student sees only the original problem.The AGD gradient probe compares fixed distributions with AGD off and on, and is an offline logit-gradient probe rather than a replay of training updates.Source: §C.1 · p. 23

C.2 DETAILED PROBE RESULTS

• EGT reduces teacher-to-student forward KL by 25.9% on shared student prefixes.Own-prefix entropy and response length describe separate trajectories, while the KL comparison uses shared student prefixes.Source: §C.2 · p. 23, Table 13 · p. 24
• HL and LH positions make up 7.67% of response tokens, while their gradient shares increase by 3.43 and 4.89 percentage points, respectively.The probe reports all four entropy buckets and normalizes gradient shares by response and problem.Source: §C.2 · p. 24, Table 14 · p. 24

D ENTROPY GAP ON A SHARED CANDIDATE SET

• The shared-set entropy decomposition separates binary membership entropy from conditional entropies inside and outside the candidate set.When the full entropy gap exceeds the stated bound Bη, restricting to the shared set preserves its sign.Source: D ENTROPY GAP ON A SHARED CANDIDATE SET · p. 25
• The shared-set, tail-bound, and margin conditions remain analytical assumptions, not quantities verified by the reported overlap statistics.The probe reports mean probability-mass overlap of 0.9848 at strongly damped positions and 0.8343 in their high-student-entropy subset.Source: D ENTROPY GAP ON A SHARED CANDIDATE SET · p. 25