PaperlayerView exact revision on arXiv

Source-linked AI summary

Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information

Guobin Shen, Xiang Cheng, Chenxiao Zhao, Lei Huang, Jindong Li, Dongcheng Zhao, Xing Yu

arXiv:2605.11609v1cs.LGcs.AIcs.CL

TL;DR

Math reasoning gains from on-policy self-distillation are inconsistent, partly because privileged context suppresses deliberation tokens. AntiSD reverses the distillation signal with bounded Jensen–Shannon ascent and entropy gating, reaching GRPO accuracy in 2 to 10× fewer steps and improving final accuracy by up to 11.5 points.

  • Problem

    On-policy self-distillation can improve several tasks but remains inconsistent on math reasoning, motivating methods that let models supply token-level credit without external teachers.

  • Method

    AntiSD ascends bounded Jensen–Shannon divergence between student and privileged-context teacher, reversing the per-token signal and gating it when teacher entropy collapses.

  • Results

    2 to 10× fewer training steps reach GRPO baseline accuracy, while final mean accuracy exceeds GRPO by +2.1 to +11.5 points across five models.

  • Takeaways & Limitations

    AntiSD provides a drop-in self-distillation replacement that improves credit assignment and math reasoning performance across models from 4B to 30B parameters.

  • Takeaways & Limitations

    The method drops the trajectory-level REINFORCE term from the distillation objective and relies on GRPO to handle trajectory-level credit assignment.

Abstract

from arXiv · show

On-policy self-distillation, where a student is pulled toward a copy of itself conditioned on privileged context (e.g., a verified solution or feedback), offers a promising direction for advancing reasoning capability without a stronger external teacher. Yet in math reasoning the gains are inconsistent, even when the same approach succeeds elsewhere. A pointwise mutual information analysis traces the failure to the privileged context itself: it inflates the teacher's confidence on tokens already implied by the solution (structural connectives, verifiable claims) and deflates it on deliberation tokens ("Wait", "Let", "Maybe") that drive multi-step search. We propose Anti-Self-Distillation (AntiSD), which ascends a divergence between student and teacher rather than descending it: this reverses the per-token sign and yields a naturally bounded advantage in one step. An entropy-triggered gate disables the term once the teacher entropy collapses, completing a drop-in replacement for default self-distillation. Across five models from 4B to 30B parameters on math reasoning benchmarks, AntiSD reaches the GRPO baseline's accuracy in 2 to 10x fewer training steps and improves final accuracy by up to 11.5 points. AntiSD opens a path to scalable self-improvement, where a language model bootstraps its own reasoning through its training signal.

1 Introduction

The introduction identifies privileged-context bias as the reason default on-policy self-distillation is unreliable for mathematical reasoning. It proposes Anti-Self-Distillation, which reverses the per-token signal and uses entropy gating to accelerate and improve training.

  • Motivation: Default self-distillation typically fails to outperform strong GRPO on AIME 2024 and 2025 across models ranging from 4B to 30B parameters.The approach yields only modest or inconsistent gains on challenging mathematical problems, despite improvements in instruction-following, scientific QA, and tool-use.
  • Motivation: Privileged verified solutions make teachers overconfident on implied structural and verifiable tokens while suppressing deliberation tokens such as “Wait,” “Let,” and “Maybe.”This structural shortcut bias explains why the student is pulled away from tokens that support re-examining alternatives.
  • Method: AntiSD ascends the student–teacher divergence instead of descending it, reversing the per-token sign and producing a naturally bounded one-step advantage.The method reverses the signal by ascending Jensen-Shannon divergence between student and teacher.
  • Results: 2 to 10x fewer training steps are needed for AntiSD to reach the GRPO baseline’s accuracy, with final accuracy improving by up to 11.5 points.An entropy-triggered gate disables the AntiSD term once the teacher’s per-token entropy collapses, making it a drop-in replacement for default self-distillation.

2 Preliminaries

The paper formulates reasoning RL with group-normalized sequence-level advantages from verifiable rewards and augments GRPO with per-token self-distillation signals. A shared model acts as both rollout-generating student and privileged-context teacher, with the signals combined additively.

  • Setup: RLVR scores each sampled trajectory with a scalar verifiable reward, while GRPO uses a group-normalized sequence-level advantage as the policy-gradient signal.The advantage normalizes each rollout reward by the within-group mean and standard deviation.
  • On-policy self-distillation: On-policy self-distillation uses privileged training-time context, such as a verified solution or environment feedback, that is unavailable at inference.The same network generates the rollout as the student and scores it under richer conditioning as the teacher.
  • On-policy self-distillation: Self-distillation supplies a per-token signal based on divergences between the student and teacher, with the choice of f determining the resulting token-level advantage.The formulation is presented as one member of a broader family of per-token f-divergences.
  • On-policy self-distillation: The adopted objective combines the trajectory-level advantage and per-token distillation contribution additively using a positive mixing weight λ.The per-token term is δt, written in policy-gradient form from −∇θLSD.

3 Anti-Self-Distillation

Anti-Self-Distillation reverses the default self-distillation signal because conditional PMI rewards shortcut tokens while suppressing deliberation tokens. It ascends a bounded Jensen-Shannon divergence and uses an entropy-triggered gate to disable the term after teacher collapse.

  • 3.1 Motivation: Conditional PMI shows that default self-distillation rewards tokens whose probability privileged context raises and penalizes tokens whose probability it lowers.The signal is δt = +ut, where ut > 0 indicates that context raises token probability and ut < 0 indicates that it lowers it.
  • 3.1 Motivation: Shortcut tokens such as “Given” and “succeeds” are rewarded, whereas deliberation tokens such as “Wait” and “Maybe” are penalized because the privileged context commits to a solution.Figure 2 shows these regimes alternating within rollouts and forming two off-diagonal lobes with opposite ut signs.
  • 3.2 Anti-Self-Distillation: AntiSD reverses the gradient direction and ascends Jensen-Shannon divergence instead of reverse KL to counteract the wrong polarity and rollout asymmetry.Its advantage is capped on the over-sampled deliberation side while remaining linear on the under-sampled shortcut side.
  • 3.2 Anti-Self-Distillation: 2 log 2 caps the AntiSD advantage on the deliberation side, absorbing ut ≤−20 spikes while preserving proportional suppression of extreme shortcut tokens.The cap targets the over-sampled, heavy-tailed deliberation lobe identified in the PMI analysis.
  • 3.2 Anti-Self-Distillation: AntiSD disables its divergence term when batch-median teacher entropy H falls below τdown and re-enables it at Hwarm after entropy recovers.This Schmitt-trigger gate prevents low-entropy teacher probabilities from making ut reflect numerical floors rather than conditional information.

4 Experiments

Across five 4B–30B models, AntiSD reaches GRPO accuracy 2–10× faster and improves final math accuracy by 2.1–11.5 points. Its gains extend to code reasoning and reflect earlier, more stable training dynamics, while ablations identify sign reversal and entropy gating as key components.

  • Main results: AntiSD reaches GRPO’s accuracy 2–10× faster across all five models and exceeds GRPO’s final mean accuracy by +2.1 to +11.5 points.The largest speedups occur on weaker baselines, while the final-accuracy advantage remains positive on the strongest models.
  • Main results: On Qwen3-8B, AntiSD’s HMMT25 pass@k lead over GRPO remains ∼7–10 points at k = 32 after reaching ∼13 points at k = 1.The sustained gap and non-converging high-k curves indicate expanded problem coverage rather than only reduced variance.
  • Code reasoning: AntiSD improves over GRPO by +1.2 points on HumanEval+ and +2.3 on MBPP+ for Qwen3-8B, transferring its per-token mechanism beyond math reasoning.The gains are smaller than on math but consistent in direction on the Dolci-RLZero code RL setup.
  • Training dynamics: AntiSD’s truncation-corrected train reward rises from ∼0.5 to ∼0.95 within ∼30 steps, whereas GRPO reaches that regime only after ∼150 steps and SD never does.AntiSD remains in a stable middle entropy band, while SD can collapse toward ∼0.1 nats or inflate past 1 nat per token.
  • Ablations: Removing sign reversal drops Qwen3-8B Avg from 65.7 to 30.6, while removing the entropy gate causes model-conditional collapse near step 90 on Qwen models.The No-gate configuration survives 200 steps on Olmo3-7B-IT, showing why entropy-triggered stabilization is cross-model insurance.
  • Continual training: Continual AntiSD matches the from-base peak at 65.0 vs 65.7 Avg using 30 post-resume steps, compared with 180 steps from base.This shows AntiSD can build on a saturated GRPO checkpoint rather than replace it.

5 Related Work

Prior work studies on-policy self-distillation, diagnoses its reasoning failures, and develops process-reward approaches for sparse credit assignment. AntiSD is positioned within these lines as a method related to privileged-context distillation and per-token process signals.

  • On-policy self-distillation: On-policy self-distillation methods use a teacher–student log-ratio, with the teacher formed by conditioning the student on verified solutions and environment feedback.They extend on-policy distillation with an external teacher and learning under privileged information, while sharing the same gradient direction.
  • Diagnoses of self-distillation: Self-distillation has been reported to degrade reasoning capability, while existing methods mainly evaluate simpler benchmarks rather than AIME / HMMT-class problems.Related diagnostics also document teacher–student capability gaps and distribution mismatch.
  • Process reward models and reward shaping: Process reward models score intermediate reasoning steps to address sparse credit assignment in RLVR, using human annotations or Monte Carlo rollout estimates.Implicit-reward methods such as PRIME derive process rewards from preference signals jointly with the policy.
  • Process reward models and reward shaping: AntiSD’s per-token signal is structurally a process reward model, but it is training-free.The supplied passage frames AntiSD’s signal as a training-free process-reward mechanism.

6 Conclusion

The paper identifies conditional pointwise mutual information as the per-token signal behind default on-policy self-distillation’s structural shortcut bias. Anti-Self-Distillation counters this bias by inverting the gradient direction and replacing reverse-KL descent with bounded Jensen–Shannon ascent.

  • Default on-policy self-distillation uses conditional pointwise mutual information between the next token and privileged context as its per-token signal.
  • The signal creates a structural shortcut bias that rewards context-implied tokens and penalises deliberation tokens driving search.
  • Anti-Self-Distillation inverts the gradient direction from the first training step, replacing reverse-KL descent with bounded Jensen–Shannon ascent.

Supplementary Material · A Proofs and Deferred Statements

The appendix proves AntiSD’s reverse-KL, PMI, potential-shaping, and JSD formulations, then establishes the properties that determine its token-level behavior and entropy-gated implementation. These results show how AntiSD reverses self-distillation’s per-token sign, bounds deliberation-side advantages, and preserves optimal policies under potential-based shaping.

  • A Proofs and Deferred Statements: The appendix derives the reverse-KL gradient identity, PMI characterization, trajectory-level telescope, JSD f-divergence form, and required properties of φ.These derivations correspond to Equations (3), (4), and (6), plus Lemmas 1–5.
  • A Proofs and Deferred Statements: Under parameter sharing, the teacher–student log-probability gap u_t equals conditional PMI between the generated token and privileged context.Bayes’ rule yields the conditional pointwise mutual information characterization.
  • A Proofs and Deferred Statements: Summing u_t over a trajectory telescopes to sequence-level PMI, making each token contribution an increment of the potential Φ_t = log πθ(c | x, y≤t).The resulting augmented advantage is potential-based reward shaping and leaves the set of optimal policies invariant for any underlying scalar reward.
  • A Proofs and Deferred Statements: The JSD f-divergence generator satisfies f′(e^−u) = −φ(u), recovering the AntiSD advantage from the divergence formulation.Substitution gives f′(e^−u) = 1/2(log 2 − softplus(u)) = −φ(u).
  • A Proofs and Deferred Statements: φ is strictly increasing, satisfies φ(0) = 0, and is bounded below by −1/2 log 2 while remaining unbounded above.Its derivative is φ′(u) = 1/2σ(u) > 0, and the lower bound is attained only as u approaches negative infinity.
  • A Proofs and Deferred Statements: 1/2 log 2 bounds the AntiSD advantage on the deliberation side, where u_t < 0, while −φ reverses the sign so deliberation tokens receive positive and shortcut tokens negative advantage.The cap re-balances contributions against the lighter shortcut side, and the entropy gate disables the term when u_t becomes floor-level noise.
  • A Proofs and Deferred Statements: The one-step algorithm computes student and stop-gradient teacher log-probabilities, sets u_i,t to their difference, and applies λ = g · λ_max after an entropy-based gate update.The gate turns on when median teacher entropy reaches H_warm, turns off below τ_down, and updates policy parameters using A_i,t = A_seq_i − λ · stopgrad(φ_i,t).

B Hyperparameters · C Self-Teacher Context Examples · D Additional Experiments

The appendix specifies shared training settings, illustrates the self-teacher’s contrastive context for math and code, and collects deferred experimental probes referenced by the main paper. AntiSD preserves the GRPO/SD configuration while adding gate parameters, and its feedback encoding follows each task’s reward structure.

  • B Hyperparameters: Five language models share a 32K maximum training sequence length, while Olmo-3-7B-Think doubles its evaluation-time maximum length for longer chains-of-thought.The evaluated models are Qwen3-8B, Qwen3-4B-Instruct-2507, Olmo-3-7B-Instruct, Olmo-3-7B-Think, and Qwen3-30B-A3B.
  • B Hyperparameters: AntiSD shares the GRPO/SD configuration and adds only gate parameters, with Hwarm and τdown auto-calibrated.The hyperparameter table identifies the added gate parameters as the only AntiSD-specific configuration block.
  • C Self-Teacher Context Examples: The self-teacher re-evaluates the student response after receiving the original prompt, a verified solution, correctness feedback, and an instruction to re-solve.The verified solution comes from a successful same-batch rollout when available, otherwise from the dataset.
  • C Self-Teacher Context Examples: The same self-teacher template is used for math and code, with only the verified-solution and feedback strings differing by task.Examples cover math benchmarks including AIME, HMMT, and MinervaMath, and code benchmarks including LiveCodeBench v6 and Dolci-RLZero.
  • C Self-Teacher Context Examples: The context places a correct reference beside the student’s assessed attempt, creating a deliberate contrastive signal for teacher re-solving.The previous assessment describes the correctness of the student’s actual rollout, not the verified solution shown above it.
  • C Self-Teacher Context Examples: Math uses binary correctness feedback, whereas code uses the continuous fraction of tests passed, matching each task’s reward structure.Math feedback is “Your answer is correct.” or “Your answer is incorrect.”; code feedback reports N of M tests passed.
  • D Additional Experiments: The additional-experiments section gathers results deferred from the main paper, with subsections corresponding to experimental probes summarized in Section 4.These experiments are collected in the appendix for space.

D.1 Component sensitivity on Qwen3-8B

On Qwen3-8B, reversed-KL ascent collapses, while gating is necessary for stable AntiSD training. The optimal entropy threshold is model-specific: 0.90 slightly beats canonical 0.93, whereas 0.95 lowers the peak and slows ignition.

  • Rev. KL ascent collapses on Qwen3-8B, reaching 30.6 Avg, or −35.1 pp from canonical.
  • The gate is necessary: the no-gate run peaks transiently near step ∼40 before collapsing by step ∼90.The reported peak is taken from 8 pre-collapse checkpoints.
  • 65.9 Avg at threshold 0.90 slightly exceeds 65.7 at canonical 0.93, while 0.95 lowers the peak to 65.4 and slows ignition by ∼4×.This threshold sensitivity differs from the Qwen3-4B-IT-2507 result, where loosening to 0.90 caused a −8.3 pp drop.

D.2 Continual AntiSD on Qwen3-4B-IT-2507

On Qwen3-4B-IT-2507, continual AntiSD briefly reaches 61.9 at step +20 before settling near 60.5, below the from-base peak of 62.8. The plateau may reflect GRPO basin constraints or a conservative gate threshold, but continual AntiSD still delivers a strong fraction of from-base improvement at lower cost.

  • Results: 61.9 is the continual AntiSD peak at step +20, before performance drifts to a plateau of ≈60.5.The plateau is 2.3 pp below the from-base AntiSD peak of 62.8.
  • Results: 2.3 pp separates the continual plateau from the from-base AntiSD peak on Qwen3-4B-IT-2507.The plateau is ≈60.5, while the from-base peak is 62.8.
  • Interpretation: The plateau may indicate that GRPO admits only part of AntiSD’s deliberation pressure or that the recalibrated gate threshold is mildly conservative.The authors do not disentangle these interpretations and state that residual AntiSD gain remains unrealised.
  • Practical takeaway: Continual AntiSD provides a strong fraction of the from-base improvement at a fraction of the cost.This is the authors’ practical takeaway for the smaller model.

E Limitations and Broader Impacts

AntiSD’s limitations concern the local scope of its conditional-PMI account and the need to test longer horizons, richer feedback, larger models, and multimodal settings. Its broader impacts include stronger open-weight reasoning models and lower training cost, alongside dual-use risks.

  • Limitations: The conditional-PMI account is a local per-step characterization, not a global-optimum statement about the combined objective.The long-horizon dynamics of the full ascent-plus-gate update remain to be understood.
  • Scope and extensions: Future tests should examine multi-call interactions, longer-horizon code reasoning, and richer privileged-context content.Suggested settings include LiveCodeBench v6, process-level critiques, partial-credit annotations, and rationale-comparison rankings.
  • Scope and extensions: Larger model scales beyond 30B and multimodal conditioning are natural settings for testing whether conditional PMI remains the dominant credit-assignment signal.These settings extend evaluation beyond the current scale and conditioning regime.
  • Broader impacts: 2 to 10× fewer steps to reach a given accuracy is among AntiSD’s positive broader impacts, alongside stronger open-weight reasoning models.The method may also provide a clearer theoretical basis for understanding default self-distillation’s underperformance and inform future training-free PRM designs.
  • Broader impacts: AntiSD’s reasoning gains are dual-use because stronger reasoning models can also be applied to adversarial purposes.The passage identifies this as a negative impact associated with improvements to LLM reasoning.
Loading 2605.11609v1…