Source-linked AI summary
Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning
Zihao Han, Tiangang Zhang, Huaibin Wang, Yilun Sun
TL;DR
Existing self-distillation methods expose teachers to full reference reasoning without testing whether this gives optimal supervision. ATESD learns training-state-conditioned teacher exposure and consistently improves OPSD across three math benchmarks and three model scales, with gains of up to +2.33 Average@12 points.
Problem
Existing methods universally give the teacher full reference reasoning, leaving whether complete exposure provides optimal supervision throughout training untested.
Method
ATESD learns teacher exposure with a training-state-conditioned Beta-policy controller optimized by discounted learning progress over held exposure decisions.
Results
+0.95, +2.05, and +2.33 Average@12 points over OPSD on Qwen3-1.7B, 4B, and 8B, respectively, across AIME 2024, AIME 2025, and HMMT 2025.
Takeaways & Limitations
ATESD establishes adaptive teacher exposure as an effective training control for reasoning self-distillation under the evaluated benchmarks, scales, and protocol.
Takeaways & Limitations
The controller selects one global exposure for all samples during each hold period, and its fixed lookahead reward and evaluated domains leave broader generalization open.
Abstract
from arXiv · showhide
On-policy self-distillation has become a strong recipe for LLM reasoning, where a privileged teacher supervises the student's own rollouts while conditioning on the reference solution. A design choice shared by nearly all such methods, however, has gone unquestioned: the teacher always sees the full reference reasoning. We argue that this default itself is part of the problem and identify a teacher-side exposure mismatch: when the teacher conditions on reasoning far beyond the student's current competence, the resulting token targets become too strong to absorb. A controlled fixed-exposure sweep makes this concrete on two fronts: 1) full exposure is not reliably the best choice, and 2) student-teacher mismatch grows monotonically as the teacher sees more privileged reasoning. This motivates treating teacher exposure not as a fixed hyperparameter but as a learnable training-time control variable. We therefore propose Adaptive Teacher Exposure for Self-Distillation (ATESD). ATESD models the reveal ratio with a lightweight Beta-policy controller conditioned on compact training-state statistics, and uses one sampled exposure for a short hold window of student updates. To make this exposure controller learnable, we optimize it with a discounted learning-progress reward that scores each held decision by its effect on the student's future improvement rather than its immediate loss change, addressing the delayed credit assignment induced by on-policy distillation. Experiments on AIME 24, AIME 25, and HMMT 25 across Qwen3-{1.7B, 4B, 8B} show that ATESD consistently outperforms competitive self-distillation and RL baselines, improving over OPSD by +0.95, +2.05, and +2.33 Average@12 points respectively, and establishing adaptive teacher exposure as an effective new axis for reasoning self-distillation.
1 Introduction
The introduction identifies teacher-side exposure mismatch as a limitation of full-reference reasoning in on-policy self-distillation. It motivates ATESD, which learns teacher exposure from delayed student learning effects rather than treating exposure as fixed or optimizing immediate loss alone.
- Motivation: Teacher-side exposure mismatch arises when privileged reasoning exceeds the student’s capability, making full-reference supervision difficult to absorb on hard problems.On easy problems, the teacher’s reasoning can remain within the student’s capability and distillation succeeds.
- Controlled evidence: α∗=0.5 consistently outperforms full exposure across seeds, while teacher–student mismatch grows monotonically with α.Difficulty-binned results further indicate that different learning regimes prefer different tested exposure levels.
- Controller learning: Naive one-step rewards miscredit exposure decisions because their benefits emerge only after subsequent student optimization steps.The introduction therefore frames exposure control as a delayed-credit-assignment problem requiring rewards based on future learning effects.
- ATESD: ATESD parameterizes continuous exposure α ∈[0, 1] with a lightweight Beta-policy controller conditioned on compact training-state statistics.The controller selects one global exposure per hold window and updates more slowly than the student through a two-timescale REINFORCE schedule.
2 Related Work
Prior work established on-policy self-distillation for reasoning, while adaptive distillation mainly changes the student’s view or teacher strategy. This paper instead makes the teacher’s privileged-reasoning exposure a continuous, learnable, training-state-conditioned control variable.
- On-Policy Self-Distillation: On-policy distillation trains students on their own rollouts under teacher guidance, avoiding the student-side distribution mismatch of offline self-distillation.OPSD extends this paradigm by using a single model as both teacher and student.
- Teacher Exposure: Prior work treats teacher access to privileged reasoning as a fixed binary choice—full or none—while adapting only the student side.This paper identifies teacher exposure as an overlooked design dimension.
- Teacher Exposure: The proposed approach builds on OPSD by turning teacher exposure into a continuous, learnable control variable conditioned on training state.This reframes privileged context from a fixed default into a training-time decision.
- Adaptive Distillation: Adaptive distillation has primarily modulated the student’s view through curricula, dynamic temperatures, adaptive signals, or teacher strategies tied to student progress.Related reinforcement-learning methods improve LLM reasoning with PPO and DPO.
3 Preliminaries
OPSD uses a problem-only student and a full-reference teacher to distill on-policy rollouts, but full teacher exposure is not reliably optimal. Increasing exposure widens teacher–student mismatch, motivating exposure as a learnable training-time control variable.
- OPSD backbone: OPSD defines a problem-conditioned student and a full-reference-conditioned teacher, training on the student’s on-policy rollout with per-token forward KL.The teacher is a frozen dense target, while gradients flow only through the student.
- Motivation: Full exposure is a fixed OPSD default that prior work had not tested as the best supervision choice throughout training.The paper formalizes exposure to enable direct empirical study across α.
- Empirical exposure sweep: α∗=0.5 is the best fixed exposure on AIME 2024 with Qwen3-1.7B, rather than full exposure.Figure 2 reports three seeds and mean±s.e.m.; mismatch proxies increase monotonically with α.
- Teacher-side exposure mismatch: As α increases, teacher targets become sharper and both on-policy KD loss tail and top-1 disagreement increase, widening the supervision gap.The resulting teacher-side exposure mismatch can push targets outside the student’s learnable range.
- ATESD motivation: ATESD treats α as a learnable training-time control variable while preserving student rollout collection and changing only the privileged reasoning available to the teacher.The exposure fraction truncates the reasoning prefix while preserving the final answer.
4 Method: ATESD
ATESD replaces full-reference teacher conditioning with continuous, exposure-controlled privileged context and learns the exposure level from training-state feedback. A Beta-policy controller holds each sampled exposure across multiple student updates and credits it using delayed learning progress plus teacher-grounded scores.
- Exposure-controlled teacher: ATESD replaces binary full-reference exposure with a continuous α_t-controlled teacher context, truncating only the reasoning prefix while retaining the final boxed answer.This isolates privileged reasoning exposure while preserving the answer constraint at every exposure level.
- Delayed controller learning: ATESD scores held exposure decisions after an L-step lookahead using discounted student learning progress and a teacher-grounded score over verified reference tokens.This delayed reward addresses effects that may not appear in immediate loss changes, while centered advantages penalize below-average held actions.
- Exposure-controlled teacher: On-policy student rollouts are scored under shared sampled prefixes, while α_t changes only the teacher’s privileged information in the token-level KL objective.Gradients flow only through the student; low exposure weakens reasoning context, whereas high exposure recovers the full-reference teacher.
- Adaptive exposure controller: The controller uses a training-state-conditioned Beta policy whose compact state includes progress, recent exposure, loss and mismatch EMAs, probe-NLL EMA, and student self-confidence.A lightweight MLP maps this state to Beta concentration parameters, producing a continuous exposure action clipped to [α_min, α_max].
- Adaptive exposure controller: Each sampled exposure α_t is held fixed for H student updates, making one decision a short global episode credited over the entire held window.The Beta distribution is constrained to be unimodal, with its mean representing preferred exposure and concentration representing confidence.
5 Experiments
Experiments show that ATESD improves OPSD across Qwen3 scales and benchmarks while reducing teacher–student supervision mismatch. Ablations support learned interior exposure and delayed credit as key components of the controller.
- Primary results: ATESD improves OPSD by +0.95, +2.05, and +2.33 Average@12 points on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, respectively.It achieves the best average performance at every evaluated model scale under the OPSD reporting protocol.
- Primary results: 65.65 Average@12 is reached by the strongest 4B run, gaining 2.05 points over OPSD and 2.95 points over GRPO.The 8B run reaches 67.13 average performance across the same benchmark suite.
- Mechanism ablations: 0.0061 mean KL results from reducing exposure from α = 1.0 to α = 0.3, versus 0.0136 under full exposure on a fixed positive trajectory.Maximum KL falls from 0.2432 to 0.0645, while the position-26 spike falls from 0.2432 to 0.0098.
- Mechanism ablations: The learned Beta exposure distribution starts broad and concentrates away from both no-reference and full-exposure extremes during training.This indicates a training-state-level policy rather than a fixed or uncontrolled exposure choice.
- Controller ablations: 59.17 Average@12 is achieved by the full delayed reward, compared with 52.22 for immediate feedback, 56.11 with delayed credit, and 58.06 with discounted lookahead.Adding the teacher-grounded score produces the final improvement in the delayed-credit ablation.
- Controller ablations: 59.17 Average@12 from the learned policy exceeds 57.20 for OPSD full exposure, 57.44 for the best fixed exposure, and 54.94 for uncontrolled stochastic exposure.The comparison rejects both manually tuned constant exposure and stochasticity alone as explanations for the gain.
6 Conclusion
The paper identifies teacher-side exposure mismatch as a bottleneck in on-policy self-distillation and proposes ATESD, a training-state-conditioned controller optimized with discounted learning progress. Across reasoning benchmarks, ATESD consistently improves over OPSD, while ablations validate exposure control, delayed credit, and feedback-driven selection.
- Conclusion: Full teacher exposure is suboptimal, while teacher–student distribution mismatch grows monotonically with exposure level.These findings identify teacher-side exposure mismatch as an overlooked bottleneck in on-policy self-distillation.
- Conclusion: ATESD learns teacher exposure with a training-state-conditioned Beta controller optimized by a discounted learning-progress reward.The method addresses exposure selection through a learned controller rather than a fixed exposure setting.
- Conclusion: ATESD consistently improves over OPSD across AIME 2024, AIME 2025, and HMMT 2025 on Qwen3-{1.7B, 4B, 8B}.The comparison uses the same evaluation protocol.
- Conclusion: Ablations show that exposure control reduces positive-trajectory mismatch, delayed credit enables exposure learning, and feedback-driven selection outperforms fixed or uncontrolled choices.These effects are demonstrated under the same student rollout protocol during training.
- Limitations and future directions: The current controller selects one global α for all samples within a hold period, motivating per-sample or difficulty-aware exposure conditioned on difficulty or student confidence.The present method deliberately studies the simpler training-state-level controller first.