Every Token Needs Its Own Teacher: Multi-teacher Adaptive On-policy Self-distillation

• Highly informative references can induce overly sharp teacher distributions that are poorly aligned with the student’s learning dynamics.Prior adaptive OPSD methods vary reference exposure across responses, but keep supervision fixed across tokens within each response.Source: §1 · pp. 1–2
• Different teachers can provide similar learning signals at some positions but opposing signals at others within the same response.This motivates considering a preferred teacher separately for each token.Source: §1 · p. 2, Figure 1 · p. 2
• A teacher-intervention pilot study found that preferences differ across tokens and that majority voting and KL-based selection can identify a preferred teacher.The study uses accuracy gain from a student-teacher-student relay over pure student generation as a proxy for teacher preference at the selected token.Source: §1 · p. 2
• On mathematical reasoning tasks, MTA-OPSD outperforms existing OPSD methods, with an average gain of up to +5.9 over the base model.The paper evaluates models from the Qwen3 family.Source: §1 · p. 2
• MTA-OPSD builds a pool of candidate teachers by varying reference truncation and exposure, then lets each token select a teacher through majority voting followed by KL-based selection.The paper describes this as its first token-level adaptive OPSD method.Source: §1 · pp. 2–3

2.1 ON-POLICY DISTILLATION AND ON-POLICY SELF-DISTILLATION

• On-policy distillation transfers teacher knowledge through dense supervision over student-generated trajectories, while OPSD uses the same model for student and teacher with privileged reference access for the teacher.Source: §2.1 · p. 3
• Highly informative references can produce overly sharp teacher distributions that mismatch student learning dynamics, motivating adaptive OPSD.Source: §2.1 · p. 3

2.2 ADAPTIVE ON-POLICY SELF-DISTILLATION

• Adaptive OPSD methods control reference content or exposure ratio to avoid overly sharp supervision while retaining dense learning signals.Examples include mechanisms such as learned reference-exposure schedules.Source: §2.2 · p. 3
• Existing adaptive methods use the same teacher and privileged reference for all tokens in a response, whereas the paper proposes token-level multi-teacher adaptation for distinct token preferences.Source: §2.2 · p. 3

• On-policy distillation minimizes token-level KL divergence between teacher and student next-token distributions along responses sampled by the student.The setup defines the response as a sequence of generated tokens conditioned on the prompt and preceding tokens.Source: §3 · p. 3
• The next-token distribution is conditioned on the prompt and preceding tokens, and c_i abbreviates that context.Source: §3 · p. 4
• In OPSD, teacher and student share the same model, but the teacher receives a privileged reference appended to the prompt as its conditioning context.The teacher distribution is the model distribution conditioned on the student context and reference.Source: §3 · p. 4
• The response is generated solely by the student, while teacher and student differ only in their conditioning contexts.Source: §3 · p. 4

4 PILOT STUDY: TOKEN MATTERS

• The pilot preserves a student-generated prefix, lets a candidate teacher generate a short subsequence, then returns control to the student to complete the response.Intervention gains are assessed against the pure student response as a proxy for teacher preference.Source: §4 · p. 4

4.1 SETUP

• The setup tests seven intervention positions per rollout and seven candidate teachers, varying token positions, reference truncation strategies, and exposure ratios.Positions include 25%, 50%, and 75% of the rollout, plus entropy and student–teacher KL extremes; teachers include prefix, suffix, and random-span truncation at 0.5 or 0.75 exposure, plus full reference.Source: §4.1 · p. 5
• The study uses Qwen3-1.7B and Qwen3-4B on 64 medium-difficulty-or-harder DeepMath-103k prompts, with intervention lengths k = 64, 128, and 256.The authors evaluate 75,264 generations across the configurations.Source: §4.1 · p. 5

4.2 RESULTS AND FINDINGS

• For Qwen3-1.7B with k = 256, preferred-teacher overlap between regular-spaced tokens is approximately 50%, indicating that preferences differ within a rollout.The pilot defines the preferred teacher as the candidate producing the largest intervention gain.Source: Table 1 · p. 5, §4.2 · p. 5
• Around 80%−90% of preferred teachers fall within the majority subset, whose teachers achieve larger gains per teacher than the minority subset for Qwen3-4B with k = 256.The largest-gain teacher belongs to the majority subset in around 80%−90% of cases.Source: Table 2 · p. 5, Table 3 · p. 5, §4.2 · p. 6
• At regular token positions, the farthest teacher by student–teacher KL distance consistently produces greater average gains than the nearest teacher.The authors identify KL distance as a useful criterion for token-level teacher preference.Source: Table 4 · p. 6, §4.2 · p. 6

5 METHOD

• MTA-OPSD assigns each token its preferred teacher from a multi-teacher pool through majority voting followed by KL-based selection.The candidate teachers are conditioned on different variants of the privileged reference.Source: §5 · p. 6, Figure 3 · p. 7

5.1 MULTI-TEACHER POOL

• The pool contains teachers sharing the student’s model but conditioned on references that differ in truncation strategy and exposure ratio.The pool supplies diverse supervision signals but does not itself determine which teacher supervises each token.Source: §5.1 · p. 6

5.2 MAJORITY VOTING

• Majority voting compares each candidate teacher’s support for the student’s token choice and retains teachers aligned with the majority direction.The procedure is performed independently at every token position.Source: §5.2 · pp. 6–7

5.3 KL-BASED SELECTION

• From the majority subset, KL-based selection chooses the teacher with the largest KL distance from the student distribution for that token.The selected teacher can therefore vary across tokens within one response.Source: §5.3 · p. 7

5.4 DISCUSSION

• Voting filters teachers by directional consensus before KL selects informativeness; KL distance is not used alone to judge teacher quality.When the majority opposes the sampled token, the farthest teacher may provide a stronger corrective signal; when it supports the token, the selected teacher can reinforce it while preserving other plausible predictions.Source: §5.4 · p. 8

6.1 EXPERIMENT SETUP

• The evaluation uses Qwen3-1.7B, Qwen3-4B, and Qwen3-8B on AMC23, AIME24, and AIME25 after training on DeepMath-103K.Baselines include two fixed-teacher OPSD methods and Paint, an adaptive OPSD method.Source: §6.1 · p. 8

6.2 MAIN RESULTS

• +5.9, +1.5, and +2.8 average improvement over the base model for Qwen3-1.7B, Qwen3-4B, and Qwen3-8B, respectively; MTA-OPSD leads on average at every scale.It achieves or matches the best result on nearly every model-benchmark combination, with particularly consistent and strong improvements on AIME25.Source: §6.2 · pp. 8–9, Table 5 · p. 9

6.3 ABLATION STUDIES

• Against training with any individual pool teacher, MTA-OPSD scores highest on both average and individual benchmarks.The authors interpret this as evidence that the pool provides diverse, sufficient, and complementary learning signals.Source: §6.3 · p. 9, Table 6 · p. 9
• MTA-OPSD outperforms both random per-token teacher selection and averaging distillation losses across teachers on every benchmark.The comparison indicates that gains do not arise solely from having multiple teachers; selecting a suitable teacher for each token is critical.Source: Table 7 · p. 9, §6.3 · p. 10
• Teacher assignments are relatively evenly distributed, and higher reference exposure does not necessarily produce a larger student-teacher KL distance.The authors note that adjusting exposure alone cannot determine teacher suitability.Source: §6.3 · p. 10
• The default two-step design outperforms all ablation variants, supporting majority voting as a reliable-subset filter followed by KL-based selection.The second row of Table 8 is the default setting.Source: Table 8 · p. 9, §6.3 · p. 10

7 CONCLUSION

• A teacher-intervention study finds that teacher preferences vary across tokens, motivating token-level selection through majority voting and KL-based selection.The paper identifies self-evolving teacher pools and more stable multi-teacher training as directions for future work.Source: §7 · p. 10

A VISUALIZATION DETAILS

• The three teachers in Figure 1 use prefix-0.5, suffix-0.5, and random-span-0.5 configurations.Each configuration uses an exposure ratio of 0.5.Source: A VISUALIZATION DETAILS · p. 14
• Figure 4 shows the entire student response associated with Figure 1.Source: A VISUALIZATION DETAILS · p. 14, Figure 4 · p. 14
• Use Figure 4 to inspect the full response rather than only the portion shown in Figure 1.Source: A VISUALIZATION DETAILS · p. 14, Figure 4 · p. 14

B ADDITIONAL MAJORITY VOTING DETAILS

• Equal positive and negative votes trigger a tie-break based on the magnitudes of the corresponding advantage signals.A tie may occur when the multi-teacher pool has an even number of teachers.Source: B ADDITIONAL MAJORITY VOTING DETAILS · p. 14
• The tie-break selects the group with the larger total absolute advantage.The majority teacher subset is then constructed using Equation (14).Source: B ADDITIONAL MAJORITY VOTING DETAILS · p. 14
• This rule retains the direction supported by the stronger aggregate teacher signal.Source: B ADDITIONAL MAJORITY VOTING DETAILS · p. 14

C ADDITIONAL EXPERIMENT SETTINGS

• Experiments use 4 H200 GPUs, a 1 × 10−6 learning rate, global batch size 32, and full-parameter training without LoRA.Methods use the veRL PPO trainer with FSDP2 actor sharding and vLLM rollouts; distillation is restricted to the teacher distribution’s top 100 entries.Source: C ADDITIONAL EXPERIMENT SETTINGS · p. 15
• Training problems use difficulty levels 5–7 for Qwen3-1.7B, 6–8 for Qwen3-4B, and 7–10 for Qwen3-8B.Source: C ADDITIONAL EXPERIMENT SETTINGS · p. 15

D QUALITY VISUALIZATION

• Figure 5 provides a qualitative comparison and quality visualization.Source: D QUALITY VISUALIZATION · p. 15, Figure 5 · p. 15

E ADDITIONAL DETAILS ABOUT KL-DISTANCE

• 49.1% of Qwen3-4B suffix-truncation cases at 0.75 exposure had larger KL distance from the student, versus 50.9% at 0.5 exposure.Thus, under the same truncation strategy, a larger reference-exposure ratio did not lead to a larger student-teacher KL distance.Source: E ADDITIONAL DETAILS ABOUT KL-DISTANCE · p. 16

F MULTI-TEACHER POOL VISUALIZATION

• Figures 6–9 provide visualizations of the multi-teacher pool.Source: F MULTI-TEACHER POOL VISUALIZATION · p. 16, Figure 6 · p. 17, Figure 7 · p. 18