From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation

Introduction

• The study compares gradients, optimizer steps, and per-task learning curves for Qwen3-1.7B with four RL-trained domain teachers, with additional SmolLM3-3B diagnostics.The teachers and student share the same initialization.Source: §1 · p. 1
• Global token averaging favors longer-response domains, and equal domain weights still retain length weighting within each domain; no rule wins on every task.Domain response averaging weights responses equally, and task-score effects depend on the distillation loss.Source: §1 · p. 2
• Adam’s first moment aligns updates despite differences in current gradients, while BF16 rounding leaves only a small fraction of changed FP32 master weights visible.The supplied passage reports that momentum-free SGD achieves higher four-task mean scores than Adam across all three averaging rules.Source: §1 · p. 2
• Top-64 intersection KL closely matches Qwen’s full-vocabulary gradient, but mathematics accuracy is 2.6 points higher than PG with response averaging and 2.1 points lower with global token averaging.The effect of the closer gradient approximation therefore depends on the averaging rule.Source: §1 · p. 2

MOPD setup

• For a prompt from domain d, the student generates a response and that domain’s frozen teacher supplies next-token probabilities at each response prefix.A batch can pool responses from multiple domains, and each response averages its valid-token losses.Source: §2 · p. 2
• Responses are generated afresh each training step, but their text remains fixed while the loss is differentiated with respect to student parameters.Source: §2 · p. 2

Distillation losses

• Full-vocabulary KL uses all vocabulary tokens, whereas the sampled-token PG loss uses the token sampled by the student.The PG loss is a surrogate whose expected gradient equals the full reverse-KL gradient, although individual samples can differ.Source: §2 · pp. 2–3
• Top-k intersection KL restricts supervision to tokens appearing in both the student’s and teacher’s top-k sets.The probabilities are renormalized within the intersection; an empty intersection gives zero loss, and tokens outside it receive zero logit gradient.Source: §2 · p. 3

Domain weighting and loss normalization

• Global token averaging weights domains by token share, domain token averaging fixes domain weights but favors longer responses, and domain response averaging weights responses equally within each domain.Source: §2 · p. 3

3.1 LOSS AVERAGING CHANGES GRADIENTS AND PER-TASK SCORES, BUT BARELY THE MEAN

• DT fixes domain weights but retains length weighting within each domain, while DR removes the covariance shift caused when response length and gradients vary together.Open-MOPD’s token-share correction recovers DT’s weighting.Source: §3.1 · p. 3
• DT and DR gradients have a mean cosine of 0.68; after 500 steps, the four-task mean-score range across averaging rules is 0.28 points with Adam-PG, 0.06 with Adam-top-64, and 0.68 with SGD-PG.Source: §3.1 · p. 3
• GT assigns mathematics about 44% of loss weight; versus DR, it raises mathematics accuracy by 3.0 points and lowers science by 2.1 points with PG, while top-64 reverses the mathematics comparison.With top-64, mathematics falls by 1.8 points and code rises by 1.5 points versus DR; Table 1 reports scores by averaging and distillation loss.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 ADAM REDUCES DIFFERENCES IN UPDATE DIRECTION

• DT and DR gradients produce one-step Adam updates with mean cosine 0.96, and saved-state Adam yields smaller between-rule differences in held-out KL change than norm-matched SGD.Adam step norms become similar during training despite distinct gradient norms, while cumulative parameter changes remain different.Source: §3.2 · p. 4

3.3 RAISING THE RESPONSE CAP SHIFTS TOKEN-AVERAGED OBJECTIVES MOST

• At step 100, raising the response cap from 4K to 8K changes mean scores by GT +1.05, DT +0.32, and DR −0.83 percentage points.Relative raw-gradient differences follow GT > DT > DR; clipping and Adam can alter that ranking.Source: §3.3 · p. 4

4.1 BF16 ROUNDING HIDES WIDESPREAD FP32 CHANGES

• About 97% of FP32 master weights change under domain response averaging, versus 7–9% after BF16 rounding; across DT and GT, the BF16 fraction is 7–11%.Rounding makes widespread movement appear sparse; about one third of FP32 changes, versus about 4% in BF16, account for 90% of squared change.Source: §4.1 · p. 5

4.2 MOMENTUM HIDES DIFFERENCES BETWEEN TEACHERS

• Teacher updates have mean cosine above 0.83 with saved Adam moments, but the mean cosine of their zero-gradient-subtracted differences is below 0.01 on domain-specific inputs and 0.24–0.35 on shared prefixes.Resetting Adam’s first moment or using norm-matched SGD reduces the teacher-update cosine to near zero; Figure 3 also compares optimizer processing across averaging rules.Source: §4.2 · pp. 5–6, Figure 3 · p. 6

4.3 MOMENTUM-FREE SGD ACHIEVES HIGHER MEAN SCORES

• 39.94 vs. 38.91 under DR, 39.46 vs. 38.63 under DT, and 39.26 vs. 38.79 under GT: momentum-free SGD scores higher than Adam on four-task means.These comparisons use the PG loss after full training runs.Source: §4.3 · p. 7

5.2 SUPERVISION BREADTH BARELY AFFECTS THE FRACTION OF CHANGED BF16 WEIGHTS

• PG, top-64, teacher-top-64, and full-vocabulary KL losses change nearly the same fraction of BF16 weights in one step, while Adam history strongly affects those counts.Even with zero current gradient, about 0.14% of BF16 weights change; resetting only the first moment lowers the fraction, while fully resetting Adam raises it.Source: §5.2 · pp. 7–8

5.1 TOP-64 INTERSECTIONS NEARLY RECOVER THE FULL-VOCABULARY GRADIENT

• More than 99.9% of student probability is retained on average by Qwen’s top-64 intersection, whose gradient direction nearly matches the full-vocabulary KL gradient.Averaging 1, 16, or 64 samples per fixed prefix raises cosine to the full-gradient reference from 0.54 to 0.87 and 0.97.Source: §5.1 · p. 7, Figure 6 · p. 8
• In SmolLM, top-64 retains 97.42% initially and 98.56% after training, yet 8.42% and 3.65% of weighted prefixes retain less than 90% of student probability.After training, relative gradient error spans 2.16–24.54% among diagnostic batches.Source: §5.1 · p. 7, Figure 7 · p. 8

5.3 TASK GAINS FROM TOP-k DEPEND ON THE AVERAGING RULE

• 2.6 percentage points higher mathematics accuracy than PG under response averaging, but 2.1 points lower under global token averaging; four-task means rise 0.16 and 0.21 points, respectively.Instruction-following and science scores decrease under response averaging, while code and instruction-following scores improve under global token averaging.Source: §5.3 · pp. 8–9, Figure 8 · p. 9

• The paper situates vocabulary truncation among divergence-based distillation and sparse or tail-aware OPD methods, comparing coverage, gradient fidelity, and task gains while accounting for sampling noise and optimizer processing.Source: §6 · p. 9
• Prior multi-teacher work addresses domain token shares and task balancing, while this paper derives response-weight effects through gradient covariance and measures Adam updates and task scores.Source: §6 · p. 9
• Related parameter and alignment studies include task arithmetic, concentrated or low-dimensional updates, and methods for conflicting gradients; this paper separates current supervision from Adam history and FP32 movement from BF16 sparsity.Source: §6 · p. 9

7 DISCUSSION

• The findings support choosing response weighting, vocabulary supervision, and optimizer settings together for the target tasks and training budget.Source: §7 · p. 9
• Closer full-vocabulary gradient approximation does not consistently improve task scores, and domain balancing retains within-domain length weighting while Adam history aligns updates.Source: §7 · p. 9
• The findings come from limited model families and scales; future work may test smaller Adam β1 and whether clipping or second-moment normalization explains differences in task outcomes.Source: §7 · p. 9

A.1 MODELS, DATA, AND TRAINING

• Four Qwen3-1.7B teachers specialize in mathematics, code, instruction following, and science after independent RL training from a common initialization.The main training comparisons use seeds 42–46.Source: §A.1 · p. 15
• Student training uses AdamW with a constant learning rate and samples one fresh response for each of 64 prompts per step.The response-length comparison raises the maximum response length from 4,096 to 8,192 tokens.Source: §A.1 · p. 15
• MOPD uses 16 prompts per domain, while single-teacher training uses all 64 prompts from one domain.Domain-balanced rules assign each domain a target weight of 0.25.Source: §A.1 · p. 15
• The four domain datasets draw on mathematics, competitive-coding, instruction-following, and STEM multiple-choice components of Nemotron-3-Nano-RL-Training-Blend.The mathematics pool combines DAPO-Math-17k and Skywork-OR1-RL-Data, excluding OmniMath.Source: §A.1 · p. 15, Table 2 · p. 15

A.2 EVALUATION AND SCORING

• Evaluation uses MATH-500, LiveCodeBench v6, IFBench strict, and GPQA Diamond, with cross-domain means weighting the four benchmarks equally.GPQA uses 32 responses per question; the other benchmarks use one greedy response per question.Source: §A.2 · p. 16

B LOSS AVERAGING: DERIVATION AND CONTROLS

• Global-token averaging weights longer-response domains more, while domain-token averaging removes that between-domain imbalance but retains within-domain length weighting.The covariance identity describes the within-domain gradient shift; domain-response averaging weights responses equally.Source: B LOSS AVERAGING: DERIVATION AND… · p. 16

B.1 FIXED-BATCH GRADIENT AND OPTIMIZER COMPARISONS

• DR–DT raw-gradient cosine ranges from 0.40 to 0.91, with mean 0.68, showing direction changes under response-length weighting despite fixed domain weights.The covariance identity is verified across 24 batch-domain groups, with maximum relative residual 1.53 × 10^-15.Source: §B.1 · p. 16

D.2 WHY WE USE TOP-k INTERSECTIONS

• Top-k intersection KL retains tokens shared by the student’s and teacher’s top-k sets, avoiding additional teacher queries for student-selected token IDs.The candidate budget applies to each model, but the supervised intersection may be smaller.Source: §D.2 · p. 17

E UNBIASEDNESS OF PG AT A FIXED PREFIX

• The PG surrogate is treated as unbiased at a fixed prefix, while individual sampled-token gradients can differ from the full reverse-KL gradient.The proposition concerns a token sampled from the student distribution with the prefix held fixed.Source: E UNBIASEDNESS OF PG AT A FIXED PREFIX · p. 18

F.1 COMPARING TEACHERS AT A SHARED STUDENT STATE

• Teacher comparisons use shared student states and optimizer moments, with both domain-specific responses and shared prefixes scored by each teacher.The comparisons include all six teacher pairs across four cached batches.Source: §F.1 · p. 18

F.2 COMPARING VOCABULARY SUPPORTS AT MATCHED PREFIXES

• Cumulative BF16 parameter changes remain much sparser than FP32 changes across PG and top-64 trajectories in single-teacher and MOPD training.The figure also tracks the fraction of coordinates containing 90% of squared displacement from initialization.Source: Figure 9 · p. 19

F.3 ADDITIONAL OPTIMIZER AND SAMPLING CONTROLS

• Mean PG gradient cosine to full KL rises from 0.54 to 0.87 and 0.97 with 1, 16, and 64 samples per prefix; top-64 exceeds 0.999.The comparison uses 12 fixed prefix subsets with two independent draws at each sample count.Source: §F.3 · p. 19

G.1 PROBABILITY MASS RETAINED BY THE CANDIDATE INTERSECTION

• Mean top-64 intersection mass exceeds 99.9% at each checkpoint, and mean retained fraction exceeds 99.97% in the sampled Qwen comparisons.Across 384 prefixes, the minimum intersection mass is 96.55%.Source: §G.1 · p. 20

G.2 COVERAGE DURING TOP-16 TRAINING

• Across 8,000 logged slices, mean top-16 intersection mass is 99.62%, retained fraction is 99.88%, and mean intersection size is 15.05.Candidate sets are fixed at rollout, while probabilities are measured during training.Source: §G.2 · p. 20
• High average top-16 coverage coexists with a lowest slice-mean mass of 86.67% and a largest slice-mean empty-intersection fraction of 1.85%.The mean empty-intersection fraction is 0.0027%.Source: §G.2 · p. 20

G.3 DOMAIN-LEVEL EFFECTS OF LOSS AVERAGING

• At step 250, instruction-following KL rises by 0.362/0.293/0.272 × 10^-3 under DR/DT/GT, while four-domain changes are 0.041/−0.002/−0.027 × 10^-3.DT and GT improve mean probe loss while moving instruction following away from its teacher.Source: §G.3 · p. 20

H.1 MODELS AND RESPONSE COVERAGE

• The SmolLM3-3B diagnostics use three RL teachers and exported students, with separate prompt splits for coverage, normalization, and probe measurements.The top-64/domain-response student and initialization each generate a natural response bank of 384 responses.Source: §H.1 · p. 20
• On the domain-response student’s natural bank, minimum top-64 intersection mass is 1.7 × 10^-8; domain-token and global-token students each have one empty intersection on the common bank.These low-coverage cases motivate the targeted gradient comparisons.Source: §H.1 · pp. 20–21
• Figure 10 plots full-vocabulary KL changes after versus before BF16 writeback, with three-batch means and individual batch values.Its symmetric-log color scale is shared, and matched-norm SGD is compared with each rule’s saved-state Adam step.Source: Figure 10 · p. 21

H.2 GRADIENT COMPARISONS ON TYPICAL AND DIFFICULT PREFIXES

• In FP32 diagnostics, mean top-64 intersection mass is 99.05%, 95.68%, and 93.12% for representative, high-entropy, and low-coverage prefix groups.The groups are formed using representative prefixes, entropy, and intersection mass.Source: §H.2 · p. 21

H.3 SUPPLEMENTARY CAPABILITY OBSERVATIONS

• MATH-500 accuracy rises from 50.60% at initialization to 53.80% after PG training with domain-response averaging and SGD.On AIME 2025, the initialization and two trained students each solve four of 30 questions, with only one solved question shared by all three.Source: §H.3 · p. 21