Beyond Domain-Routed Supervision: Cross-Teacher Guarding for Multi-Teacher On-Policy Distillation

1 INTRODUCTION

• Outcome-level rewards lack direct feedback on intermediate reasoning steps, whereas OPD gives dense token-level supervision on student-generated trajectories.MOPD extends this approach by routing domain-specific samples to specialized teachers, but updates from all domains still act on the shared student.Source: §1 · p. 1
• Math–Code joint training generally preserves or improves single-domain OPD performance, but both Medical pairings show frequent conflicts and lower Medical performance.The conflicting-layer proportion rises further in later training stages for the Medical pairings.Source: §1 · p. 2
• ROAD MOPD uses non-routed teachers as cross-domain critics on the same student trajectories, retaining routed supervision while seeking to reduce interference.The auxiliary feedback provides local cues about potentially incompatible learning signals.Source: §1 · p. 2
• 5.6–8.8 percentage points: ROAD MOPD improves Medical accuracy over Vanilla MOPD in high-conflict settings, while also improving overall performance in low-conflict and three-domain settings.Across domain combinations and teacher–student configurations, it mitigates degradation in adversely affected domains, preserves low-conflict gains, and generalizes to three domains.Source: §1 · pp. 2–3
• Cross-teacher agreement strengthens token corrections supported by both teachers and downweights tokens with conflicting signals.At the gradient level, auxiliary supervision defines a reference direction for projecting components of the target-domain update that conflict with it.Source: §1 · p. 2
• Domain-routed supervision can produce persistent conflicting updates to shared parameters when domain updates are poorly compatible.These conflicts accompany asymmetric performance degradation, making update compatibility important for joint learning.Source: §1 · p. 2

2.1 ON-POLICY DISTILLATION

• OPD research refines objectives, optimization, token selection, and scaling, but does not test compatibility with capabilities represented by other teachers.The paper positions its approach as having domain-specialized teachers jointly assess the same student-generated prefixes.Source: §2.1 · p. 3

2.2 MULTI-TEACHER ON-POLICY DISTILLATION

• MOPD methods route teachers or adjust domain updates, whereas ROAD MOPD uses non-routed teachers to assess the target-domain trajectory and guide token credit assignment and preventive gradient constraints.The related work also distinguishes gradient-conflict methods that modify already-computed task gradients.Source: §2.2 · p. 3

3.1 PROBLEM SETUP AND ON-POLICY DISTILLATION

• The setup treats OPD as sequence-level reverse-KL distillation on student-generated responses, then routes each prompt to its domain teacher in equal-weight MOPD.The routed and auxiliary teachers score the same student-generated tokens under teacher forcing, without additional rollouts.Source: §3.1 · pp. 3–4

3.2 CROSS-TEACHER-GUIDED CREDIT ASSIGNMENT

• ROAD MOPD adjusts routed-teacher token rewards using auxiliary-teacher agreement, amplifying shared correction directions and attenuating conflicting signals.When agreement is zero, the token reward scale recovers standard OPD; clipping limits amplification and attenuation.Source: §3.2 · pp. 4–5

3.3 PREVENTIVE MULTI-TEACHER GRADIENT PROJECTION

• Preventive projection uses auxiliary gradients on the same target-domain trajectories to remove conflicting update components while retaining a minimum fraction of target-gradient norm.The constraint is local to the same-trajectory auxiliary loss, not a guarantee of auxiliary-domain performance after an optimizer step.Source: §3.3 · pp. 5–6

3.4 OVERALL OBJECTIVE AND TRAINING

• Training sequence-mean-aggregates response losses; auxiliary losses guide correction only, while projected target gradients are averaged and globally clipped for one AdamW step.The overall distillation objective includes only target-domain losses.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• Evaluation spans 11 Code, Math, and Medical benchmarks, comparing four representative approaches using Qwen3-4B and Qwen3-1.7B students.The paired settings use domain-specialized teachers and 10,000 sampled prompts per domain.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

• ROAD MOPD achieves the highest two-domain average across Math × Medical and Code × Medical with both student sizes, beating the strongest overall baseline by 0.72–2.89 points.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 FURTHER ANALYSIS

• In Math × Code, a 1.25-point Code gain yields All Avg. 55.40, 0.35 above ExOPD; across three domains, ROAD MOPD leads each domain and reaches 52.89 All Avg.Source: Table 2 · p. 8, Table 3 · p. 8, §4.3 · p. 8

4.2 MAIN RESULTS

• For 4B students, ROAD MOPD reaches Medical averages of 48.23 and 47.75 in the two pairs, near the 47.90 Single OPD reference, while leading Math and Code averages.Source: §4.2 · p. 8
• For 1.7B students, Medical averages of 36.89 and 36.51 exceed Single OPD’s 35.67, yielding two-domain means 1.33 and 2.89 points above ExOPD despite lower Math and Code averages.Source: §4.2 · p. 8

4.3 FURTHER ANALYSIS

• Same-trajectory preventive projection improves all four domain-average accuracies over PCGrad by 0.18–2.12 percentage points; independent domain-gradient pairs are mostly non-negative.Source: Figure 4 · p. 9, §4.3 · p. 9
• Removing credit assignment or gradient protection lowers all four domain averages; full projection lowers two-domain means by 0.83 and 0.44 points versus adaptive retention.Source: Table 4 · p. 9, §4.3 · p. 9
• Disagreement quintiles track rising negative-gradient-cosine batch rates (Math 24% to 40%; Medical 46% to 76%), supporting agreement-guided token credit assignment.Source: Figure 3 · p. 8, §4.3 · p. 9

5 CONCLUSION

• Across mathematics, medicine, and code, ROAD MOPD mitigates high-conflict degradation, retains gains in compatible settings, and achieves the best distilled-model average in each domain under joint three-domain training.The authors note that auxiliary scoring and gradient computation add training cost, and that evaluation is limited to three text-based reasoning domains.Source: §5 · p. 10

A.1 OPD DERIVATION AND TRAINING LOSS

• The reverse-KL OPD objective is reformulated as relative-reward maximization, and scaling the relative-reward term yields the sampled-response objective.For multiple auxiliary teachers, their token agreements are averaged; opposing assessments can cancel, recovering standard OPD when mean evidence is zero.Source: §A.1 · p. 15

A.2 DERIVATION OF NORM-CONSTRAINED PROJECTION

• Sequential projections against conflicting peers use a shared norm floor relative to the original target tensor, but do not guarantee simultaneous non-conflict with every peer.The floor trades complete conflict removal for retention of the target-domain signal.Source: §A.2 · pp. 16–17

A.3 COMPLETE TRAINING PROCEDURE

• The training step scores stored student-generated trajectories with the teachers and reference, then applies token modulation and peer projection; only projected target gradients enter the update.Auxiliary gradients are discarded after projection and are not merged into the update.Source: §A.3 · pp. 17–18

B.1 MODELS AND TEACHER–STUDENT CONFIGURATIONS; B.2 TRAINING DATA AND PROMPT CONSTRUCTION

• The setup uses full-parameter student training while teachers and the shared reference remain frozen and score responses.The paired-domain prompt pools contain 10,000 prompts per domain; domain labels select the routed teacher, and ground-truth answers are not used as outcome rewards.Source: §B.1 · p. 18, §B.2 · pp. 18–19

B.3 TRAINING HYPERPARAMETERS AND INFRASTRUCTURE

• The shared modulation parameters are τ = 1, ρg = 0.5, and [λmin, λmax] = [0.5, 1.5]; paired-domain training uses bidirectional projection with m = 0.6.The unbounded agreement mapping already lies within [0.5, 1.5], so clipping does not further restrict it in exact arithmetic.Source: §B.3 · p. 19
• Paired- and three-domain runs use one node with eight NVIDIA B200 GPUs, with six hosting the student and rollout replicas and two hosting teacher scoring.In three-domain training, the balanced sampler forms batches of 336 prompts per domain.Source: §B.3 · p. 19
• Matching rollout budgets does not ensure matching optimizer-update counts: Open-MOPD uses four sequential minibatches per batch, while ExOPD and CaMOPD use one update per batch.CaMOPD also changes prompt allocation across domains through its alternating schedule.Source: §B.3 · p. 20

B.4 BASELINE IMPLEMENTATIONS

• Vanilla MOPD routes each prompt to its domain teacher without cross-teacher modulation or gradient projection, whereas ExOPD uses a fixed extrapolation coefficient of 1.25.ExOPD disables the cross-teacher gate, gradient projection, and domain-balancing modules.Source: §B.4 · p. 20
• CaMOPD ranks examples by rollout-time teacher–student gaps, retains the smallest prefix covering 80% of score mass, and prioritizes Medical in the three-domain allocation.Paired-domain update ratios and recovery branches vary by domain pairing.Source: §B.4 · p. 20
• Open-MOPD weights domains using token shares and teacher–student gaps, then refreshes advantages on each of four sequential optimizer minibatches.The weights remain fixed within each rollout batch.Source: §B.4 · p. 20
• PCGrad projects each domain’s gradient against the original opposing gradient when their dot product is negative, then sums the corrected gradients.These gradients come from their respective domains’ responses, not same-trajectory auxiliary supervision.Source: §B.4 · p. 21

Evaluation datasets and aggregation

• Eleven benchmarks span Code, Math, and Medical, with domain averages computed as equal-weight means over benchmarks and the overall average weighting all 11 equally.The overall average is not the mean of the three domain averages because Code has three benchmarks while the other domains have four.Source: §B.5 · pp. 21–22

Generation setup

• Offline evaluation uses vLLM with temperature 1.0, top-p 1.0, a 16,384-token completion limit, and prompts that disable thinking mode.Source: §B.5 · p. 21

Mathematics

• Math reports Avg@32 from 32 completions per question, using the last boxed answer and math-verify; it equals pass@1 on those samples, not pass@32.Source: §B.5 · p. 21

Code

• Code reports Avg@4 from four completions per problem, counting a completion only when it passes both original and additional EvalPlus tests.LiveCodeBench uses its v6 code-generation split and checks public and private tests.Source: §B.5 · p. 21

Medical

• Medical reports Avg@1 multiple-choice accuracy from one completion per question, requiring an answer that parses to an available option and exactly matches the reference.Source: §B.5 · pp. 21–22

Teacher-disagreement and gradient-conflict probe

• The exploratory fixed-checkpoint probe groups batches into five within-domain disagreement quintiles and measures negative cosine between routed- and auxiliary-teacher gradients on identical responses.Each quintile contains 50 batches; gradients are raw full-parameter OPD gradients without token modulation, projection, or optimizer updates.Source: §B.5 · p. 22

C.1 DETAILED BENCHMARK RESULTS

• Table 7 gives per-benchmark ablation and PCGrad breakdowns, while Table 8 lists available single-domain checkpoints across all 11 benchmarks.The single-domain checkpoints are Qwen3-4B models trained with OPD on 10,000 in-domain prompts for three epochs and evaluated across all benchmarks.Source: Table 7 · p. 22, §C.1 · p. 22, Table 8 · p. 23

C.2 FULL RESULTS FOR LOW-CONFLICT AND THREE-DOMAIN SETTINGS

• Tables 9 and 10 provide full per-benchmark Qwen3-4B results for low-conflict Math × Code and three-domain settings, with domain averages weighting benchmarks equally.Single OPD uses the matching in-domain checkpoint as the reference.Source: §C.2 · p. 23, Table 9 · p. 23, Table 10 · p. 24

C.3 EXTENDED TOKEN-LEVEL VISUALIZATIONS

• Figure 5 expands selected Math and Medical trajectories to 256-token contexts, marking the main-text windows; the examples illustrate coefficients rather than estimate their distribution.The selected trajectories were chosen to show attenuation and amplification near answer mentions.Source: §C.3 · p. 24, Figure 5 · p. 25
• Figures 6 and 7 show three additional Math and Medical response excerpts in fixed windows, selected without inspecting coefficients or answer correctness.The Medical excerpts are generated reasoning, not verified answer explanations.Source: §C.3 · p. 26, Figure 6 · p. 26, Figure 7 · p. 27

Loss and generation behavior

• Across three 58-update runs, Figure 8 tracks the task-0 primary signed policy surrogate, per-domain response lengths, and the share of responses reaching the token cap.The surrogate decreases sharply early, but the single-run traces do not establish convergence of both domain objectives or comparative test accuracy.Source: §C.4 · p. 28, Figure 8 · p. 28

Gradient-norm retention during projection

• The adaptive norm ratios stay at or above 0.6 in all six domain traces; in Math × Medical, full projection falls below 0.6 in 40 of 58 Math updates and reaches 0.253, while adaptive projection remains at 0.600.The Medical branch never crosses the floor, so its adaptive and full-projection curves coincide.Source: §C.4 · p. 29, Figure 9 · p. 29

• Compared with Vanilla MOPD, auxiliary-teacher scoring and gradient processing add training overhead, although reusing student trajectories avoids extra rollout generation.Source: D LIMITATIONS · p. 30
• Experiments cover three domains and evaluate joint training on at most three domains, leaving broader domain collections and larger teacher pools for future evaluation.Source: D LIMITATIONS · p. 30