Latent-MOPD: Latent Multi-Teacher On-Policy Distillation

1 INTRODUCTION

• Prior representation-level OPD studies focus on one teacher, leaving open how distinct specialists’ representation targets should jointly train one student backbone.The challenge is to select and organize targets so the student integrates capabilities from multiple specialists.Source: §1 · pp. 1–2
• Representation targets depend on the teacher–student relationship: same-family models differ more near the output, while cross-family similarities show a middle-layer trough.This motivates aligning late states with a shared prediction-relevant role.Source: §1 · p. 2
• In a same-family all-layer representation-only control, interleaved teacher domains collapse training within twenty updates, whereas domain-pure batches remain stable.Sustained representation-only supervision also falls below initial aggregate performance in the cross-family setting.Source: §1 · p. 2
• Latent-MOPD routes each student rollout to a specialist that supplies both token predictions and hidden-state targets, with a crossfade shifting supervision from states to predictions.Teachers remain frozen, and neither teachers nor the projection map are needed at inference.Source: §1 · pp. 2–3
• The same-family student beats token-only, representation-only, and uniform-averaging baselines on all nine benchmarks; with cross-family teachers, it beats both single-channel baselines on all six.The same-family student also surpasses the per-benchmark best teacher on five benchmarks at the same model size.Source: §1 · p. 3

2 RELATED WORK

• OPRD, LastOPD, and PR-OPD study representation supervision from one teacher; this work instead selects and organizes targets from several specialists.OPRD pairs layers by relative depth, LastOPD uses last-layer supervision with a crossfade, and PR-OPD aligns an agent with its skill-conditioned copy.Source: §2 · p. 3
• Figure 3 reports matched-block CKA before training and contrasts domain-pure with interleaved updates in an all-layer representation-only control.The caption describes a middle-layer similarity trough across families and collapse under interleaving.Source: Figure 3 · p. 4
• Prior multi-teacher methods combine outputs or align features off-policy, whereas this work reuses existing specialists and adds hidden-state targets from the routed teacher.MOPD routes student rollouts to frozen teachers for token-level distillation.Source: §2 · p. 4
• The parameter-merged initialization tests whether 62 distillation steps improve over the merged model while retaining a single deployable student.Model soups average compatible weights, while task arithmetic combines task-specific parameter changes.Source: §2 · p. 4

• Each student response is evaluated by the prompt’s domain specialist, which supplies both token targets and hidden-state targets.Hard routing keeps the selected specialist fixed throughout the response.Source: §3 · pp. 4–5
• The token channel routes each prompt to its domain teacher and scores the student’s top-k candidates using teacher-to-student log-probability ratios.Candidate weights are renormalized within the student’s top-k support, while log probabilities retain full-vocabulary normalization.Source: §3 · pp. 4–5
• Same-family training aligns the final three layers, while cross-family training uses the final layer and a shared trainable map across routed teachers.Same-family states use the identity map; the cross-family map bridges differing hidden widths.Source: §3 · p. 5
• Representation penalties are averaged over selected response positions and layers, with the same total representation weight whether one or three layers are selected.Teachers remain frozen; cross-family gradients also update the shared map.Source: §3 · p. 5
• On each teacher’s update clock, token supervision ramps up as representation supervision fades; the per-teacher windows are 10 steps same-family and 7 cross-family.Domain-pure updates rotate among teachers; the crossfade is followed by token-only updates.Source: §3 · pp. 5–6, Figure 4 · p. 5
• The paper also tests a same-family parameter-merged student initialization before applying the 62-step crossfade recipe.Source: §3 · p. 6

4.1 SETUP

• The experiments distill existing specialists into a 1.5B student for 62 updates, using 17,856 domain-balanced prompts and final-checkpoint evaluation.Comparisons include routed token-only, representation-only, and joint baselines; general-retention suites are excluded from distillation.Source: §4.1 · pp. 6–7

4.2 SAME-FAMILY TEACHERS: SURPASSING INDIVIDUAL SPECIALISTS

• The 1.5B student surpasses the per-benchmark best teacher on five of nine same-family benchmarks, with Norm 1.05 versus the best-teacher envelope at 1.The five benchmarks are BBH, MuSR, MBPP, MBPP+, and Minerva.Source: §4.2 · p. 7
• Latent-MOPD improves over token-only, representation-only, and averaging baselines on all nine same-family benchmarks.Crossfade improves eight of nine benchmarks over the matched constant-weight control (Norm 1.05 vs. 0.94).Source: §4.2 · p. 7

4.3 EXTENSION TO CROSS-FAMILY TEACHERS

• With 7B cross-family teachers, the linear-map student beats both single-channel baselines on all six benchmarks; its Norm is 0.26 versus 0.16 for MOPD-style.The MLP variant also beats both single-channel baselines on all six benchmarks and has Norm 0.27.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 EXPLORING A PARAMETER-MERGED INITIALIZATION

• After 62 steps from a parameter merge, Latent-MOPD improves five of six benchmarks and reaches Norm 1.03, above token-only at 0.97 and representation-only at 0.91.The student also exceeds the best same-family teacher on BBH, MBPP, and MBPP+.Source: §4.4 · p. 7, Table 3 · p. 8

Layer selection

• Three-layer supervision improves eight of nine same-family benchmarks, raising Norm from 0.99 to 1.05; cross-family last-layer supervision reaches Norm 0.26 versus 0.25 for three layers.Cross-family benchmark wins split three to three, while the same-family representation-only controls favor three layers over last-layer and all-layer alternatives on seven of nine benchmarks each (Norm 0.73 vs. 0.64).Source: §5 · p. 8

Training dynamics

• Same-family and cross-family runs rise from a MATH-500 base of 84.1 to 92.0 and 87.7, respectively, at step 62.Both remain above the base from step 20 through the later token-only phase.Source: §5 · p. 8

Teacher matching

• On teacher-disagreement positions, Latent-MOPD shifts the math row’s maximum similarity from the code teacher to the math teacher, while code and logic retain domain-matched maxima.The math matching-teacher margin increases by 0.0648 on selected positions and 0.0327 over all positions, with positive paired intervals in both cases.Source: Figure 6 · p. 9, §5 · p. 9

Update organization

• Domain-pure blocks reach 91.7 on MATH-500, while interleaved updates fall to 9.8 by step 30 and finish at 43.2 versus the 84.1 base score.The control holds teachers, loss, coefficient, and budget fixed; interleaving reaches 22.8 at step 20.Source: §5 · p. 9

General-benchmark performance

• On three suites absent from training, same-family scores stay within 1.1 points of the base; ARC-Challenge and WinoGrande improve, while HellaSwag shifts from 44.7 to 44.2.These suites are scored by answer-option likelihood without sampling.Source: §5 · p. 9

Representation-alignment dynamics

• By update 10 per domain, normalized math, code, and logic representation losses fall to 0.30, 0.38, and 0.39 of their first values, remaining low after latent supervision ends.At the final recorded updates, they are 0.13, 0.19, and 0.15; the losses exclude the schedule weight β.Source: §5 · p. 9

Case studies

• In the reported case studies, Latent-MOPD preserves recurrence boundaries, rewrite invariants, and counting constraints that MOPD-style misses.The authors connect these examples to supervising hidden states that can encode intermediate reasoning information.Source: §5 · p. 9

Conclusion

• The 1.5B same-family student beats token-only and representation-only baselines on all nine benchmarks and exceeds the per-benchmark best teacher on five.With cross-family teachers, it beats both single-channel baselines on all six benchmarks; from a parameter-merged initialization, it improves five of six benchmarks over the initial merge.Source: §6 · p. 10
• Latent-MOPD combines routed late-layer alignment, shared width projection, domain-pure updates, and per-teacher crossfades.Teachers remain frozen, and deployment uses only the student without the projection map.Source: §6 · p. 10

A EXPERIMENTAL SETUP AND REPRODUCTION

• A 1.5B DeepSeek-R1-Distill-Qwen student uses three same-family 1.5B teachers or three cross-family 7B teachers, all existing checkpoints reused without additional teacher training.Each prompt’s math, code, or logic label routes both token and representation supervision to its corresponding specialist.Source: A EXPERIMENTAL SETUP AND REPRODUCTION · p. 13
• The 62-step on-policy budget generates 7,936 responses from 1,984 prompt presentations, using four responses per prompt and 32 prompts per update.The 17,856-prompt pool is evenly split across math, code, and logic; blocks are domain-pure and ordered math–code–logic.Source: A EXPERIMENTAL SETUP AND REPRODUCTION · p. 14
• The token loss sums over 16 candidate tokens per position, averages over valid response positions, and weights packed microbatch losses by their response count relative to 16.The token-only, representation-only, combined, and uniform-averaging arms share prompts, batches, optimizer, and update budget under their specified objectives.Source: A EXPERIMENTAL SETUP AND REPRODUCTION · pp. 14, 16
• Same-family representation supervision uses the last three layers and all response positions with a 10-step per-teacher crossfade; cross-family defaults use the last layer and final 2,000 positions with a 7-step crossfade.Representation and token weights fade on each active teacher’s own update clock; the constant-weight control keeps both weights at 1 for all 62 updates.Source: A EXPERIMENTAL SETUP AND REPRODUCTION · pp. 14–15
• Same-family runs use an identity map, while the cross-family default uses a shared, trainable, bias-free linear map from 1,536 to 3,584 dimensions, initialized by ridge fit.The map is discarded after training and inference uses only the student; selecting three rather than 28 layers reduces analytic feature storage by about 89.3%, but does not imply a matching reduction in peak GPU memory.Source: A EXPERIMENTAL SETUP AND REPRODUCTION · pp. 15–16
• Main benchmark scores use final checkpoints, generally at temperature 0.7 and top-p 0.95, with task-specific sampling and answer-option scoring protocols.The study does not select the best intermediate checkpoint; AIME25 uses temperature 0.6 and 16 samples per problem.Source: A EXPERIMENTAL SETUP AND REPRODUCTION · p. 16

B DATASETS AND EVALUATION COVERAGE

• Training prompts come from math, code, and procedural-reasoning sources; student-generated trajectories receive routed teacher supervision, while dataset solutions and answer verifiers are not training targets or rewards.Training and evaluation contain no shared problem instances, as checked before training.Source: B DATASETS AND EVALUATION COVERAGE · p. 17
• Evaluation spans competition math, programming, logic and structured reasoning, plus ARC-Challenge, HellaSwag, and WinoGrande, which are absent from the training pool.The frozen Reasoning Gym evaluation set contains 925 scored problems; BBH uses 1,080 problems, and general suites are scored by answer-option log likelihood.Source: B DATASETS AND EVALUATION COVERAGE · pp. 17–18

On-policy distillation

• On-policy distillation scores student-generated responses with both student and frozen teacher distributions at each response prefix, using the student’s top 16 candidates for the token channel.Candidate weights are normalized within selected support, while the log-probabilities retain full-vocabulary normalization.Source: C OBJECTIVES AND SUPERVISION DESIGNS · p. 18

Token objective

• The token reward contrasts student and teacher log-probabilities, and rewards are scaled by the sample standard deviation within each prompt-domain group without mean subtraction.The routed token-only baseline uses this channel without representation supervision.Source: C OBJECTIVES AND SUPERVISION DESIGNS · p. 18

Latent supervision

• The representation objective matches selected student and teacher hidden states before the language-model head, using a projector when widths differ and a stop-gradient on the teacher side.The hidden-state penalty is normalized by feature norm and hidden width; direct alignment is used for compatible same-family models.Source: C OBJECTIVES AND SUPERVISION DESIGNS · pp. 18–19

Multi-teacher supervision

• Routing selects one domain specialist for both channels, whereas uniform averaging gives each of the three teachers one-third weight on the same math prompt in the schematic.Latent-MOPD retains the routed teacher’s token target and adds a hidden-state target from that same teacher; the feature tiles do not indicate token or layer correspondence.Source: Figure 7 · p. 19, C OBJECTIVES AND SUPERVISION DESIGNS · p. 19

Crossfade

• In same-family multi-teacher training, starting from the crossfade and giving each specialist its own update clock outperforms constant weights (Norm 1.05 vs. 0.94).For the cited same-lineage single-teacher pair matching the student and same-family math teacher, sustained latent supervision scores higher on MATH-500 than crossfade.Source: C OBJECTIVES AND SUPERVISION DESIGNS · p. 19

Same-family channel comparisons

• Norm reaches 1.05, versus 0.90 for MOPD-style, 0.64 for both OPRD-style and Rep-only (last 1), and 0.73 for Rep-only (last 3).Default Latent-MOPD improves over MOPD-style and all three representation-only baselines on all nine benchmarks, and exceeds the best teacher on BBH, MuSR, MBPP, MBPP+ and Minerva.Source: D ADDITIONAL RESULTS AND CHANNEL… · p. 20

Layer comparisons

• Last-three-layer supervision improves over last-layer supervision on eight of nine benchmarks under crossfade (Norm 1.05 vs. 0.99).It also exceeds last-layer and all-layer alternatives on seven of nine benchmarks each under representation-only supervision.Source: D ADDITIONAL RESULTS AND CHANNEL… · p. 20

Cross-family comparisons

• With cross-family teachers, the default last-layer variant beats MOPD-style on all six benchmarks; the last-three-layer variant also beats both single-channel baselines on all six.The default has Norm 0.26 versus 0.25 for the three-layer variant, so increasing layer count gives no aggregate improvement in this comparison.Source: D ADDITIONAL RESULTS AND CHANNEL… · pp. 20–21

Parameter-merge initialization

• After 62 distillation steps, a parameter-merge initialization improves on five of six benchmarks, with Norm rising from 0.87 to 1.03.The student exceeds the best individual teacher on BBH, MBPP, and MBPP+.Source: D ADDITIONAL RESULTS AND CHANNEL… · p. 21

Output mixing and routed supervision

• Latent-MOPD improves over uniform averaging on every reported benchmark, including GYM (40.7 to 52.4), BBH (60.0 to 66.3), MBPP (61.1 to 65.7), and MBPP+ (50.7 to 55.7).The default crossfade recipe also improves every math, code, and logic score over routed token-only MOPD-style.Source: D ADDITIONAL RESULTS AND CHANNEL… · p. 21

Logic-domain training dynamics

• On GYM-200, the same-family configuration rises from 37.94 initially to 62.56 at step 62, while the cross-family configuration reaches 41.19.Both remain above the base from step 20 onward; evaluations use eight samples per problem.Source: D ADDITIONAL RESULTS AND CHANNEL… · p. 21

Measurement setup

• The diagnostic compares matched decoder-block outputs on 24,568 paired positions from 192 prompts, using full native feature widths and no learned projector.Positions beyond 2,560 input tokens are excluded; CKA uncertainty is estimated by bootstrapping prompts.Source: E LAYER-WISE REPRESENTATION SIMILARITY · p. 21

Native profiles

• Same-family native CKA averages 0.987–0.992 over blocks 1–25 but falls to 0.945–0.962 over the final three blocks; cross-family profiles reach 0.116–0.124 at block 13.Cross-family similarity recovers by block 21, while both settings show a final-block drop.Source: E LAYER-WISE REPRESENTATION SIMILARITY · p. 21

Sensitivity to high-activation dimensions

• After masking each model’s top 1% of dimensions by mean absolute activation, the cross-family trough remains at block 13 (0.430–0.451), while same-family final-block CKA rises to 0.948–0.976.Masked same-family minima shift to blocks 18, 19, and 22 for JustRL, Archer2, and ProRL, respectively.Source: Figure 9 · p. 22, E LAYER-WISE REPRESENTATION SIMILARITY · p. 22

Connection to layer selection

• Downstream tests favor the last three same-family layers over the last layer, while cross-family one- and three-layer choices split benchmark wins and have similar aggregate scores.CKA suggests candidate targets but does not establish which layers suffice or are optimal for transfer.Source: E LAYER-WISE REPRESENTATION SIMILARITY · p. 22

F BATCH ORGANIZATION AND TRAINING STABILITY

• At step 62, domain-pure batches score 91.7/56.7 on MATH-500/GYM-200, while interleaved batches score 43.2/12.8 and remain below the untrained base.Interleaved performance falls below the base by step 10 and collapses by step 20; the runs differ only in batch organization, with one teacher per sample in both.Source: F BATCH ORGANIZATION AND TRAINING… · pp. 22–23
• After 120 interleaved updates—1.94× the original budget—MATH-500 ends at 84.50 and GYM-200 at 41.56, still below the domain-pure step-62 scores.From step 100 to 120, endpoint gains are only 0.67 and 0.37 percentage points, respectively.Source: F BATCH ORGANIZATION AND TRAINING… · p. 23, Figure 10 · p. 23

G TEACHER-OUTPUT ALIGNMENT AFTER DISTILLATION

• The alignment diagnostic compares both students with three teachers on identical prefixes, selecting positions at or below the domain median for mean pairwise teacher-output cosine.The fixed selection retains 48 math, 60 code, and 60 logic positions; paired intervals are bootstrapped across chains or prompts.Source: G TEACHER-OUTPUT ALIGNMENT AFTER… · pp. 23–24
• On selected teacher-disagreement positions, Latent-MOPD raises the math matching-teacher margin by 0.0648 (95% interval [0.0184, 0.1324]); code and logic changes have intervals spanning zero.Across all positions, the math margin increases by 0.0327 ([0.0077, 0.0659]).Source: G TEACHER-OUTPUT ALIGNMENT AFTER… · p. 24

Readout-depth diagnostic

• The fitted Jacobian lens maps intermediate residuals to the final residual basis before normalization and vocabulary readout, with 27 fitted indices and a separately measured native head.The lens is calibrated independently on 150 prompts spanning math, code, and logic.Source: §G.1 · p. 24

Results across readout depth

• At readout index 26, the math margin increases by 0.0337 (95% interval [0.0117, 0.0612]); at the native head, it increases by 0.0655 ([0.0196, 0.1257]).Code and logic have smaller native-head changes, with intervals spanning zero.Source: Figure 11 · p. 24, §G.1 · p. 25

Interpreting the margin

• The math margin is relative: at index 25, similarity to all three teachers decreases, but similarity to the math teacher decreases less.The measurement concerns distributions exposed by fitted readouts and does not show that earlier representations lack useful information.Source: §G.1 · p. 25

H CASE STUDIES OF GENERATED SOLUTIONS

• Across the examples, Latent-MOPD gets the answers and key constraints right, while MOPD-style errs in recurrence boundaries, rewrite invariants, or counting setup.The examples include a valid-state-transition result, the Eulerian recurrence boundary, and the 8-right/8-up path count.Source: H CASE STUDIES OF GENERATED SOLUTIONS · pp. 25–26

I EXTENDED RELATED WORK

• Latent-MOPD adds selected hidden states from the same routed teacher to student-generated responses, extending token-level multi-teacher OPD beyond output targets.Prior representation-level OPD studies use a single teacher or the student itself; prior multi-teacher OPD approaches focus on output targets.Source: I EXTENDED RELATED WORK · pp. 26–27

J SCOPE AND EXTENSIONS

• Pre-distillation CKA profiles do not establish which layers suffice for transfer or identify the computations behind downstream gains.Measurements of learned projectors and update directions at training-target states could help connect representation geometry to improvements.Source: I EXTENDED RELATED WORK · p. 27, J SCOPE AND EXTENSIONS · p. 28
• The evaluation compares output mixing, routed token supervision, representation-only supervision, their combination, and parameter merging with fixed domain labels and a shared tokenizer.Channel, layer-selection, crossfade, and batching controls test distinct design choices.Source: J SCOPE AND EXTENSIONS · p. 28