One Teacher, Many Views: Reducing Reference Sensitivity in On-Policy Self-Distillation

• On-policy distillation combines student-generated trajectories with dense teacher feedback, unlike fixed-trajectory supervised fine-tuning or solution-level reinforcement-learning rewards.OPSD uses the student’s problem-and-prefix state while a stop-gradient teacher also observes a verified reference solution.Source: Introduction · p. 1
• In OPSD, changing a valid reference can substantially alter the teacher’s next-token target for the same problem and student prefix, creating conflicting updates.Alternative correct references may differ in decomposition, intermediate quantities, notation, ordering, or reasoning path; the paper calls this reference-induced target instability.Source: Introduction · p. 1
• MCPD reduced reference sensitivity on 80.4% of 500 evaluated problems; in the illustrated state, its top-5 targets fully overlapped versus one shared token for full-reference OPSD.The comparison held the student trajectory and evaluated states fixed and used the same pair of valid references for both methods.Source: Introduction · p. 1, Figure 1 · p. 2
• MCPD’s lower sensitivity is not explained by target flattening: it had lower full-vocabulary entropy on 75.6% of 270 problems, with mean entropy 0.33 versus 0.34 for OPSD.The problems came from the same matched evaluation set.Source: Introduction · p. 2
• The goal is to preserve useful privileged guidance while reducing dependence on one complete presentation, not to remove privileged information or force all valid paths to identical predictions.The reference provides task-relevant evidence but also exposes the teacher to one particular solution presentation.Source: Introduction · p. 2
• MCPD replaces one complete-reference target with the expected teacher distribution over sampled partial views, the exact target for minimizing expected teacher-to-student KL.A single partial view may remain arbitrary; MCPD marginalizes across views, approximating the expectation with a small Monte Carlo sample.Source: Introduction · p. 2

Knowledge Distillation

• Knowledge distillation transfers capabilities through predictive distributions, outputs, or representations, including intermediate reasoning, but most reasoning-distillation methods use fixed references or teacher trajectories independent of student reasoning.This off-policy training creates a mismatch between expert prefixes and the imperfect states encountered during student generation (Song and Zheng 2026).Source: Related Work · p. 3

On-Policy Distillation

• On-policy distillation supplies dense teacher feedback at intermediate states along student-sampled reasoning rollouts, combining on-policy state coverage with fine-grained supervision (Agarwal et al. 2024; Ko et al. 2024; Jin et al. 2026).This trains the student on states it is likely to visit during inference.Source: Related Work · p. 3

Privileged On-Policy Self-Distillation

• MCPD targets reference-induced instability in privileged self-distillation by sampling partial reference views and distilling the marginal teacher distribution formed by averaging their predictions.Prior OPSD conditions the teacher on a complete reference along the student-generated trajectory (Zhao et al. 2026).Source: Related Work · p. 3
• AVSD combines three fixed privileged views and preserves consensus plus view-specific residuals, but its targets can retain reference-specific information tied to wording, ordering, or reasoning path.MCPD achieves higher average accuracy than AVSD across all backbones (Table 1).Source: Related Work · p. 3

On-Policy Privileged Self-Distillation

• Standard privileged OPSD provides dense supervision on the student’s own trajectory, but its target at every state is determined by one complete reference solution.The teacher evaluates the same student state as the student while additionally observing the reference.Source: On-Policy Privileged Self-Distillation · p. 3

Reference-Induced Target Instability

• Reference-induced sensitivity compares teacher targets for two valid references at the same problem, student trajectory, and rollout positions using Jensen–Shannon divergence.Zero indicates invariant targets; larger values indicate greater reference-dependent variation in supervision.Source: Reference-Induced Target Instability · pp. 3–4

Privileged Context as a Distribution

• MCPD samples ordered partial views that reveal only part of a reference, then averages their teacher predictions in probability space to form a valid categorical target.The teacher and student state remain fixed across views; only the privileged conditioning context changes.Source: Privileged Context as a Distribution · p. 4

Expected-View Distillation

• The probability-space average is the exact target for minimizing expected forward-KL loss over random partial views, up to a student-independent term.The expected loss also includes a term capturing disagreement among view-conditioned teacher predictions.Source: Expected-View Distillation · p. 4

Monte Carlo Approximation

• For a fixed reference, finite-view Monte Carlo estimation is unbiased and its sampling variability decreases proportionally to 1/K, but this does not guarantee lower sensitivity across references.Reduced cross-reference sensitivity is an empirical property of MCPD, not a consequence of the 1/K variance reduction.Source: Monte Carlo Approximation · p. 5

Monte Carlo Privileged Distillation

• MCPD trains on the student’s on-policy trajectory, averaging teacher next-token distributions from K partial views of the same reference at each student state.All teacher evaluations share the same problem and student-generated prefix; the references and views are never observed by the student.Source: Monte Carlo Privileged Distillation · p. 5

Mathematical Reasoning

• MCPD achieves the highest average accuracy across all four mathematical-reasoning backbones, surpasses GRPO across Qwen3 models, and improves AIME25 accuracy over OPSD by 2.0 points on Qwen3-8B and 2.3 on Qwen3-4B.On Qwen3-4B, MCPD also improves AIME25 accuracy over AVSD by 5.6 points.Source: Mathematical Reasoning · p. 6

Science Reasoning

• MCPD achieves the highest overall SciKnowEval accuracy, raising average score from 49.7 with OPSD to 50.9, a gain of 1.2 percentage points.Compared with OPSD, MCPD improves Biology by 2.4 points and Physics by 1.8; compared with AVSD, it gains 3.0 points in Physics.Source: Table 2 · p. 6, Science Reasoning · p. 6
• On math-trained Qwen3-8B, MCPD has the highest average SciKnowEval score, leads in Chemistry, Physics, and Materials, and matches AVSD in Biology.This demonstrates transfer to out-of-domain scientific reasoning alongside improved in-domain mathematical performance.Source: Table 3 · p. 6, Science Reasoning · p. 6

Ablation Studies

• On DeepSeek-R1-Distill-Qwen-7B, averaging K = 4 views raises average performance from 40.3 with one view to 41.2.Across evaluated view counts, MCPD exceeds OPSD and remains stronger than AVSD; the single-view result illustrates sensitivity to a particular masking realization.Source: Ablation Studies · p. 6
• With sampled-token supervision, MCPD improves over OPSD by 4.6 percentage points on AIME25 and 9.4 on HMMT25.Compared with AVSD, MCPD matches AIME25 accuracy and improves HMMT25 by 3.4 points.Source: Ablation Studies · p. 6
• With K = 4, all tested masking ratios exceed OPSD on Qwen3-14B and exceed both OPSD and AVSD on DeepSeek-R1-Distill-Qwen-7B.Qwen3-14B averages rise from 55.9 with OPSD to 56.8–57.0; DeepSeek averages range from 40.9 to 41.5, versus 40.1 with OPSD and 39.8 with AVSD.Source: Ablation Studies · p. 6, Figure 2 · p. 7, Figure 3 · p. 7

Conclusion

• MCPD treats privileged context as a distribution over partial reference views and averages view-conditioned predictions in probability space, the exact target for expected teacher-to-student forward-KL.This marginalization preserves useful reference information while reducing sensitivity to reference-specific artifacts and reasoning paths.Source: Conclusion · p. 7
• With the problem and student state fixed, replacing one valid reference solution with another can induce substantially different teacher targets, a limitation termed reference-induced target instability.Source: Conclusion · p. 7
• MCPD achieves the strongest average performance across four mathematical-reasoning backbones and improves scientific reasoning and out-of-domain transfer from mathematics to science.The conclusion also reports robustness across masking ratios, benefits from multiple partial views, and effectiveness under sampled-token distillation.Source: Conclusion · p. 7
• Future work includes extending MCPD beyond mathematical and scientific reasoning to settings such as multimodal reasoning.Source: Conclusion · p. 7

Target Entropy Analysis

• The analysis uses 270 problems with the same reference pairs, student trajectories, and matched token positions as the reference-sensitivity analysis.Source: Target Entropy Analysis · p. 10
• Entropy is computed for OPSD’s complete-reference teacher targets and MCPD’s probability-space average of K = 4 view-conditioned targets, then averaged across references and matched student states per problem.Entropy uses the natural logarithm and is measured in nats.Source: Target Entropy Analysis · p. 10
• MCPD has comparable, slightly lower mean entropy than OPSD and lower entropy on 75.6% of evaluated problems, so reduced reference sensitivity is not explained by target flattening.The comparison is reported in Table 5.Source: Target Entropy Analysis · p. 10

Reference-Sensitivity Implementation Details

• The evaluation uses 500 DeepMath-103K training problems, Qwen3-8B, and the first two verified references; the same checkpoint serves as student and frozen teacher.Problems are sampled with seed 42.Source: Reference-Sensitivity Implementation… · p. 10
• One student rollout is held fixed across both references and methods, with teacher distributions evaluated at 20 matched positions spanning 10%–90% of its length.The teacher uses enable_thinking=True; OPSD conditions on the complete reference and returns next-token distributions at the fixed rollout prefixes.Source: Reference-Sensitivity Implementation… · p. 10
• MCPD samples four views per reference by masking 30% of reference tokens, averages their teacher distributions in probability space, and repeats this with five mask banks.The five reproducible mask-bank seeds are 42, 43, 44, 45, and 46; unmasked-token order is preserved.Source: Reference-Sensitivity Implementation… · p. 10
• Using Jensen–Shannon divergence, MCPD has lower reference sensitivity than OPSD on 80.4% of the 500 evaluated problems.A positive per-problem reduction means MCPD is less reference-dependent for the same problem, student trajectory, reference pair, and matched student states.Source: Reference-Sensitivity Implementation… · pp. 10–11

Computational Cost

• MCPD adds approximately 11.6% wall-clock time over OPSD, costs comparably to AVSD, and GRPO requires approximately 7.8× MCPD’s time.Timing used DeepSeek-R1-Distill-Qwen-7B on four NVIDIA A100 80GB GPUs; MCPD used K = 4 views, and GRPO used 500 steps with eight rollouts per prompt.Source: Computational Cost · p. 11

Mathematical Reasoning

• MCPD uses four independently sampled and resampled masked-reference views per example, averaging frozen-teacher next-token distributions in probability space before computing the distillation loss.Each view masks 30% of reference tokens for Qwen3 models or 40% for DeepSeek-R1-Distill-Qwen-7B; unmasked tokens retain their order.Source: Mathematical Reasoning · p. 11

Evaluation Setup

• Evaluation reports Avg@12 for mathematical reasoning and Avg@16 for scientific reasoning, with Avg@K estimating expected single-response accuracy rather than success at least once.Math uses 12 sampled responses per problem; SciKnowEval L3 uses 16 across biology, chemistry, physics, and materials science.Source: Evaluation Setup · p. 12

Full Numerical Results for Main-Paper Figures

• The numerical results underlying Figures 3 and 5 use DeepSeek-R1-Distill-Qwen-7B, with averages computed across AIME24, AIME25, and HMMT25.The supplied passages identify the results’ scope but do not state the exact benchmark values.Source: Full Numerical Results for Main-Paper… · p. 12

Masking-Ratio Results

• With K = 4, every evaluated masking ratio outperforms full-reference OPSD in average accuracy across AIME24, AIME25, and HMMT25.The experiments use DeepSeek-R1-Distill-Qwen-7B; the supplied passages do not provide the exact values.Source: Masking-Ratio Results · p. 12, Table 7 · p. 12

Number-of-Views Results

• At ρ = 40%, aggregating multiple partial reference views improves over using a single masked view on average across AIME24, AIME25, and HMMT25.The experiments use DeepSeek-R1-Distill-Qwen-7B; the supplied passages do not provide the exact values.Source: Number-of-Views Results · p. 12, Table 8 · p. 12

• For each example, the student generates one on-policy trajectory from the problem alone, and only the student parameters are updated.The teacher is frozen during this procedure.Source: MCPD Training Algorithm · pp. 12–13
• At the student’s shared states, the frozen teacher evaluates K independently sampled partial-reference views and averages its distributions in probability space.Because the student trajectory is shared across views, variation among teacher targets arises only from the partial reference context.Source: MCPD Training Algorithm · p. 13