PaperlayerView exact revision on arXiv

Source-linked AI summary

Calibrating Teacher--Student Discrepancy for On-Policy Distillation

Qiangqiang He, Jin Li, MingCai Chen

arXiv:2609.21619v1cs.AI

TL;DR

Standard OPD’s teacher–student discrepancy includes teacher self-deviation, and privileged context can amplify those teacher-side shifts. Cal-OPD estimates the self-deviation region with positive and negative privileged interventions, then retains only discrepancy beyond it; on mathematical reasoning benchmarks, it outperforms standard OPD and variants while using a smaller share of the original discrepancy.

  • Problem

    The teacher–student discrepancy used by OPD reflects not only capability differences but also token- and context-dependent teacher self-deviation, which privileged context can amplify.

    Source: §1 · pp. 1–2

  • Method

    Cal-OPD uses positive and negative privileged interventions to estimate the teacher’s self-deviation region and retains only the discrepancy beyond it for optimization.

    Source: §1 · p. 2

  • Results

    Using about 52–65% of the original teacher–student discrepancy, Cal-OPD consistently outperforms standard OPD and its variants across model scales on mathematical reasoning benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Effective OPD depends on identifying which teacher–student discrepancies are worth learning rather than indiscriminately optimizing all of them.

    Source: §5 · pp. 10–11

    TSD is measured on student-generated rollouts rather than through an exhaustive characterization of all possible contextual variation.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

On-policy distillation (OPD) improves reasoning models by learning the token-level discrepancy between a stronger teacher and an on-policy student. However, this discrepancy does not purely reflect the capability gap between the teacher and the student: it also contains deviations arising from the teacher itself, which are consequently mixed into the observed teacher--student discrepancy and indiscriminately learned by standard OPD during training. This issue is further exacerbated by privileged OPD, where privileged information induces larger teacher-side likelihood shifts, thereby encouraging the student to learn more of the teacher's own deviation. We introduce \textbf{Calibrated On-Policy Distillation (Cal-OPD)}, which estimates the teacher's self-deviation region through positive and negative privileged interventions and calibrates the original teacher--student discrepancy by retaining only the component that lies beyond this region. Experiments on mathematical reasoning benchmarks show that, while retaining only about 52--65\% of the original teacher--student discrepancy as the optimization signal, Cal-OPD consistently outperforms standard OPD and its variants across model scales.

1 Introduction

On-policy distillation provides token-level supervision on student-generated trajectories, but its teacher–student discrepancy also includes teacher self-deviation, which privileged context can amplify. Cal-OPD estimates that deviation using contrasting privileged interventions and retains only discrepancy beyond the estimated region.

1 Introduction

  • OPD trains on student-generated trajectories with token-level likelihood discrepancies, addressing the distribution mismatch of fixed-data off-policy distillation.Compared with sparse outcome-level rewards in RLVR, OPD supplies token-level guidance throughout a trajectory.Source: §1 · p. 1
  • Standard OPD’s discrepancy reflects both the capability gap and teacher self-deviation, which varies across tokens and is concentrated in context-sensitive surface-form tokens.Reasoning tokens tend to remain relatively stable under contextual interventions.Source: §1 · p. 1
  • Privileged context induces further teacher-distribution shifts, amplifying teacher-side deviations in the discrepancy that the student learns.The paper connects such shifts in prior work to information asymmetry, shortcut behavior, and degraded thinking-model performance.Source: §1 · p. 2
  • Cal-OPD estimates the teacher’s self-deviation region with positive and negative privileged interventions, then retains only the teacher–student discrepancy beyond that region.The interventions calibrate the teacher reference rather than directly supervising the student.Source: §1 · p. 2
  • Using only about 52–65% of the original discrepancy, Cal-OPD consistently outperforms standard OPD and its variants across model scales while alleviating response-length expansion.The reported experiments use mathematical reasoning benchmarks.Source: §1 · p. 2

2 Understanding Teacher Self-Deviation

Teacher self-deviation (TSD) is context-induced change in teacher likelihood on a fixed student token; analyses find it arises without task knowledge, persists across richer contexts, and concentrates on surface-form tokens.

2.1 Preliminaries

  • TSD is the teacher-likelihood change caused by teacher-only context interventions while the problem, student prefix, and evaluated token remain fixed.The intervention-derived region estimates teacher-likelihood variability on student-generated trajectories, rather than exhaustively characterizing all contextual variation.Source: §2.1 · p. 3

2.2 Teacher Self-Deviation Is Not Reliable Knowledge

  • At τ=0.01, task-agnostic instructions affect 29.8% of tokens; solution-level privilege expands TSD to 39.6% while retaining 98.2–98.6% of earlier affected positions.The affected positions are largely retained even as richer contexts broaden the set.Source: §2.2 · pp. 4–5, Figure 2 · p. 5
  • Reversing answer correctness still yields 88.7% directional agreement and 74.7% shared deviation, while solution-level interventions reach 80.3% positional overlap but 66.1% shared deviation.Across intervention groups, TSD shows substantial overlap, high directional agreement, and predominantly shared variation.Source: Figure 3 · p. 5, §2.2 · p. 6

2.3 TSD Concentrates on Surface-Form Tokens

  • At τ=0.01, high-ranked natural-language forms have significant-TSD rates above 89%, versus below 9.3% for low-ranked mathematical and symbolic forms.The table’s highest-ranked forms include “maybe” and “however”; its lowest-ranked forms include digits and mathematical notation.Source: Table 2 · p. 6, §2.3 · p. 7

3 Calibrated On-Policy Distillation

Cal-OPD estimates a teacher self-deviation region and distills only the teacher–student discrepancy that remains beyond it.

3 Calibrated On-Policy Distillation

  • Two contrasting positive and negative interventions probe the teacher’s contextual variation, while a relaxation factor accounts for the finite probe when estimating the TSD region.The relaxation factor λ is constrained to λ ≥ 1.Source: §3 · p. 7
  • Cal-OPD removes the discrepancy covered by the estimated TSD region and retains only the residual beyond its boundary.This calibrated residual is the component treated as useful supervision.Source: §3 · p. 7
  • Cal-OPD follows the standard OPD objective but replaces the token-level OPD advantage with the calibrated advantage.Source: §3 · p. 7

4 Experiments

Across two teacher–student scales, Cal-OPD outperforms standard OPD and its variants, while intervention and relaxation-factor analyses show that calibration can over-filter supervision. Its training dynamics also show shorter responses and faster training despite extra teacher computation.

4.1 Experimental Setup

  • The experiments compare six mathematical reasoning benchmarks, two Qwen3 teacher–student scales, and five representative OPD baselines.Methods were trained on the same 15,560 question–solution pairs; Cal-OPD used evaluative-feedback interventions and λ=5.Source: §4.1 · pp. 8–9

4.2 Main Results

  • 53.1 and 69.0 are Cal-OPD’s highest average performances in the two configurations, with gains over student baselines and standard OPD.+3.9 and +2.4 over student baselines, and +2.3 and +3.1 over standard OPD; Cal-OPD is best on 7 of 12 benchmark–configuration pairs.Source: §4.2 · p. 9

4.3 Analysis and Ablations

  • Ceval reaches an average of 53.1, while solution-level Csol lowers the average to 49.0, consistent with over-filtering task-relevant supervision.Instruction interventions retain nearly 70% of the teacher–student discrepancy; Ceval provides stronger calibration without task-specific solution knowledge.Source: §4.3 · p. 9, Table 4 · p. 9
  • Cal-OPD ends at 9.3K response tokens versus standard OPD’s 11.8K and trains approximately 1.26× faster despite extra teacher computation.For the main Ceval configuration, the retained-discrepancy ratio falls from 65% to 52%, while the zero-advantage token ratio rises from 27% to 34%.Source: §4.3 · p. 9, Figure 4 · p. 10
  • At λ=5, Cal-OPD performs optimally while retaining approximately 52% of the discrepancy; at λ=80, performance is 1.7 points below the λ=1 baseline.Larger λ expands the estimated TSD region, reducing retained discrepancy to roughly 20% and over-filtering task-relevant supervision.Source: §4.3 · p. 10, Figure 5 · p. 10

5 Conclusion

The conclusion identifies teacher self-deviation as a component of OPD’s teacher–student discrepancy and presents Cal-OPD as a way to filter it. Across mathematical reasoning benchmarks and model scales, Cal-OPD outperforms standard OPD and its variants while retaining only part of the original discrepancy.

5 Conclusion

  • Teacher self-deviation appears without task-specific knowledge, is largely insensitive to intervention semantics and correctness, and concentrates on surface-form tokens.The conclusion states that standard OPD’s discrepancy contains substantial TSD.Source: §5 · p. 10
  • Cal-OPD estimates the teacher’s self-deviation region and removes the TSD-explained portion of the OPD signal before optimization.It retains the teacher–student discrepancy beyond the estimated region.Source: §5 · p. 10
  • Across mathematical reasoning benchmarks and teacher–student scales, Cal-OPD outperforms standard OPD and its variants while retaining 52–65% of the original discrepancy.The conclusion also reports mitigation of response-length expansion during training.Source: §5 · p. 10

A.1 Related Work

The related work traces distillation from knowledge transfer to on-policy and privileged-context methods, alongside work questioning the uniform reliability of teacher supervision. It distinguishes Cal-OPD’s focus on controlled contextual variation in teacher likelihoods from prior reliability measures.

A.1 Related Work

  • Knowledge distillation transfers capabilities from stronger teachers to weaker students, with later methods extending it to generation, compression, and reasoning rationales [Hinton et al., 2015].Reasoning approaches include Fine-tune-CoT [Ho et al., 2023], Distilling Step-by-Step [Hsieh et al., 2023], and DeepSeek-R1 [Guo et al., 2025].Source: §A.1 · p. 15
  • On-policy distillation supervises trajectories sampled from the student, reducing training–inference distribution mismatch [Agarwal et al., 2024].Extensions adjust supervision using reward extrapolation, uncertainty, outcome calibration, or adaptive horizons [Yang et al., 2026a; Jin et al., 2026; Ke et al., 2026; Hou et al., 2026a; Hou et al., 2026b].Source: §A.1 · p. 15
  • Privileged and context-augmented distillation supplies the teacher with training-only information, while related work reports that privileged context can alter teacher behavior and complicate likelihoods as transferable-knowledge signals.Prior work includes solution-conditioned OPSD [Zhao et al., 2026a], outcome-guided routing [Yu et al., 2026b], and methods addressing information asymmetry [Wu et al., 2026].Source: §A.1 · p. 15
  • Prior teacher-reliability methods use uncertainty, outcomes, reasoning position, rollout quality, or privileged-reference effects to characterize supervision heterogeneity.Studies also report greater contextual sensitivity for stylistic tokens than reasoning tokens [Pan et al., 2026; He et al., 2026].Source: §A.1 · pp. 15–16
  • The related work identifies controlled contextual variation as an unmodeled basis for defining a finite-intervention self-deviation region to calibrate OPD discrepancy.This contrasts with approaches centered on uncertainty, outcome, position, trajectory quality, or specific privileged-reference effects.Source: §A.1 · p. 16

A.2 Data Collection and Preparation for TSD Analysis

The TSD analysis builds solution-augmented DAPO-17K data, collects on-policy Qwen3-1.7B rollouts, and rescores fixed trajectories under teacher-side contextual interventions. The setup uses length-matched negative solution contexts and analyzes approximately 60 million tokens across the baseline and eight interventions.

A.2 Data Collection and Preparation for TSD Analysis

  • Reference solutions were generated for DAPO-17K problems, and filtering by ground-truth final answers retained 15,560 question–solution pairs.The retained set covers 90.18% of 17,255 problems; 1,695 problems were unsolved.Source: §A.2 · p. 16
  • One on-policy response per retained question was generated with Qwen3-1.7B in thinking mode, yielding 6,528 question–response pairs for TSD analysis.Collection stopped at approximately 60 million student-generated tokens; responses averaged 9,270 tokens.Source: §A.2 · p. 16
  • The figures show significant TSD prevalence across thresholds and teachers, and retention across thresholds and teacher models.The prevalence caption reports that solution-level privilege broadens the affected set; the retention caption says less informative interventions retain nearly all affected positions under solution-level privilege, with substantially lower retention in the reverse direction.Source: Figure 6 · p. 17, Figure 7 · p. 18
  • Negative solution-level contexts use a different problem’s reference solution, length-matched to within 10% of the current solution.This constraint reduces the possibility that positive–negative differences are driven primarily by contextual length rather than content.Source: §A.2 · p. 17
  • Teacher rescoring holds the problem, student rollout, prefix, and evaluated token fixed while changing only teacher-side context.The teacher recomputes token-level log-likelihoods rather than generating new trajectories; approximately 60 million tokens are rescored under a baseline and eight interventions.Source: §A.2 · pp. 17–18, Table 5 · p. 17

A.3 TSD Semantic Consistency Across Thresholds and Model Scales

Across thresholds and teacher scales, TSD remains substantial without task-specific knowledge, while richer privileged context expands affected token positions. Contrasting intervention semantics continue to yield directionally aligned, predominantly shared deviations.

Prevalence of Significant TSD

  • For Qwen3-8B, task-agnostic instructions’ union prevalence falls from 29.8% at τ=0.01 to 10.1% at τ=0.10, comparable to evaluative feedback at 29.1% and 9.2%.Solution-level privilege consistently has the highest prevalence; the same relative pattern across intervention groups holds for Qwen3-4B-Thinking-2507.Source: §A.3 · p. 18

Retention Across Intervention Groups

  • At τ=0.10, retention into the solution group remains at least 93.4% for Qwen3-8B and 94.3% for Qwen3-4B-Thinking-2507.Retention is asymmetric: positions affected under instruction, evaluation, and answer interventions are largely retained under solution-level privilege, while reverse retention is substantially lower and declines at stricter thresholds.Source: §A.3 · p. 19

Semantic Consistency

  • At τ=0.10, answer-level agreement reaches 93.6% for Qwen3-8B and 94.5% for Qwen3-4B-Thinking-2507, with corresponding SDR values of 76.9% and 79.6%.Across thresholds, positional overlap generally decreases as fewer tokens remain significant under both interventions, while directional agreement stays consistently high; solution-level interventions show higher overlap but lower SDR.Source: §A.3 · p. 19, Figure 8 · p. 19

Robustness Across Thresholds and Teacher Scales

  • Across configurations, the qualitative conclusions remain unchanged: TSD emerges without task-specific knowledge, richer privilege preserves and expands affected positions, and contrasting semantics induce aligned, predominantly shared TSD.The analysis spans three significance thresholds and two teacher scales.Source: §A.3 · pp. 18–19

A.4 Token-Level TSD Patterns Across Contextual Interventions

Across contextual interventions, TSD-prone token forms remain broadly stable, especially at the ranking extremes, even as richer privileged context increases deviation magnitudes. Surface-language expressions are more TSD-prone than mathematical and symbolic forms.

Stability Across Intervention Polarity

  • Positive and negative intervention variants share 18/24, 22/24, 24/24, and 23/24 top-ranked forms for instruction, evaluation, answer-level privilege, and solution-level privilege, respectively.Their bottom-24 overlaps are 24/24, 23/24, 24/24, and 24/24 in the same order, so reversing polarity or correctness generally leaves the most TSD-prone forms unchanged.Source: §A.4 · pp. 19–20

Stability Across Intervention Types

  • Across all eight intervention conditions, any two share at least 18/24 top-ranked forms and 22/24 bottom-ranked forms; 16/24 top forms and 22/24 bottom forms occur in every ranking.High-ρτ forms are dominated by natural-language expressions, while low-ρτ forms are dominated by digits, mathematical symbols, and notation.Source: §A.4 · pp. 19–20, Table 6 · p. 20
  • Mean top-24 ρτ ranges from 61.1% to 91.5%, versus 3.5% to 8.2% for bottom-24 forms, an approximately 11–18× gap within individual conditions.This separates natural-language surface expressions from predominantly numerical and symbolic forms across interventions.Source: §A.4 · p. 20

Context Modulates Strength More Than Token Identity

  • Under positive interventions, top-24 mean |Δ| rises from 0.080 under instruction and 0.084 under evaluation to 0.182 under answer-level privilege and 0.436 under solution-level privilege.Negative interventions show a similar overall increase, while largely the same surface-form tokens remain TSD-prone and numerical or symbolic forms comparatively stable.Source: §A.4 · pp. 20–21

A.5 Trajectory-Level Case Study of TSD

Trajectory examples show that TSD is broadly distributed across reasoning traces rather than confined to mathematical content or identifiable reasoning errors. Deviations frequently occur on discourse and reasoning-management expressions.

TSD Throughout a Correct Trajectory

  • In the correct rollout, strong deviations frequently occur on expressions such as Hmm, Wait, But, However, Maybe, and Alternatively, while much numerical and symbolic content remains comparatively stable.This pattern appears even though the student reaches the correct answer early and repeatedly revisits its interpretation.Source: §A.5 · p. 24

TSD Is Not Necessarily Localized Around Reasoning Errors

  • In the incorrect rollout, substantial deviations do not concentrate around the identifiable conceptual error or repeated 93.5, but appear broadly on expressions such as First, Hmm, Wait, Alternatively, and Therefore.The correct relation is x=65, yet even a privileged solution directly resolving the mistake does not make the induced likelihood variation a localized reasoning-error signal.Source: §A.5 · pp. 24–25

Takeaway

  • Across correct and incorrect trajectories, TSD is broadly distributed and particularly pronounced on tokens that organize or redirect reasoning, rather than selectively aligning with mathematical correctness or error locations.The examples motivate not treating context-induced teacher variation indiscriminately as transferable supervision.Source: §A.5 · p. 25

A.6 Training Details and Hyperparameters

The training details specify a shared OPD recipe, token-level distillation settings, and method-specific configurations for the compared baselines and Cal-OPD. Cal-OPD uses evaluative feedback interventions to estimate the teacher self-deviation region rather than distilling those interventions directly.

Shared Training Configuration

  • All methods train for 100 steps with 256 on-policy trajectories per step, one rollout per question, one PPO epoch, and a mini-batch size of 256.The shared setup uses response length 16,384, temperature 1.0, top-p 1.0, AdamW at 1 × 10^-6, cosine scheduling, weight decay 0.01, and gradient clipping at 1.0; entropy and KL regularization are zero.Source: §A.6 · p. 25, Table 10 · p. 26

Distillation-Specific Configuration

  • The distillation module queries teacher token-level log-probabilities on student-generated trajectories, using top-16 teacher tokens and including all student tokens in the loss.Student and teacher chunk sizes are 1024 and 128; the reinforce-mode per-token loss is clamped at 10.Source: §A.6 · p. 25

Method-Specific Hyperparameters

  • ExOPD uses λexo=1.25, EOPD applies forward KL above teacher entropy 0.8 with mixing coefficient 1.0, and Uni-OPD targets a correct-to-incorrect ratio of 0.5 with margin 0.4.Uni-OPD calibrates margins within groups of rollouts for the same question; Table 11 summarizes method-specific hyperparameters.Source: §A.6 · pp. 25–26, Table 11 · p. 27

Privileged-OPD and Cal-OPD

  • Cal-OPD uses evaluative feedback interventions with relaxation factor λ=5 to estimate the TSD region, and those interventions are not directly distilled into the student.Privileged-OPD instead provides the verified reference solution only to the teacher, prepended to the problem and student prefix; solutions exceeding the 12,288-token prompt limit are truncated.Source: §A.6 · p. 27

A.7 Additional Training Dynamics

Across additional training dynamics, efficiency, and retention-matched controls, Cal-OPD’s behavior depends on calibration scope. Retaining the residual beyond the estimated TSD region outperforms controls that only reduce signal magnitude or remove tokens.

A.7 Additional Training Dynamics

  • Cal-OPD’s top-16 token overlap rose to approximately 0.693 while OPD had higher probability-mass overlap, consistent with broader token coverage rather than stronger dominant-mass matching.The dynamics suggest Cal-OPD covers more teacher-supported token patterns while preserving a broader student distribution and multiple plausible reasoning modes.Source: §A.7 · p. 27, Figure 9 · p. 28
  • Solution-level calibration over-filtered discrepancy: entropy rose to about 0.36 as response length fell to roughly 5.5K tokens, leaving too little effective supervision.The estimated TSD region is excessively broad, and this variant has the lowest downstream performance among the reported variants.Source: Figure 9 · p. 28, §A.7 · pp. 28–29
  • Instruction-level calibration tracked OPD closely: entropy stabilized near 0.20, top-16 probability-mass overlap near 0.886, and token overlap near 0.684.The calibration signal is too weak to separate TSD from transferable knowledge, so the method largely degenerates to standard OPD behavior.Source: §A.7 · p. 28
  • Privileged-OPD kept entropy near 0.25 and top-16 mass overlap near 0.880, whereas solution-level calibration reached about 0.36 entropy and 0.859 overlap.The passages attribute these distinct failure modes to privileged-OPD’s overconfident shortcut collapse and solution-level calibration’s signal collapse.Source: Figure 10 · p. 29, §A.7 · p. 29

A.8 Efficiency Analysis

  • 14.90 vs. 18.80 minutes per step for Qwen3-4B-Thinking-2507→Qwen3-1.7B; larger-teacher settings instead took 50.05 vs. 35.19 and 48.76 vs. 31.26.Cal-OPD adds two teacher forward passes per step, which dominate when the 30B-A3B teacher is substantially larger than the student; it can reduce total training time when the size gap is moderate.Source: Table 12 · p. 29, §A.8 · pp. 29–30

A.9 Retention-Matched Control on Teacher–Student Discrepancy

  • OPD’s pre-clipping gradient norms fell from about 120 and 65 to 35 and 15; with clipping threshold C=1, raw gradients were already attenuated by one to two orders of magnitude.Cal-OPD changes the relative contribution of token advantages by removing the TSD-explained component, not by reducing overall gradient magnitude.Source: Figure 11 · p. 30, §A.9 · p. 30
  • 50.8 for Advantage-Sync matched OPD, 50.1 for Token-Sync, and 52.7 for the strongest TSD-Filter remained below Cal-OPD; tighter thresholds scored 51.1 and 49.4.The controls show that global attenuation, random token removal, and hard TSD-based filtering do not reproduce Cal-OPD’s gains; its soft calibration retains discrepancy beyond the estimated TSD region.Source: Table 13 · p. 31, §A.9 · p. 31
Loading 2609.21619v1…