PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Disagreement Is Learnable: Token Teachability in On-Policy Distillation

Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, Hongxia Yang

arXiv:2605.26844v1cs.LG

TL;DR

Selective OPD methods prioritize salient tokens, but raw disagreement does not reveal which teacher signals are learnable. This paper introduces token teachability and TA-OPD, finding that selecting teachable positions can often outperform full-token OPD with only 5% retained tokens across Qwen settings.

  • Problem

    Selective OPD lacks evidence about which token-level teacher signals are actually learnable, because entropy and disagreement measure salience rather than learning effect.

  • Method

    A fixed-context diagnostic formalizes support-aligned teacher–student disagreement as token teachability and uses it to select high-teachability positions for TA-OPD.

  • Results

    With only 5% retained tokens, TA-OPD often matches or surpasses full-token OPD and improves over entropy- and divergence-based methods across Qwen3 and Qwen2.5 settings.

  • Takeaways & Limitations

    Selective OPD should prioritize locally learnable teacher signals rather than merely salient tokens.

  • Takeaways & Limitations

    The evaluation focuses on math-heavy reasoning prompts and Qwen-family pairs, while the diagnostic measures same-context KL reduction and token budgets are not wall-clock savings.

Abstract

from arXiv · show

On-policy distillation (OPD) trains a student on its own rollouts with token-level teacher supervision. Recent selective OPD methods exploit the non-uniformity of OPD signals by prioritizing high-entropy or high-disagreement tokens. We revisit this principle and ask: which token-level teacher signals are actually learnable? Using a fixed-context diagnostic that measures same-context teacher-student KL reduction, we show that raw KL disagreement is a coarse proxy for learning value. It conflates learnable disagreement, where the teacher assigns corrective mass to the student's top-K candidates, with incompatible disagreement, where the teacher places mass mostly off the student's current support. We formalize this local compatibility as token teachability and show that it better predicts fixed-context improvement than raw KL alone. Motivated by this finding, we propose Teachability-Aware OPD (TA-OPD), a lightweight token-position selection method that applies OPD loss to high-teachability positions without reward models or verifiers. Across Qwen2.5 and Qwen 3 teacher-student settings, TA-OPD often surpasses full-token OPD with only 5% retained tokens and improves over entropy- and divergence-based baselines. Our results reframe selective OPD as selecting learnable teacher signals rather than merely salient tokens.

1 Introduction

The introduction argues that OPD’s dense token supervision is non-uniform in learning value: raw KL disagreement can mix learnable and incompatible signals. It presents token teachability and TA-OPD, which selects support-aligned supervision and often matches or surpasses full-token OPD with 5% retained tokens.

  • Motivation: OPD trains students on their own rollouts with token-level teacher supervision, reducing distribution mismatch but exposing unequal usefulness across token signals.The teacher supervises states the student actually visits, yet dense supervision is not uniformly valuable.
  • Motivation: Selective OPD methods prioritize high-entropy or high-disagreement positions, but these criteria measure salience rather than the learning effect of token supervision.Two tokens with similar KL disagreement can differ in their actual learning value.
  • Diagnostic: The fixed-context diagnostic measures same-context teacher–student KL reduction to separate token-level learning value from rollout, sampling, context-shift, and positional-interaction effects.It freezes student-generated prefixes as a context bank and compares initial and trained students against the same teacher distribution.
  • Token teachability: Raw KL disagreement conflates learnable disagreement, where corrective teacher mass lies within the student’s top-K support, with incompatible disagreement, where teacher mass falls mostly off support.Support-aligned disagreement better predicts useful learning and KL-disagreement reduction than raw disagreement alone.
  • TA-OPD: 5% retained tokens: TA-OPD often matches or surpasses full-token OPD and improves over budget-matched entropy- and divergence-based methods across Qwen3 and Qwen2.5 settings.TA-OPD scores positions by support-aligned teacher–student disagreement, applies OPD only to high-teachability positions, and requires no reward models or verifiers.

2 Related Work

Prior distillation work emphasizes matching teacher and student behavior, while recent OPD analyses link gains to high-probability student tokens and compatible reasoning patterns. This work operationalizes compatibility at the token level by separating learnable from incompatible disagreement.

  • LLM distillation: LLM distillation transfers teacher behavior through outputs, intermediate representations, or generated trajectories to compress reasoning, instruction-following, and alignment capabilities.The supplied passage frames distillation as useful under limited compute.
  • LLM distillation: Recent OPD analyses attribute gains to alignment on high-probability student tokens and compatible reasoning patterns.
  • LLM distillation: Self-distillation and context-conditioned distillation emphasize that effective supervision depends on the information state and trajectory distribution.
  • LLM distillation: Token-level decomposition of disagreement into learnable and incompatible components reframes selective OPD as identifying signals the student can absorb.

3 Diagnosing Token Teachability

This section shows that raw KL disagreement is a coarse measure of OPD learning value because it mixes support-aligned, learnable corrections with off-support mismatch. A fixed-context diagnostic and local-support decomposition identify token teachability as the stronger predictor of improvement, including within the high-KL Q3 region.

  • Fixed-context diagnostic: Fixed-context rescoring isolates local KL reduction by freezing on-policy prefixes and evaluating checkpoints on the same states.This controls rollout resampling variance and avoids conflating token learning with downstream context changes or answer-level success.
  • Local-support decomposition: Teachability measures whether teacher corrections are locally reachable: high compatibility mass indicates support-aligned reweighting, whereas low mass indicates off-support mismatch.The decomposition distinguishes learnable disagreement, which is large and reachable, from incompatible disagreement, which is large but off-support.
  • Predictive evidence: 0.086–0.087 vs. 0.043–0.045: across K = 8, 16, 32, learnable disagreement DL has roughly twice the standardized regression coefficient of incompatible disagreement DI.Positive bootstrap gaps of +0.041–+0.044 show that DL carries the more stable fixed-context gain signal after controlling for entropy, position, and teacher entropy.
  • Q3 intervention: Within Q3, high-DL tokens produce positive gain, while low-DL tokens produce negative gain and high-DI tokens are weak.Q3 is therefore informative but coarse: teachability separates learnable high-KL supervision from off-support mismatch.
  • Robustness: Positive high-support gaps persist across held-out contexts, GSM8K-COT prompts, stronger 8B/14B teachers, and support sizes.All main prompt-cluster intervals remain above zero, supporting teachability as a local property rather than an artifact of one run.

4 Teachability-Aware OPD

TA-OPD selects token positions using teachability: disagreement weighted by how much teacher probability lies on the student’s top-K support. It applies the standard reverse-KL OPD objective to retained positions, while fixed-context KL reduction remains a diagnostic; the selector achieves the largest reduction and efficiency at 3% and 5% budgets.

  • Training objective: The training objective is reverse-KL OPD on masked positions, whereas fixed-context KL reduction is used only for diagnosis.Token budgets count KL-supervised response positions rather than proportional wall-clock savings.
  • Teachability score: Teachability measures locally learnable disagreement by combining disagreement with teacher mass on the student’s top-K support.High support coverage indicates support-aligned correction, whereas low coverage indicates off-support disagreement.
  • Token selection: TA-OPD retains the top-n valid response positions by teachability under retention ratio ρ, alongside budget-matched entropy, disagreement, random, and full-OPD selectors.The method can optionally mix entropy with teachability to test complementarity with the entropy axis.
  • Implementation: TA-OPD requires no reward model, verifier, or additional labels, and typically computes disagreement and compatibility from teacher and student top-K log-probabilities.Unless stated otherwise, K = 16; when full teacher scores are unavailable, the support-coverage estimate uses a lower bound.
  • Fixed-context selector check: At 3% and 5% budgets, TA-OPD gives the largest fixed-context KL reduction and gain per kept token among the compared selectors.Table 2 defines G3% and G5% as same-context KL reductions and G/K as reduction normalized by actual retained ratio.

5 Experiments

Across four teacher–student settings, TA-OPD achieves the best average score at a 10% supervised-token budget, with gains especially evident for smaller or cross-backbone students. Budget sweeps further show that performance is non-monotonic, supporting token quality over token quantity as the key constraint.

  • Main benchmark results: TA-OPD achieves the best average score in all four teacher–student settings at a 10% supervised-token budget.The averages are 44.89, 56.87, 54.65, and 30.62 across the four groups.
  • Main benchmark results: For Qwen3-4B to Qwen3-1.7B, TA-OPD raises average score over Full OPD from 42.37 to 44.89 and leads on AIME24, GPQA-Diamond, and IFEval.The gains are strongest when the student is smaller or the teacher–student pair is mismatched.
  • Main benchmark results: In stronger-teacher settings, TA-OPD leads selected benchmarks but is not universally best: it leads AIME24 and MATH-500 with the GRPO-tuned teacher, while TA-OPD+Entropy is strongest on AIME25 with the 14B teacher.TIP or entropy remain competitive on HumanEval and IFEval, and TA-OPD nearly matches Full OPD in average score for the 14B teacher.
  • Budget sweep: Budget performance is non-monotonic: for Qwen3-14B to Qwen3-4B, TA-OPD is best at 10% with average 54.65, while TA-OPD+Entropy is strongest at 5% with 54.47.Increasing the budget to 30% or 50% does not consistently improve average performance, and several metrics regress relative to 5–10% settings.
  • Budget sweep: 57.89 is the best 5% average score for Qwen3-8B-GRPO to Qwen3-4B, achieved by TA-OPD+Entropy; TA-OPD gives the best MATH-500 score at 5%.TA-OPD also gives the best average among teachability-based selectors at 50%, reaching 57.90.

6 Conclusion

The conclusion identifies token teachability as a better lens for learning value in on-policy distillation than raw KL disagreement alone. It uses this support-aware concept to motivate Teachability-Aware OPD, a lightweight token-position selection method.

  • The study asks which token-level teacher signals are actually learnable in on-policy distillation.
  • Raw KL disagreement is a coarse proxy because it conflates learnable disagreement with incompatible disagreement.Learnable disagreement has high teacher-mass coverage over the student’s top-K support, whereas incompatible disagreement falls mostly off support.
  • Token teachability formalizes support-aware learning value and motivates Teachability-Aware OPD, a lightweight token-position selection method.

7 Limitations … D Local-Support Decomposition

The paper’s limitations concern evaluation scope, safety guarantees, and the interpretation of selective supervision. Its diagnostic protocol uses fixed contexts and local-support metrics to assess whether teacher corrections remain near the student’s salient alternatives.

  • 7 Limitations: The analysis centers on math-heavy reasoning prompts and Qwen-family teacher–student pairs, with only one cross-backbone distillation setting.Broader multilingual, dialogue, code-specialized, and non-Qwen evaluations are identified as needed to test generality.
  • 7 Limitations: TA-OPD filters supervised token positions rather than pruning transformer computation, so its reported token budget should be interpreted accordingly.
  • A Use of AI Assistants.: The authors used large language model assistants for language polishing, grammar checking, and formatting refinement, while retaining responsibility for the technical content.They state that the ideas, method, experiments, analyses, claims, and final writing were developed, verified, and approved by the authors.
  • B Ethical Considerations: TA-OPD can reduce exposure to noisy or incompatible teacher signals but provides no guarantees about factuality, safety, bias, or harmful content.Unsafe teacher behavior may still transfer when its tokens appear locally learnable, so practical use should include data filtering and safety evaluation.
  • C Fixed-Context Diagnostic Protocol: The fixed-context diagnostic scores all checkpoints on the same bank of valid student response prefixes, preventing differences caused by resampling rollouts.Support diagnostics compare student and teacher top-K token sets, while compatibility measures teacher mass on the student support; scalars are normalized by clipping the 5–95 percentile range to [0, 1].
  • C Fixed-Context Diagnostic Protocol: Fixed-context gain uses full-vocabulary probabilities when available and top-K support diagnostics for robustness checks when they are not.Prompt-cluster bootstrap confidence intervals account for heavy-tailed, rollout-correlated token gains.
  • D Local-Support Decomposition: Local-support decomposition stores student and teacher top-K sets and restricts analysis to locally salient alternatives rather than full-vocabulary tail noise.Scalar scores use within-rollout-batch quantile normalization, consistently applied to entropy, disagreement, and compatibility-derived scores.
  • D Local-Support Decomposition: The decomposition audits top-K overlap, top-K Jaccard, shared teacher top-K mass, and CBC as proxies for how much teacher correction remains near student support.

E Low-Entropy with High-Divergence Heterogeneity Diagnostics

Within the low-entropy, high-divergence region, high-DL token selection is the only clearly positive live intervention, while support-aligned selectors remain positive in exact-budget controls. This pattern is robust across multiple support proxies, which produce positive high–low gaps.

  • Live quadrant intervention: High-DL selection is the only clearly positive live intervention in Q3, whereas low-DL is negative and high-DI is weak.All three masks retain roughly the same token ratio.
  • Exact-budget fixed-context controls: Support-aligned selectors remain positive under exact-budget fixed-context controls, while random selection is consistently negative.Every selector keeps the same number of target tokens.
  • Support-definition audit: Teacher-mass, top-K, Jaccard, and CBC support proxies all yield positive high–low support gaps within Q3.The quadrant result is stable across these alternative support definitions.

F Regression Diagnostics for Learnable Disagreement

The regression diagnostic predicts fixed-context token gains using standardized predictors and compares compatibility-based extensions with a baseline disagreement model. Standardized coefficients and prompt-cluster gaps are emphasized, while nonlinear binned checks test whether the observed separation depends on linearity.

  • Diagnostic specification: The dependent variable is the fixed-context gain Gfix_i,t, with scalar predictors standardized within the diagnostic dataset.The baseline includes student entropy, local disagreement, their interaction, normalized token position, and teacher entropy.
  • Diagnostic specification: The diagnostic compares adding compatibility Ct alone against replacing raw disagreement with the decomposed pair DL_t and DI_t.These extensions are evaluated relative to the baseline predictor set.
  • Inference and reporting: Confidence intervals use prompt-cluster bootstrap, matching the diagnostic’s prompt-level evaluation structure.Prompt-cluster gaps are also used as a primary diagnostic quantity.
  • Inference and reporting: Standardized coefficients and prompt-cluster gaps are prioritized because token gains are heavy-tailed and noisy, while R2 is reported only as an incremental sanity check.The stated source of noise is that a single local update can affect later prefixes and formatting tokens.
  • Nonlinear sanity check: Binned residualized gains by DL, DI, and Ct do not indicate that the Fig 3A separation is an artifact of a purely linear model.The nonlinear sanity check uses the same fixed-context bank.

G Budget and Downstream Boundary Checks · H Impact to Model Fusion

TA-OPD’s benefit is concentrated in a modest supervision budget and depends on selecting tokens with both teacher correction and local support alignment. The downstream selector ablation favors TA-OPD over alternative selectors, while model fusion provides a broader context for distillation-based capability aggregation under limited compute.

  • G Budget and Downstream Boundary Checks: The fixed-context gain peaks around 3–5% retained supervision and saturates at 10%, indicating that selected-supervision quality limits improvement more than budget alone.This budget trend is reported for TA-OPD and low-entropy with high-divergence compatibility controls.
  • G Budget and Downstream Boundary Checks: Macro-averaged performance is non-monotonic: TA-OPD or TA-OPD+Ent. is strongest at several low-budget points, while larger budgets help or hurt across teacher and benchmark mixes.Table 11 summarizes the Qwen3-4B student budget sweep from Table 4.
  • G Budget and Downstream Boundary Checks: 54.65 average score: TA-OPD leads at the same 10% supervised-token budget, ahead of raw KL (53.76), C-only selection (54.19), Q3-only selection (52.46), and Q3+high-C selection (53.32).The comparison uses the Qwen3-8B-GRPO to Qwen3-4B setting.
  • G Budget and Downstream Boundary Checks: The selector ablation supports requiring both a meaningful teacher correction and local support alignment for useful OPD targets.TA-OPD’s advantage over raw KL, C-only, Q3-only, and Q3+high-C selection motivates this interpretation.
  • G Budget and Downstream Boundary Checks: Early smoke checks found that GSM8K-COT followed the fixed-context trend, whereas MATH-hard and capped AIME served only as low-resolution boundary checks.The complete downstream evidence is reported in the main benchmark tables, while the sparse smoke table is omitted.
  • G Budget and Downstream Boundary Checks: Within the low-entropy, high-divergence TIP region, high teachability is beneficial while low teachability is harmful.This live-intervention result is summarized in Table 7.
  • G Budget and Downstream Boundary Checks: Table 8 provides an exact top-N matched fixed-context analysis at K = 16, controlling for the number of retained tokens across selectors.The analysis functions as a confound check because all selectors retain exactly the same number of tokens.
  • H Impact to Model Fusion: Model fusion aggregates capabilities from multiple LLMs under limited compute through parameter-space merging or distillation-based fusion, alongside work on scaling, budget-aware parameter management, calibration, quantization, and weight-space geometry conflict.Related work also studies implicit or logit-level fusion, though the supplied passage is truncated.

I Proof of Proposition 1

The proof begins by defining the token gradient and update, then invokes β-smoothness of the fixed-context loss. The surrounding analyses characterize teachability through decomposition, budget sweeps, and selector ablations.

  • Proof setup: The proof defines g_t = ∇_θℓ_t(θ) and Δ_t = −ηg_t before applying β-smoothness of L_fix.
  • Empirical support: Learnable disagreement has a consistently larger standardized coefficient than incompatible disagreement in the prompt-cluster bootstrap regression decomposition.The reported ΔR2 decomposition is measured in 10^-3 units relative to the entropy+divergence baseline.
  • Budget analysis: The teachability selector peaks around 3–5% effective target-token budget and saturates at 10%.
  • Selector ablation: The downstream selector ablation states that TA-OPD is not reducible to raw KL, compatibility mass C, or the low-entropy with high-divergence Q3 region.
Loading 2605.26844v1…