PaperlayerView exact revision on arXiv

Source-linked AI summary

When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation

Yinuo Jiang, Yongjie Ye, Zhou Tao, Xiang Zhuang, Qiang Zhang, Huajun Chen, Tiankai Li

arXiv:2608.03632v1cs.AI

TL;DR

Existing OPD selection methods may trust teacher signals that are weakly grounded in the task input. SA-OPD filters signals that combine low input-groundedness with high optimization impact, consistently outperforming Vanilla OPD and competitive selective baselines across LLM and VLM benchmarks.

  • Problem

    Existing OPD selection criteria overlook whether teacher token-level judgments are grounded in the current input rather than generic priors, formatting conventions, or reasoning templates.

  • Method

    SA-OPD estimates token-level input-groundedness and filters only signals with both low input-groundedness and high optimization impact, without external verification labels.

  • Results

    SA-OPD consistently outperforms Vanilla OPD and competitive selective methods across LLM and VLM settings, achieving the best or tied-best results across the reported benchmarks.

  • Takeaways & Limitations

    The results identify input-groundedness as a crucial axis for reliable dense supervision and support filtering high-impact, weakly grounded teacher signals in OPD.

  • Takeaways & Limitations

    Fixed selection ratios can be brittle across tasks and models, and overly aggressive filtering may discard valid teacher signals and weaken useful OPD supervision.

Abstract

from arXiv · show

On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals. Recent selective OPD methods improve this process by prioritizing signals that are confident, informative, or learnable. However, the assumptions overlook a fundamental failure mode of language models: their token-level judgments can be driven by input-agnostic language priors, formatting conventions, or stereotyped reasoning templates rather than task-specific evidence. We refer to such optimization-relevant but weakly input-grounded supervision as spurious signals in OPD, which may produce large gradients while contributing little task-improving direction. To mitigate this issue, we propose SA-OPD, a Spurious-Signal-Aware On-Policy Distillation framework that identifies and filters misleading token-level supervision based on input-groundedness and optimization impact. SA-OPD introduces a lightweight input-groundedness proxy estimating whether a token-level distillation signal truly depends on the input. It then filters only tokens that simultaneously exhibit low input-groundedness and extreme distillation divergence, thereby removing high-impact spurious updates and achieving fine-grained OPD optimization. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that SA-OPD consistently outperforms Vanilla OPD and competitive selective methods. These results establish input-groundedness as a key dimension for OPD supervision selection and offer a simple, effective strategy for mitigating spurious updates.

1 Introduction

The introduction identifies spurious teacher signals as a failure mode in on-policy distillation: token-level supervision may reflect language priors, formatting conventions, or stereotyped reasoning rather than task-specific input evidence. It proposes SA-OPD, which estimates input-groundedness and filters tokens that are both weakly grounded and highly divergent, with experiments showing consistent gains over Vanilla OPD and selective methods.

  • Motivation: On-policy distillation reduces distribution mismatch by supervising student-policy rollouts with dense token-level teacher signals.Unlike offline or teacher-generated trajectories, OPD trains on states induced by the student’s evolving policy.
  • Problem: Existing OPD selection methods prioritize informative, learnable, or confident signals but may treat teacher judgments as reliable despite language priors, formatting conventions, and stereotyped reasoning patterns.These factors can make token-level judgments weakly tied to the current task input.
  • Method: SA-OPD estimates input-groundedness by comparing teacher–student divergence under the original input context and a residual no-prompt context retaining the generated prefix.Little divergence change after removing the task input indicates that the signal is less input-grounded.
  • Method: SA-OPD filters only tokens with both low input-groundedness and high absolute teacher–student divergence, targeting high-impact updates weakly supported by the input.The framework requires no external verification labels.
  • Results: Experiments in large language model and vision-language model distillation show that SA-OPD consistently outperforms Vanilla OPD and competitive selective methods.The evaluation covers both LLM and VLM settings.

2 Related Work

Prior work established OPD as a broad post-training paradigm and increasingly emphasized token-level supervision selection. SA-OPD extends this literature by targeting teacher signals that are high-impact but weakly grounded in the current input.

  • On-Policy Distillation: OPD improves open-ended generation and reasoning through reverse-KL-style objectives and supervision on student-generated mistakes.Subsequent work addresses scalability, stability, and generalizability, while extending OPD to self-distillation, hybrid RL-distillation, and multimodal distillation.
  • Token-Level Selection: Token-level selection studies identify unequal token contributions to policy improvement or collapse, including high-entropy forking tokens, covariance-based criteria, branch points, and overconfident errors.These criteria come from recent reinforcement-learning studies of post-training behavior.
  • Token-Level Selection: Existing selection criteria assess uncertainty, divergence, or learnability but not whether teacher judgments are grounded in the current input.Consequently, signals driven by language priors, formatting conventions, or stereotyped reasoning templates may appear informative while remaining weakly input-grounded.
  • SA-OPD: SA-OPD computes teacher–student divergence with the original prompt and a residual no-prompt context, then filters tokens that are weakly input-grounded and high-impact.The framework treats barely changing divergence after prompt removal as evidence that the teacher signal is dominated by input-agnostic priors.

3 Method

SA-OPD identifies spurious OPD supervision as token-level signals that are both weakly input-grounded and highly impactful, then filters these signals using a lightweight prompt-removal proxy. Dynamic filtering bounds removed loss mass while retaining the core low-groundedness, high-impact criterion.

  • Signal decomposition: SA-OPD decomposes token-level teacher–student divergence into input-grounded and prefix-predictable prior-driven components.The grounded component varies with the current input, whereas the prior component is predictable from the response prefix alone.
  • Signal decomposition: Prior-dominated updates can have large magnitude but weak alignment with the task-improving direction, increasing stochastic gradient energy as nuisance gradients.The method therefore interprets coherent grounded updates as signal and prior-induced update energy as noise in an effective gradient SNR view.
  • Spurious-signal criterion: Spurious OPD signals are defined by the intersection of low input-groundedness and high optimization impact, rather than low groundedness alone.This targets updates that are both largely input-agnostic and sufficiently large to affect local optimization dynamics.
  • Input-groundedness estimation: The Input-Grounding Gap estimates input dependence by comparing teacher–student divergence with the original prompt against a no-prompt condition using the same student-generated prefix.A large gap suggests input-grounded judgment, while a small gap suggests generic language priors, formatting conventions, or stereotyped reasoning patterns.
  • Token-level filtering: The token filter selects tokens with simultaneously low input-groundedness and large absolute distillation signals, then optimizes reverse-KL only on retained tokens.Dynamic adaptation of selection ratios keeps removed loss mass bounded, avoiding excessive removal of useful distillation supervision.

4 Experiments

Experiments evaluate SA-OPD across LLM and VLM distillation settings, comparing it with standard and selective OPD methods under matched training conditions. SA-OPD achieves consistent gains across benchmarks and scales, while ablations and token analyses support combining input-groundedness with optimization impact.

  • Experimental setup: Experiments use Qwen3 and Qwen3.5 teacher–student pairs, with additional DeepSeek-R1 and Qwen3.5 pairings testing generalization.The main pairs are Qwen3-4B-Instruct →Qwen3-1.7B and Qwen3.5-35B-A3B →Qwen3.5-2B.
  • Experimental setup: The evaluation covers six VLM benchmarks for visual understanding and reasoning and five LLM benchmarks for mathematical reasoning.VLM tasks include EvoChart, MMIFEval, CountQA, MathVision, Geo3K, and MathVista; LLM tasks include AIME 2024, AIME 2025, Math500, AMC 2023, and a further benchmark.
  • Baselines: SA-OPD is compared with Vanilla OPD, ExOPD, TIP, and FiRe-OPD using the same data, models, and compute budget.The baselines include standard and selective OPD methods.
  • Performance: SA-OPD achieves the best result on all six visual tasks, raising average scores from 50.5 to 54.0 for visual understanding and from 60.4 to 63.5 for visual reasoning versus Vanilla OPD.It also outperforms the strongest prior OPD variant on every visual benchmark, including a +2.8 gain on CountQA.
  • Scalability: SA-OPD yields positive gains across model sizes in both VLM and LLM settings, indicating robustness to teacher–student scale variations.The scalability analysis tests whether effectiveness depends on model scale or teacher strength.
  • Ablation study: Ablations show that low input-groundedness and high divergence are jointly necessary: single-axis filters and proxy replacement underperform SA-OPD.Divergence-only filtering can remove useful correction signals, whereas input-groundedness-only filtering may remove tokens with limited optimization impact.
  • Filtered-token analysis: Filtered tokens include 69.9% content words, and many low-entropy tokens are high-impact and weakly input-dependent, showing that entropy alone misses some spurious signals.The analysis sampled 500 visual-understanding examples; many filtered content tokens were predictable from local textual context.

5 Conclusion · A Supplementary Theory

The paper identifies spurious signals as a key failure mode in on-policy distillation and proposes SA-OPD to filter high-impact teacher supervision that is weakly grounded in task inputs. Results suggest that input-groundedness improves the reliability, effectiveness, and stability of dense OPD supervision.

  • 5 Conclusion: SA-OPD addresses spurious signals as an unexplored failure mode in on-policy distillation.The framework targets misleading dense teacher supervision during student training.
  • 5 Conclusion: Dense teacher supervision can be high-impact yet weakly grounded in the task input.Such signals can trigger large but misleading student updates.
  • 5 Conclusion: Spurious supervision may cause students to inherit input-agnostic language priors from teachers.These priors are identified as one source of weakly grounded signals.
  • 5 Conclusion: Formatting conventions and stereotyped reasoning templates are additional sources of spurious teacher signals.Both are described as capable of driving supervision without strong task-input grounding.
  • 5 Conclusion: SA-OPD estimates the input-groundedness of token-level teacher–student divergence.This estimate is used to identify whether dense distillation supervision is grounded in the input.
  • 5 Conclusion: Input-groundedness is a crucial axis for reliable dense supervision in OPD.The conclusion frames grounding as central to selecting useful distillation signals.
  • 5 Conclusion: Filtering high-impact but weakly grounded teacher signals can improve OPD effectiveness and stability.The conclusion compares this implication against Vanilla OPD and competitive selective OPD baselines.

A.1 Prior-Induced Gradients Have Weak Input-Specific Alignment

Theorem 1 shows that prior-induced gradients have weak input-specific alignment when score directions depend weakly on the task input after conditioning on response context. Such updates can consume substantial gradient energy while contributing little coherent task improvement.

  • A.1 Prior-Induced Gradients Have Weak Input-Specific Alignment: κ_t ≪1 implies weak input-specific alignment even when the prior-induced gradient has large gradient energy.The condition is weak input dependence of the score direction after conditioning on C_t.
  • A.1 Prior-Induced Gradients Have Weak Input-Specific Alignment: The score function decomposes into a response-context-predictable component and an input-dependent residual δs_t.The residual satisfies E[δs_t | C_t] = 0, while C_t excludes the task input.
  • A.1 Prior-Induced Gradients Have Weak Input-Specific Alignment: The prior-induced gradient can align with the input-specific task direction only through the input-dependent score residual δs_t.This follows because the response-context-measurable component cannot contribute to input-specific alignment.
  • A.1 Prior-Induced Gradients Have Weak Input-Specific Alignment: For generic or template-like tokens, weak variation of the student score direction with the task input yields prior-driven updates that consume substantial gradient energy but add little coherent task improvement.This conclusion holds up to the scale of the two gradients and does not require the prior-driven gradient itself to be small.

A.2 Low-SNR OPD Updates Induce Parameter Drift

The analysis models low-input-grounded, prior-driven OPD updates as nuisance updates with little persistent task-improving alignment but non-negligible variance. Even when these updates have zero mean, their cumulative effect causes expected parameter drift that grows linearly with optimization steps.

  • Effective SNR: When prior-driven variation dominates, the effective batch-level SNR denominator is governed mainly by the second-moment energy of the prior-driven update.This identifies the variance contribution that makes low-SNR updates consequential despite weak directional alignment.
  • Nuisance-update model: Low-input-grounded, prior-driven updates can have little expected task-improving alignment while retaining non-negligible second-moment energy.The nuisance model requires only the absence of a persistent directional signal, not small update magnitudes.
  • Parameter drift: Under the martingale-difference assumption, zero-mean prior-driven OPD updates induce expected squared parameter deviation that grows linearly with K.The theorem compares the local SGD-style trajectory with a signal-only grounded reference trajectory.
  • Parameter drift: When the nuisance-update variance is constant, the expected squared drift reduces to η2Kv.This is the stated simplification when vk = v for every optimization step k.

B Experimental Details · B.1 Training Details

The experiments use a verl-based implementation with PyTorch 2.10, CUDA 12.9, Python 3.12, and eight NVIDIA H20 GPUs. Training hyperparameters are provided in Table 6.

  • B.1 Training Details: The implementation is built upon the verl framework.
  • B.1 Training Details: Experiments use PyTorch 2.10.
  • B.1 Training Details: The software environment uses CUDA 12.9.
  • B.1 Training Details: The experiments run with Python 3.12.
  • B.1 Training Details: Training is performed on 8 NVIDIA H20 GPUs.
  • B.1 Training Details: All training hyperparameters are detailed in Table 6.

B.2 Evaluation Details · C Detailed Algorithm

The paper evaluates SA-OPD zero-shot on broad LLM and VLM benchmark suites under specified sampling and generation settings. Its algorithm adds input-grounding-gap computation and token filtering to Vanilla OPD, with dynamic control of filtered loss mass.

  • B.2 Evaluation Details: VLM evaluation covers six benchmarks spanning visual understanding and visual reasoning, while LLM evaluation covers five mathematical-reasoning benchmarks.VLM benchmarks are EvoChart, MMIFEval, CountQA, MathVision, Geo3K, and MathVista; LLM benchmarks are Math500, AMC 2023, AIME 2024, AIME 2025, and MinervaMATH.
  • B.2 Evaluation Details: Benchmark scoring follows each task’s standard exact-match or official answer-extraction protocol.This protocol is applied across the listed LLM and VLM benchmarks.
  • B.2 Evaluation Details: All evaluations use zero-shot generation with temperature = 1.0 and top-p = 0.95.Maximum new tokens are 18,000 for LLM tasks, 4,096 for VLM visual reasoning, and 1,024 for VLM visual understanding.
  • C Detailed Algorithm: SA-OPD differs from Vanilla OPD only by computing the input-grounding gap and applying the resulting token-level filtering mask.Algorithm 1 summarizes the training procedure, which samples student rollouts, computes full-prompt and residual no-prompt coefficients, and derives the input-grounding gap.
  • C Detailed Algorithm: Dynamic FLMR constraints limit the filtered loss-mass ratio because fixed selection ratios can remove different loss amounts across tasks and training stages.The issue is especially pronounced in visual understanding, where filtered tokens may occupy a large fraction of the OPD objective; Algorithm 2 summarizes the adaptive procedure.
  • C Detailed Algorithm: The procedure constructs a candidate filtered set from low input-grounding-gap tokens and then optimizes the OPD loss on retained tokens.The final filtered set F is removed before updating θ with the resulting gradient.

D Computational Overhead · E Filtered Token Example

SA-OPD adds residual no-prompt teacher–student divergence computation to estimate input-groundedness, without introducing trainable parameters or extra backward passes. The filtered-token examples show that it removes tokens with high optimization impact but weak input grounding, likely reflecting language priors or template preferences.

  • D Computational Overhead: SA-OPD estimates token-level input-groundedness by recomputing the student-generated response under a residual no-prompt context.The same on-policy response is reused, while teacher–student divergence is evaluated without the original prompt.
  • D Computational Overhead: SA-OPD adds no trainable parameters, auxiliary networks, or additional backward passes.The extra scoring computations are used only to construct the filtering mask.
  • D Computational Overhead: The additional computation requires one extra student pass and one extra teacher pass under the residual context.These passes support the input-grounding-gap estimate used by SA-OPD.
  • D Computational Overhead: 2.64% to 7.53%: measured training overhead across tasks compared with Vanilla OPD.The empirical comparison uses the same experimental settings as the main results.
  • E Filtered Token Example: Each filtered-token visualization reports F, the full-prompt teacher–student divergence, and ∆, the input-grounding gap.Tokens selected by SA-OPD are outlined in red.
  • E Filtered Token Example: SA-OPD filters tokens with large optimization impact but small input-grounding gaps.Such supervision is more likely driven by language priors or template preferences than input-specific evidence.
Loading 2608.03632v1…