Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation

Zhiwei Zhang; Zechen Sun; Fei Zhao; Kang Peng; Bin Liang; Huayu Deng; Yao Hu; Kam-Fai Wong; Mu Chuan

Source-linked AI summary

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation

Zhiwei Zhang, Zechen Sun, Fei Zhao, Kang Peng, Bin Liang, Huayu Deng, Yao Hu, Kam-Fai Wong, Mu Chuan

arXiv:2609.02998v1Sep 2, 2026cs.LGcs.AIcs.CL

TL;DR

Vanilla OPD supplies dense supervision without verifying whether the teacher is reliable for each prompt, risking propagation of confident errors. TGOPD verifies teacher reliability with verifier-scored probes, routes passing prompts to OPD and failing prompts to GRPO, and improves performance across evaluated settings while using idle teacher capacity.

Problem
Vanilla OPD applies dense teacher supervision uniformly across prompts despite prompt-varying reliability and does not directly verify teacher correctness.
Method
TGOPD uses verifier-scored teacher probes to estimate prompt-level reliability, routing each prompt exclusively to dense OPD or verifier-grounded GRPO.
Results
TGOPD outperforms Vanilla OPD in all six domain–scale settings and raises the seven-benchmark multi-domain average by 1.14 points at 4B and 0.95 points at 35B.
Takeaways & Limitations
Prompt-level verification preserves dense teacher supervision when it passes and selectively withholds it when reliability fails, while reusing otherwise-idle teacher capacity.
Takeaways & Limitations
The implementation omits GRPO’s standard-deviation normalization, and controlled comparisons with normalized variants remain future work.

Abstract

from arXiv · show

On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.

1 Introduction

OPD provides efficient dense token-level supervision but can propagate confident teacher errors when reliability varies by prompt. TGOPD verifies teacher reliability with probes, then routes each prompt to OPD or verifier-grounded GRPO, improving results while reclaiming idle teacher capacity.

Motivation: OPD converts trajectory-level supervision into dense per-token guidance through reverse-KL learning from a stronger teacher.This dense feedback can enable teacher-level accuracy roughly an order of magnitude faster than RLVR.
Motivation: 59% of the measured hour has teacher-node utilization below 5% under Vanilla OPD, motivating probe generation during otherwise-idle capacity.The measured 4B run averages 9.8% teacher-node GPU utilization while rollout and training nodes remain busy.
Reliability diagnosis: 84% of the teacher’s highest-confidence sampled responses are incorrect on low-reliability code prompts, compared with 61% on math.Because reverse KL concentrates probability on high-probability teacher behavior, such confident errors can be amplified by Vanilla OPD.
Reliability diagnosis: Teacher self-confidence separates reliable from unreliable prompts poorly on code, with AUROC 0.51 versus 0.73 on math.The diagnostic uses ten offline teacher responses per prompt; q(10) is their verifier pass rate.
TGOPD: TGOPD estimates prompt-level reliability from verifier-scored teacher probes and uses dense OPD only when the reliability check passes.When the check fails, it routes the prompt to verifier-grounded GRPO instead of blending the two signals.
Results: TGOPD outperforms Vanilla OPD in all six domain–scale settings across mathematics, code, and instruction following.The largest gains occur on code, where confidence is least informative about teacher reliability.

2 Preliminaries

The preliminaries contrast dense but unverifiable OPD supervision with coarse but verifier-grounded GRPO. TGOPD makes this supervision choice per prompt using verifier-scored teacher probes and exclusive routing.

Setup and notation: The student samples groups of on-policy rollouts, and a binary verifier scores each rollout against a ground-truth outcome.The verifier uses unit tests for code and a rule-based judge for mathematics and instruction following.
On-policy distillation: OPD minimizes mode-seeking reverse KL and assigns each sampled token a teacher–student log-likelihood-ratio advantage.The coefficient β scales the teacher signal, while every token receives its own weighted learning signal.
On-policy distillation: OPD’s token-level signal does not use the verifier, so it measures student–teacher distance without directly assessing teacher correctness.This supports sample efficiency but can concentrate the student on the teacher’s highest-probability behavior.
GRPO with verifiable rewards: GRPO replaces unavailable or withheld teacher supervision with a verifier-grounded scalar advantage shared across every token in a rollout.Its sign is determined by the verifier outcome relative to the group mean, making the signal trajectory-level rather than token-specific.
The gap: TGOPD addresses prompt-varying reliability by routing each prompt exclusively to OPD or GRPO rather than applying one fixed rule or blending signals.The verifier is applied to teacher probes as well as student rollouts.

3 Teacher-Gated On-Policy Distillation

TGOPD estimates teacher reliability per prompt using verifier-scored probes, then routes each prompt exclusively to dense OPD or verifier-grounded GRPO. The gate admits teacher supervision only when the reliability estimate passes a threshold, while overlapping probes with idle teacher time preserves asynchronous efficiency.

Teacher Reliability Probe and Gate: TGOPD audits teacher reliability on each prompt with a small set of independently generated teacher probes scored by the verifier.The empirical pass rate qT(x) estimates expected verifier reward; with binary rewards and i.i.d. probes, it is unbiased, though finite-budget decisions remain approximate.
Teacher Reliability Probe and Gate: The hard gate opens when qT(x) reaches threshold τ, admitting dense OPD; otherwise, it withholds teacher supervision and uses verifier-grounded GRPO.The gate opens when at least ⌈τKT⌉ probes pass, and the main experiments use KT = 3 and τ = 2/3.
Gate-Conditioned Supervision Routing: TGOPD retains dense OPD’s token-level signal when the audit passes and falls back to trajectory-level verifier supervision when it fails.The fallback can still train the prompt when the student rollout group contains reward variation, assigning group-relative advantages to attempts above or below the mean.
Gate-Conditioned Supervision Routing: OPD and GRPO are selected rather than blended, so each prompt receives exactly one supervision signal.The binary selector makes Vanilla OPD and pure GRPO degenerate endpoints when the gate is fixed open or closed for every prompt.
Asynchronous Implementation: Teacher scoring remains unconditional, as in Vanilla OPD, while both candidate advantages are formed before the gate selects the update signal.The scoring pass is a batched forward evaluation over already-generated student tokens and is not the throughput bottleneck.

4 Experiments

TGOPD improves distillation across domain and scale settings by verifying teacher reliability per prompt, routing reliable prompts to dense OPD and unreliable ones to verifier-grounded alternatives. Its gains extend to multi-domain training and coincide with substantially higher teacher utilization, while threshold and fallback ablations clarify how the gate contributes.

4.2 Main Results: TGOPD outperforms Vanilla OPD in all six domain×scale settings, with largest average gains on code (+3.0 at 4B, +2.9 at 35B).It also ranks first among distillation methods on 7 of 14 in-domain benchmark columns and surpasses the domain teacher on six.
4.2 Main Results: At 35B, TGOPD achieves positive transfer on LiveCodeBench (+3.0 over base), while every other distillation method falls below the untrained base model.TGOPD also surpasses the teacher by +1.3 on LCB and +1.1 on OJBench in this setting.
4.2 Main Results: Multi-domain seven-benchmark averages rise from 53.40 to 54.54 (+1.14) at 4B and from 60.99 to 61.94 (+0.95) at 35B.TGOPD wins 6 of 7 columns at 4B and 5 of 7 at 35B.
4.3 Extension to Multi-Domain OPD: Probe generation raises teacher utilization to 78.9% and 82.8% under SOPD and to 66.6% and 57.7% under MOPD at 4B and 35B, respectively.Idle time falls to 0–2% throughout, with lower MOPD saturation because probing is domain-routed.
4.5 Ablation: Gate Threshold: Accuracy peaks near τ = 3/5: AIME 2026 reaches 73.1, AIME 2025 64.4, and HMMT-Feb 58.0, exceeding Vanilla OPD by +1.9, +3.3, and +3.7.Higher thresholds reject useful teacher signals; HMMT-Feb falls below Vanilla OPD at τ =5/5 (52.0 vs. 54.3).
4.6 Ablation: What the Gate Does After It Closes: Masking improves over the ungated baseline by +1.08 points on average, versus +1.20 for the full GRPO fallback, leaving roughly 90% of TGOPD’s benefit after blocking alone.The fallback leads in three of four settings, but masking wins at 4B MOPD.

5 Related Work

Prior work makes OPD more reliability-aware through distributional signals or outcome evidence. These approaches motivate instance-aware teacher supervision but leave room for prompt-level verification before applying dense rewards.

Foundations: OPD uses student rollouts and teacher token-level supervision, unlike RLVR’s single trajectory-level reward.Dense feedback can improve sample efficiency relative to sequence-level reward supervision.
Motivation: Existing analyses identify failures when teacher knowledge is not locally exploitable, student states drift, or stronger teachers induce negative transfer.Domain routing does not verify the selected teacher on each prompt before applying its token-level rewards.
Distributional reliability: Distributional methods regulate OPD using teacher entropy, teacher–student agreement, likelihood ratios, or compatibility signals.These methods adapt the distillation signal without directly establishing outcome correctness.
Outcome-aware reliability: Outcome-aware methods use verifier agreement, environmental correctness, or verifier-scored continuations to regulate trajectory- or token-level supervision.They provide complementary evidence for reliability-aware distillation.

6 Conclusion

TGOPD addresses Vanilla OPD’s unconditional trust in teacher supervision with a verifier-based prompt-level gate. It improves performance across evaluated domains and scales, while reclaiming idle teacher capacity, with automatic verification and binary routing remaining scope boundaries.

Method: TGOPD admits dense OPD only when verifier-scored teacher probes pass a prompt-level reliability audit; otherwise it routes prompts to verifier-grounded GRPO.The method selectively withholds teacher supervision rather than weakening it for every prompt.
Results: TGOPD outperforms Vanilla OPD in all six single-domain settings across 4B and 35B students in mathematics, code, and instruction following.It also increases the seven-benchmark average by 1.14 and 0.95 points at the corresponding scales in multi-domain training.
Compute: 9.8% to 78.9% teacher-node GPU utilization is achieved in the measured 4B single-domain run by using otherwise-idle teacher capacity for probes.A matched 35B code run incurs a 5.9% increase in mean step time.
Scope: TGOPD currently requires an automatic verifier and makes a binary routing decision.The paper identifies open-ended tasks and uncertainty-aware gates as directions for future work.

A Contributors

The paper lists Zhiwei Zhang, Zechen Sun, Fei Zhao, Kang Peng, Bin Liang, Huayu Deng, Yao Hu, Kam-Fai Wong, and Mu Chuan as contributors.

Contributors: Contributors are Zhiwei Zhang, Zechen Sun, Fei Zhao, Kang Peng, Bin Liang, Huayu Deng, Yao Hu, Kam-Fai Wong, and Mu Chuan.

B Training Hyperparameters and Infrastructure

The experiments retain the same optimization and online-probing protocol across model scales while changing model-dependent infrastructure settings. The 4B setup uses five 8-GPU nodes, with separate trainer, rollout, and teacher roles.

Configuration: The method is defined by the gate and routed advantage in equation 6, while IcePop/TIS is an implementation-level safeguard for asynchronous execution.
Configuration: Both model scales follow the same optimization and online-probing protocol; only model-dependent system settings differ.
Infrastructure: The 4B runs use five 8-GPU nodes: two trainers, two student-rollout nodes, and one teacher node.For 4B 16K runs, static memory fraction and request concurrency are capped at 0.60 and 64, respectively.
Online probing: At both scales, the frozen teacher draws three fresh online probe responses per prompt, and the gate opens when at least two pass verification.The reliability test is unchanged across scales despite different resource topologies.

C Evaluation Benchmark Details

The evaluation benchmarks are documented with independent-run counts and reported metrics, with multi-run scores averaged across generations. LiveCodeBench and OJBench use specified task aggregations.

Table 7 records each benchmark’s independent generation runs and the metric reported in the main evaluation tables.For benchmarks with multiple runs, the reported score is averaged across those runs.
LiveCodeBench evaluation includes only its code-generation subtask.
OJBench reports an overall score aggregated across its C++ and Python splits.

D Per-Configuration GPU Utilization Traces

Figure 6 shows one-hour, 15-second-resolution utilization traces across single- and multi-domain configurations. TGOPD fills baseline idle intervals, while multi-domain routing creates domain-dependent oscillations; matched timing shows the overlap is substantial but not free.

All eight traces use 15-second samples from steady-state one-hour windows, with both curves in each panel sharing an axis.The panels cover single- and multi-domain OPD at 4B and 35B scales.
5.9%: TGOPD’s mean step time increase across 500 aligned 35B CodeIO rollout cycles, while mean decode throughput changes by less than 0.1%.
Baseline utilization alternates between brief scoring spikes and long zero-utilization stretches caused by waiting for student rollouts.These idle intervals are not removed by changing teacher size or scoring batching because their duration is set by rollout generation.
TGOPD keeps utilization in a high band by consuming intervals between scoring passes without exceeding baseline peak loads.The matched timing comparison indicates that this overlap is substantial but incomplete.
Multi-domain traces swing between roughly 40% and 85% because only the domain-specialist engine owning the current prompt is active.Single-domain traces remain near 79–83%, whereas multi-domain node occupancy changes with batch domain mix.