PaperlayerView exact revision on arXiv

Source-linked AI summary

Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation

Zhiwei Zhang, Zechen Sun, Fei Zhao, Kang Peng, Bin Liang, Huayu Deng, Yao Hu, Kam-Fai Wong, Mu Chuan

arXiv:2609.02998v1cs.LGcs.AIcs.CL

TL;DR

Vanilla on-policy distillation applies dense teacher supervision uniformly, without verifying reliability for each prompt. TGOPD uses verifier-scored teacher probes to route prompts to distillation or verifier-grounded GRPO; it outperforms Vanilla OPD across all six single-domain settings and raises multi-domain seven-benchmark averages at both scales.

  • Problem

    Vanilla OPD and GRPO-only training use fixed supervision rules across prompts, although teacher reliability and the preferable learning signal may vary by prompt.

    Source: §2 · p. 3

  • Method

    TGOPD uses verifier-scored teacher probes to estimate prompt-level reliability, routing each prompt exclusively to dense OPD if the check passes or verifier-grounded GRPO otherwise.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    By 1.14 points at 4B and 0.95 points at 35B, TGOPD raises the seven-benchmark average in multi-domain training and outperforms Vanilla OPD in all six single-domain settings.

    Source: §6 · p. 12

  • Takeaways & Limitations

    Results support verifying teacher reliability per prompt, retaining dense supervision when the audit passes and withdrawing it when it fails.

    Source: §1 · p. 3

    A finite probe budget does not guarantee reliable accept/reject decisions, although the main experiments found K_T = 3 sufficient.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.

1 Introduction

TGOPD verifies teacher reliability per prompt before admitting dense on-policy distillation, using otherwise-idle teacher capacity for probes. It outperforms Vanilla OPD across six domain–scale settings while increasing teacher-node utilization.

1 Introduction

  • OPD converts student rollouts into dense reverse-KL token guidance, reaching teacher-level accuracy roughly an order faster than RLVR, but mode-seeking can amplify confident errors [5].Unlike RLVR's single scalar reward per trajectory, OPD provides a teacher signal at every token.Source: §1 · pp. 1–2
  • Mean teacher-node GPU utilization rises from 9.8% to 78.9%, and cluster utilization from 51.5% to 69.5%, with modest measured end-to-end overhead.In asynchronous OPD, the teacher scores completed student rollouts and otherwise waits for a batch to become ready.Source: Figure 1 · p. 1, §1 · pp. 1–2
  • TGOPD uses repeated verifier-scored teacher outcomes for prompt-level admission before dense supervision, unlike prior trajectory- or token-level outcome controls.Distributional methods EOPD [6], TrOPD [24], and REOPOLD [7] capture uncertainty, compatibility, or optimization risk rather than directly testing correctness; RG-OPD [2], RLSD [26], and SPOT [16] use outcome feedback for trajectory- or token-level decisions.Source: §1 · p. 2
  • Self-confidence barely separates higher- from lower-reliability prompts on code (AUROC 0.51), versus 0.73 on math; among low-reliability prompts, highest-confidence sampled answers are incorrect 84% on code and 61% on math.Source: §1 · p. 2
  • Verifier-scored teacher probes estimate prompt reliability; TGOPD routes each prompt exclusively to dense OPD when qT(x)≥τ or verifier-grounded GRPO otherwise, if student rewards vary.The two branches are routed rather than blended.Source: §1 · p. 2
  • TGOPD outperforms Vanilla OPD in all six domain–scale settings across 4B and 35B students in mathematics, code, and instruction following.The largest gains occur on code, where confidence is least informative about teacher reliability.Source: §1 · p. 3

2 Preliminaries

The preliminaries contrast dense but unverified token-level OPD with verifier-grounded trajectory-level GRPO, motivating per-prompt exclusive routing between signals based on teacher probes.

Setup and notation

  • The setup uses a student policy, a frozen possibly domain-routed teacher, groups of on-policy rollouts, and a binary verifier for each outcome.The verifier uses unit tests for code and a rule-based judge for mathematics and instruction following.Source: §2 · p. 3

On-policy distillation (OPD)

  • OPD minimizes mode-seeking reverse KL using teacher–student log-likelihood ratios for dense token-level updates, which can mislead when the teacher is unreliable.Its objective compares student and teacher distributions but does not use the verifier to assess correctness.Source: §2 · p. 3

GRPO with verifiable rewards

  • GRPO [17] centers verifier rewards within each rollout group and assigns every token the same trajectory-level advantage, whose sign reflects the verifier outcome relative to the group mean.The presentation omits the standard-deviation normalization [13].Source: §2 · p. 3

The gap

  • TGOPD addresses prompt-varying teacher reliability by using the verifier on teacher probes and student rollouts, routing each prompt exclusively to OPD or GRPO.Vanilla OPD and GRPO-only training apply a fixed supervision rule uniformly across prompts; TGOPD never blends the two signals on one prompt.Source: §2 · p. 3

3 Teacher-Gated On-Policy Distillation

TGOPD verifies teacher reliability per prompt with verifier-scored probes, admitting dense OPD only when a hard threshold passes and otherwise using verifier-grounded GRPO. Probe decoding overlaps the teacher’s otherwise-idle window in asynchronous training.

3 Teacher-Gated On-Policy Distillation

  • TGOPD uses a hard prompt-level gate to select either dense OPD or verifier-grounded GRPO; the two signals never jointly supervise a prompt.Vanilla OPD and pure GRPO are the gate’s two endpoints when the gate is fixed open or closed for every prompt.Source: §3 · p. 3, §3.2 · p. 5

3.1 Teacher Reliability Probe and Gate

  • Unlike entropy or teacher–student likelihood agreement, the gate uses verifier-scored outcomes to estimate prompt-level teacher reliability before choosing a supervision branch.Distributional proxies measure uncertainty or compatibility but cannot by themselves distinguish equally confident correct and incorrect answers.Source: §3.1 · p. 4
  • With KT=3 and τ=2/3, the gate opens when at least two of three verifier-scored teacher probes pass.The empirical pass rate estimates the probability that a teacher sample solves the prompt and is unbiased; a KT=5 threshold sweep finds a broad optimum around a simple majority.Source: §3.1 · pp. 4–5

3.2 Gate-Conditioned Supervision Routing

  • When the gate closes, verifier-grounded GRPO controls the update if student rollouts have reward variation; equal rewards yield zero centered advantage.Both candidate advantages are formed for each prompt, but only the selected signal enters the update. The implementation omits GRPO standard-deviation normalization as a convention, not a gate requirement.Source: §3.2 · p. 5

3.3 Algorithm and System Realization

  • Teacher probes run concurrently with student generation, overlapping otherwise-idle teacher time; teacher scoring remains unconditional as in vanilla OPD.In the measured run, teacher-node GPU utilization rose from 9.8% to 78.9% with modest end-to-end overhead because most probe work used previously idle capacity.Source: §3.3 · p. 6

4 Experiments

Across two student scales and three domains, TGOPD beats Vanilla OPD in every single-domain setting and improves multi-domain benchmark averages while reclaiming idle teacher capacity. Ablations show that a majority-vote-like threshold works well and that blocking unreliable teacher signals accounts for most of the gains.

4.1 Experimental Setup

  • Experiments use 4B and 35B students across math, code, and instruction following, with frozen domain-specific GRPO teachers and matched student initialization, prompt pools, and training pipelines.Baselines include Vanilla OPD, TrOPD, RG-OPD, and an RLSD-style variant; evaluations use in-domain math, code-generation, and instruction-following benchmarks.Source: §4.1 · pp. 6–7

4.2 Main Results

  • TGOPD beats Vanilla OPD in all six domain-by-scale settings, with average gains of +3.0/+2.9 on code, +1.6/+1.9 on IF, and +1.5/+1.2 on math at 4B/35B.TGOPD ranks first among distillation methods in 7 of 14 in-domain benchmark columns and surpasses the domain teacher in six.Source: Table 1 · p. 7, §4.2 · pp. 7–8

4.3 Extension to Multi-Domain OPD

  • At 4B, TGOPD raises the seven-benchmark average from 53.40 to 54.54 (+1.14), winning 6 of 7 columns; at 35B, it rises from 60.99 to 61.94 (+0.95), winning 5 of 7.The multi-domain gate routes each prompt to its domain-specialist teacher and uses KT = 3 verifier-scored probes, without modification to the gating procedure.Source: Table 2 · p. 8, §4.3 · p. 8

4.2 Main Results

  • At 35B, TGOPD is the only compared distillation method with positive LiveCodeBench transfer (+3.0 over base), surpassing the teacher by +1.3 LCB and +1.1 OJBench.Every other distillation method scores below the untrained base model on LCB; at 4B, TGOPD closes 55% of the base-to-teacher LCB gap versus 21% for Vanilla OPD.Source: §4.2 · p. 8

4.4 Compute Efficiency: Reclaiming Idle Teacher Capacity

  • Teacher probes raise utilization from 5.0–9.8% without probes to 78.9%/82.8% in SOPD and 66.6%/57.7% in MOPD (4B/35B), while idle time falls to 0–2%.Cluster-wide utilization gains are +18.0/+11.8 points in SOPD and +12.7/+8.6 in MOPD (4B/35B); stable rollout-node utilization indicates no sustained contention but does not establish zero wall-clock overhead.Source: Table 3 · p. 9, Figure 4 · p. 9, §4.4 · pp. 9–10

4.5 Ablation: Gate Threshold

  • With KT = 5 on 4B math, all three benchmarks peak at τ = 3/5: AIME 2026 scores 73.1, AIME 2025 64.4, and HMMT-Feb 58.0.At τ ≤ 2/5 the gate filters less and gains over Vanilla OPD are smaller; at τ ≥ 4/5 useful signals are rejected, with HMMT-Feb falling to 52.0 versus 54.3 at τ = 5/5. The peak is near the default τ = 2/3 and suggests a broad optimum around majority vote.Source: Figure 5 · p. 10, §4.5 · p. 10

4.6 Ablation: What the Gate Does After It Closes

  • Masking alone retains roughly 90% of TGOPD’s benefit, while GRPO fallback wins in three of four settings; masking wins 4B MOPD (+1.57 vs. +1.14).The fallback is the preferred default for consistency, though the margins are small; it can provide a relative signal for mixed-outcome rollout groups and is inert when all outcomes are uniform.Source: Table 4 · p. 11, Table 5 · p. 11, §4.6 · p. 11

5 Related Work

Related work spans on-policy and multi-teacher distillation, distribution-based reliability signals, and methods incorporating outcome evidence. TGOPD verifies repeated teacher rollouts at the prompt level before routing each prompt exclusively to OPD or GRPO.

5.1 On-Policy Distillation and Multi-Teacher Frameworks

  • OPD trains on student rollouts with dense teacher supervision, whereas multi-teacher OPD routes prompts to domain experts to combine capabilities in one student.Reverse-KL supervision is mode-seeking [5] [1] [19], and dense feedback can improve sample efficiency compared with sequence-level rewards; vanilla OPD applies it uniformly across prompts, while multi-teacher systems route by domain [14] [15] [22].Source: §5.1 · p. 12
  • TGOPD addresses a gap in multi-teacher OPD: domain-based routing does not verify teacher reliability on each prompt before admitting dense supervision.Prior analyses identify failures from locally unusable teacher knowledge, prefix drift, and negative transfer to smaller students [9], [4], [10].Source: §5.1 · p. 12

5.2 Reliability-Aware Distillation and the Confidence–Correctness Gap

  • Distribution-based OPD methods adapt guidance through model-distribution signals to address uncertainty, diversity, distribution mismatch, or optimization stability, but do not directly verify outcome correctness.EOPD applies forward KL at high-entropy teacher tokens [6], TrOPD uses teacher–student decoding agreement [24], REOPOLD stabilizes implicit token rewards [7], and REOPD extrapolates token rewards without an external verifier [21]; GATES uses sampled tutor-trace agreement when no external grader is available [20]. PW-OPSD finds high teacher entropy can reflect non-viable uncertainty or benign solution diversity [12], while ESR finds off-policy degradation is not fully explained by KL or entropy [29].Source: §5.2 · p. 12
  • Prior outcome-aware methods condition supervision on verifier feedback, environmental correctness, or verifier-scored candidates, while TGOPD verifies repeated complete teacher rollouts before routing each prompt exclusively to OPD or GRPO.RG-OPD uses verifier feedback and a likelihood gap [2]; RLSD uses environmental correctness [26]; SPOT scores student continuations [16]; related data-selection methods verify static responses or tokens [3] [18] [25]. TGOPD makes a prompt-level decision rather than blending signals or calibrating trajectories, tokens, or candidate branches.Source: §5.2 · p. 12

6 Conclusion

TGOPD verifies teacher reliability with prompt-level, verifier-scored probes before allowing dense OPD, routing prompts that fail the audit to verifier-grounded GRPO. It outperforms vanilla OPD across the reported single- and multi-domain settings while reusing idle teacher capacity, but requires an automatic verifier and makes binary routing decisions.

6 Conclusion

  • TGOPD beats vanilla OPD in all six single-domain settings and raises the seven-benchmark average by 1.14 and 0.95 points in the corresponding multi-domain settings.The results cover 4B and 35B students in mathematics, code, and instruction following.Source: §6 · p. 12
  • TGOPD admits dense OPD only when a prompt-level audit of verifier-scored teacher probes passes, routing other prompts to verifier-grounded GRPO.This selectively withholds teacher supervision rather than weakening it on every prompt.Source: §6 · p. 12
  • Teacher-node GPU utilization rises from 9.8% to 78.9% in the measured 4B single-domain run, while a matched 35B code run increases mean step time by 5.9%.The probes largely reuse otherwise-idle teacher capacity.Source: §6 · p. 12
  • TGOPD requires an automatic verifier and binary routing, motivating future work on outcome-grounded reliability estimation for open-ended tasks and uncertainty-aware gates.Source: §6 · p. 12

B Training Hyperparameters and Infrastructure

The 4B and 35B experiments use the same online teacher-probing rule, while infrastructure scales with the model. Within controlled comparisons, Vanilla OPD and TGOPD share training settings and topology; IcePop/TIS is an asynchronous-engine safeguard, not part of the method definition.

Training configuration

  • Vanilla OPD and TGOPD share student initialization, data, optimization settings, and system topology in controlled comparisons; IcePop/TIS is an asynchronous-engine safeguard, while the method is defined by the gate and routed advantage in equation 6.The retained run-record settings are reported for both model scales.Source: B Training Hyperparameters and… · p. 14
  • Table 6 summarizes training configurations by model scale, with both scales following the same optimization and online-probing protocol and differing only in model-dependent system settings.The supplied caption does not specify the individual configuration values.Source: Table 6 · p. 15

Infrastructure

  • The 4B runs use five 8-GPU nodes, while the 35B runs use seven 8-GPU nodes, with trainer, rollout, and teacher roles allocated across them.The 4B topology is two trainer nodes, two student-rollout nodes, and one teacher node; the 35B topology is four trainer nodes, two student-rollout nodes, and one teacher node. For 4B 16K runs, SGLang static memory fraction is capped at 0.60, and maximum running requests and teacher-server concurrency at 64.Source: B Training Hyperparameters and… · p. 15

Probe realization

  • At both scales, the frozen teacher generates three fresh online probe responses per prompt, and the gate opens when at least two pass the verifier.The same reliability test is used at 4B and 35B; resource topology changes with scale, but the training rule does not.Source: B Training Hyperparameters and… · p. 15

C Evaluation Benchmark Details

In-domain evaluation reports main-table scores averaged across independent generation runs when multiple runs are used. LiveCodeBench results cover only code generation, while OJBench reports an aggregate across its C++ and Python splits.

C Evaluation Benchmark Details

D Per-Configuration GPU Utilization Traces

One-hour utilization traces show TGOPD probes filling the baseline teacher’s rollout-phase idle gaps, while multi-domain utilization oscillates with domain routing. Matched timing shows the audit overlaps substantially with idle capacity but is not strictly free.

D Per-Configuration GPU Utilization Traces

  • All eight steady-state traces cover one hour at 15-second resolution, with both curves in each panel sharing a common axis.The raw traces preserve the time axis absent from the mean, idle-fraction, and marginal-distribution summaries.Source: D Per-Configuration GPU Utilization… · p. 16
  • 5.9% increase in mean step time accompanies TGOPD in the matched 35B CodeIO runs, while mean decode throughput changes by less than 0.1%.The comparison covers 500 aligned rollout cycles under identical settings; the audit is not strictly free.Source: D Per-Configuration GPU Utilization… · p. 16
  • Baseline teacher utilization alternates between scoring-pass spikes and zero-utilization student rollout intervals, which shrinking the teacher or changing scoring batches would not remove.The idle fraction in Table 3 measures these intervals, whose duration is set by student rollout rather than teacher scoring speed.Source: D Per-Configuration GPU Utilization… · p. 16
  • TGOPD probes occupy gaps between baseline scoring passes without exceeding baseline peaks, filling teacher idle time while matched timing shows the overlap is substantial but incomplete.The figure plots one-hour traces for each configuration, with shaded regions indicating reclaimed teacher capacity.Source: D Per-Configuration GPU Utilization… · p. 16, Figure 6 · p. 17
  • Multi-domain TGOPD utilization swings between roughly 40% and 85%, versus a tight 79–83% band for single-domain runs, because only the engine for the current prompt domain is active.The three specialist engines are allocated 4 GPUs to code, 2 to math, and 2 to instruction following; this routing produces 0–2% idle fraction despite lower multi-domain means.Source: D Per-Configuration GPU Utilization… · p. 16

Related papers

Loading 2609.02998v1…