PaperlayerView exact revision on arXiv

Source-linked AI summary

SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling

Haoran Xu, Hongyu Wang, Yifei Gao, Jiaze Li, Xiaofeng Zhang, Xiaosong Yuan

arXiv:2606.09304v1cs.CLcs.LG

TL;DR

On-policy distillation can depend on fragile student–teacher trajectory alignment and unreliable teacher preferences across tokens. SG-OPD uses verifier-guided teacher sampling and sign-consistency gating, improving over OPD and ExOPD across four competition-level math benchmarks while remaining stable under stronger extrapolation.

  • Problem

    OPD relies on fragile trajectory-level student–teacher alignment and uniform token-level teacher reliability, including on verifier-correct rollouts.

  • Method

    SG-OPD uses a binary verifier as a teacher-trust signal through phased teacher sampling and sign-consistency-gated token-level extrapolation or interpolation.

  • Results

    Across four competition-level math benchmarks, SG-OPD improves over OPD and ExOPD and remains stable at extrapolation strengths that cause uniform extrapolation to diverge.

  • Takeaways & Limitations

    The results support verifier-guided control of teacher trust at both trajectory and token granularities within competition-level mathematical reasoning.

  • Takeaways & Limitations

    Evaluation is limited to competition-level mathematics with binary trajectory-level verification, and the approach has not yet been validated on other verifier-style tasks.

Abstract

from arXiv · show

On-policy distillation (OPD) trains a student on its own trajectories with dense per-token supervision from a stronger teacher, and often outperforms off-policy distillation and standard reinforcement learning. However, we find that its effectiveness implicitly relies on two assumptions that frequently break in practice: trajectory-level alignment between the student and the teacher, and uniform token-level reliability of the teacher's preferences. We therefore propose Sign-Gated On-Policy Distillation (SG-OPD), which uses a binary verifier as a trust signal for the teacher at two complementary granularities: phased teacher sampling mixes in verifier-endorsed teacher rollouts at cold-start, and a sign-consistency gate extrapolates the distillation update on tokens where the teacher agrees with the verifier-correct direction and interpolates it where it disagrees. Experiments on competition-level mathematical reasoning benchmarks show that SG-OPD consistently outperforms standard OPD, with average gains of 1.98 and 7.50 at the per-sample and per-question levels, respectively.

1 Introduction

SG-OPD addresses two recurring weaknesses of on-policy distillation: fragile trajectory-level alignment and nonuniform token-level teacher reliability. It combines phased teacher sampling with sign-consistency gating, using a verifier to strengthen reliable supervision and mute conflicting signals.

  • Motivation: OPD can fail when student cold-start trajectories diverge from teacher support, making reverse-KL supervision noisy.This reflects a broken trajectory-level alignment assumption in strong-to-weak settings.
  • Motivation: Even on verifier-correct rollouts, teacher token preferences can oppose verifier-correct directions and penalize tokens supporting valid trajectories.This reflects nonuniform token-level teacher reliability.
  • Method: SG-OPD uses a binary verifier as a teacher-trust signal at trajectory and token granularities.Its two mechanisms are phased teacher sampling and sign-consistency-gated extrapolation or interpolation.
  • Method: Phased teacher sampling mixes verifier-endorsed teacher rollouts early, then anneals toward fully on-policy student rollouts.The schedule bridges cold-start trajectory mismatch while retaining on-policy training later.
  • Method: Sign-consistency gating extrapolates consensus-token updates and interpolates conflict-token updates.Consensus means teacher agreement with a verifier-correct direction; conflict means disagreement.
  • Results: Experiments on competition-level mathematical reasoning benchmarks show SG-OPD consistently outperforms existing on-policy distillation baselines and remains stable where uniform extrapolation collapses.The method lets students inherit reliable teacher supervision while backing off when teacher preferences are unreliable.

2 Related Work

Related work spans on-policy distillation and mixed-policy optimization. OPD provides dense per-token feedback from student trajectories, while recent methods address mismatch or combine supervised and verifiable-reward training.

  • On-policy distillation: OPD samples from the student and minimizes reverse KL to provide dense per-token feedback, but incurs student–teacher mismatch.Classical KD instead fits the student to a frozen teacher, commonly through teacher-response SFT or sequence-level SeqKD.
  • On-policy distillation: Video-OPD addresses cross-modal misalignment, while G-OPD recasts OPD as a related alternative.The supplied passage introduces G-OPD’s reframing but does not include its full description.
  • Mixed Policy optimization: GRPO and successors optimize binary verifiable rewards with group-normalized advantages, following the spirit of classical trust regions.This line of work motivates mixed-policy optimization alongside on-policy distillation.
  • Mixed Policy optimization: Recent mixed-policy methods integrate SFT and GRPO through post-hoc weight extrapolation, fixed-prior SFT re-weighting, or teacher-trajectory mixing.Examples include ExPO, CHORD, and teacher-trajectory mixing, which bridges cold-start with off-policy expert data; DFT is described as analogous.

3 Preliminaries and Failure Modes of OPD

This section formalizes the student–teacher setup and verifier- and token-level OPD signals, then identifies fragile trajectory alignment and non-uniform teacher reliability as OPD failure modes. These observations motivate using the verifier as a teacher-trust signal at two granularities.

  • Setup: A prompt produces a student trajectory with binary verifiable outcome reward, under student πθ, frozen teacher π∗, and reference policy πref.
  • Verifier signal: The sample-level verifier signal a1(t) is GRPO-style reward normalization across G rollouts, constant per binary-reward trajectory, and used only as teacher-trust supervision.
  • OPD signals: OPD uses the dense reverse-KL token advantage a2(t)=log πθ(yt) −log π∗(yt); λ=1 recovers OPD, while λ>1 extrapolates beyond the teacher and larger λ degrades training.
  • Failure mode 1: OPD’s trajectory-level alignment assumption fails at cold-start when weak-student rollouts are unlikely under the teacher, making reverse-KL updates noisy; increasing λ amplifies this mismatch noise.
  • Failure mode 2: Even verifier-correct rollouts contain conflict tokens where a1(t)a2(t) ≤0, and a non-trivial fraction persists throughout training, revealing non-uniform teacher reliability.

4 Method: SG-OPD

SG-OPD combines phased teacher sampling with token-level sign-consistency gating, using a binary verifier to guide training at sample and token granularity. Verified teacher guidance stabilizes cold-start and is annealed away, while token updates extrapolate consensus signals and soften conflicts.

  • Method overview: SG-OPD uses a binary verifier at two levels: sample-level phased teacher sampling and token-level routing by advantage-sign agreement.The method couples verifiable-reward RL with OPD through a teacher anchor and a sign-consistency mechanism.
  • Phased Teacher Sampling: PTS injects verified teacher rollouts during cold-start, retaining only correct trajectories and discarding incorrect ones.A fraction ρ of each mini-batch is sampled from the teacher, and only verifier-approved trajectories define the auxiliary teacher-anchor loss.
  • Phased Teacher Sampling: The teacher anchor follows a three-phase cosine schedule and reaches α(t)=0, after which training becomes fully on-policy.The schedule uses α0, αend, and phase boundaries P1 and P2 to anneal teacher guidance away after cold-start.
  • Sign-consistency gating: The sign-consistency gate compares verifier-induced GRPO advantage a1(t) with teacher-induced OPD advantage a2(t), marking consensus when gt = 1 and conflict when gt =0.A uniform combination can amplify directions opposed by one signal, so SG-OPD routes tokens before forming the policy-gradient advantage.
  • Sign-consistency gating: Consensus tokens receive stronger extrapolation, whereas conflict tokens use softened OPD by default; β = 1 recovers OPD and β =0 masks conflicts.The gate-disabled formulation reduces to G-OPD, while detached clipping weights limit domination by large OPD outliers without changing token classification.

5 Experiments

SG-OPD is evaluated on four competition-level mathematical reasoning benchmarks under a controlled strong-to-weak distillation setup. It achieves the best aggregate avg@32 and pass@32 accuracy, while ablations show complementary benefits from sign-gating and phased teacher sampling.

  • Experimental setup: Training uses 57 K difficulty-filtered DeepMath problems, eight rollouts per prompt, and 100 optimizer steps; evaluation covers AIME24, AIME25, HMMT25-Feb, and HMMT25-Nov.Evaluation reports avg@32 and pass@32 with temperature 1.0, top-p 1.0, and a 16,384-token generation budget.
  • Main results: 29.53 avg@32 (+1.98 over OPD, +1.54 over ExOPD) and 59.17 pass@32 (+7.50 over OPD, +5.00 over ExOPD) are SG-OPD’s highest AVG results.The main comparison includes SeqKD, GKD, OPD, and ExOPD under the strong-to-weak setting.
  • Per-benchmark results: On avg@32, SG-OPD gains +5.00 over OPD on AIME25 and +2.39 on AIME24, matches OPD at 18.02 on HMMT25-Feb, and gains +0.52 on HMMT25-Nov.The largest AIME25 gain coincides with the highest sign-conflict fraction, while HMMT subsets remain in the 18–20% avg@32 range.
  • Per-benchmark results: 76.67 on AIME24 (+6.67 over OPD) and 66.67 on AIME25 (+16.67) show larger SG-OPD improvements under pass@32 than under avg@32.The passage attributes this exploration advantage to aggressive extrapolation on consensus tokens.
  • Ablations: At λhigh = 1.8, sign-gated extrapolation reaches 28.78, whereas uniform ExOPD collapses to 24.71; the gated result is +0.79 above the best uniform ExOPD.The sign-consistency gate widens the safe range of consensus-token extrapolation strength.
  • Ablations: PTS alone raises AVG from 27.55 to 28.59, the correctness filter is essential, and simultaneous SG-OPD reaches 29.53 versus 28.85 for time-separated training.The two granularities are complementary with nearly additive gains, and the strongest token-level configuration uses λhigh = 1.8 with interp and β = 1.

6 Analysis

SG-OPD’s evaluation gain is attributed to token-level gating rather than extra teacher access, with the gate addressing persistent sign conflicts in reasoning-pivot tokens. The sample-level anchor and token-level gate are complementary, though HMMT25-Feb remains a failure case.

  • Source of the gain: 29.53 vs 28.59 AVG: SG-OPD outperforms PTS-only on held-out benchmarks despite comparable training-set performance.The higher evaluation performance is therefore unlikely to come from additional teacher samples alone.
  • Source of the gain: High sign-conflict fractions persist throughout training, especially on high-magnitude reasoning-pivot tokens inside incorrect chains.The reported fraction counts only tokens with strictly non-zero advantages on both signals.
  • Why both granularities matter: The sample-level anchor and token-level gate are complementary: removing PTS leaves early exploration unresolved, while continued teacher injection can distort the mature on-policy distribution.SG-OPD pulls ahead once the student’s training-set accuracy is high enough for on-policy gradients to escape incorrect-trajectory dominance.
  • Failure modes and source of the gain: −2.19: SG-OPD does not improve over ExOPD on HMMT25-Feb, where elevated conflict rates can suppress benign tokens.The benchmark’s small problem set and multi-stage reasoning chains are identified as contributing factors.

7 Conclusion

SG-OPD couples verifiable-reward reinforcement learning with on-policy distillation through phased teacher sampling and a sign-consistency gate. Across four competition math benchmarks, it improves over OPD and ExOPD while remaining stable.

  • 7 Conclusion: SG-OPD couples verifiable-reward RL with OPD through phased teacher sampling and a sign-consistency gate.Phased teacher sampling anchors the student near a teacher-correct neighborhood at cold-start before annealing to zero; the gate uses the sign of verifiable advantage as a token-level certificate.
  • 7 Conclusion: Across four competition math benchmarks, SG-OPD improves over OPD and ExOPD and remains stable.

Limitations … E Sign-Agreement Case Study

The appendices detail SG-OPD’s limitations, derivations, variance considerations, hyperparameters, implementation, training curves, and sign-agreement diagnostics. The method is validated only on binary-verifier mathematical reasoning, while several transfer and scaling questions remain open.

  • Limitations: SG-OPD is validated only on competition-level mathematical reasoning with a binary trajectory-level verifier.The token-level gate has not yet been validated on other verifier-style tasks such as code with unit tests, tool use, or reward-model verification.
  • Limitations: Whether conflict fractions and safe extrapolation ranges transfer across model sizes or longer horizons without retuning the T = 100-step schedule remains open.The main results use a single Qwen3-1.7B / Qwen3-4B-RL-Math pair and fixed PTS phase boundaries.
  • A Additional Derivation Details: The appendix supplies derivation details for OPD and implementation formulas, while the main text defines SG-OPD’s advantages, sampling schedule, stability weight, and loss terms.The appendix states that these additions are not needed to follow the main algorithm.
  • B Variance Analysis of the Token-Level Approximation: The token-level approximation is exact in expectation at per-token discount 0 but introduces additional gradient variance.Its worst-case variance ratio is bounded by trajectory length T, although empirical variance is much smaller in the dense-credit regime.
  • C Full Hyperparameters: Experiments use Qwen3-1.7B-Non-Thinking as student and a step-500 Qwen3-4B-Non-Thinking-RL-Math checkpoint as teacher, with the reference initialized from the student.Shared tokenization makes the per-token reverse-KL advantage a2(t) well-defined without re-tokenization.
  • C Full Hyperparameters: Training uses 57 K filtered DeepMath-103K problems, 2,048-token prompt caps, 16,384-token response caps, and a T = 100 optimizer-step budget.Over-long prompts are filtered rather than truncated, and total_epochs is only a safety cap.
  • D Extended Training Curves: Across all four benchmarks, SG-OPD tracks ExOPD through step ∼25 during PTS warm-up and separates after step ∼30 when PTS turns off.The appendix provides per-benchmark training-reward curves generated from the best-run logs.
  • E Sign-Agreement Case Study: The appendix includes the full sign-agreement diagnostic referenced in §6.The diagnostic is presented as Fig. 7.

F Full Ablation Runs

The ablations examine SG-OPD across 50+ hyperparameter-sweep runs and compare its sign-consistency gating and phased teacher sampling against uniform extrapolation. They show that aggressive uniform extrapolation collapses across all four benchmarks, whereas SG-OPD recovers the same strength through sign-consistency gating, while cross-task robustness remains unclaimed.

  • The default recipe fixes the PTS ratio and phase schedule, sweeps λhigh over {1.5, 1.8}, and selects the conflict fallback using validation AVG.
  • The authors do not claim universal robustness across tasks, and cross-task transfer of these defaults remains future work.
  • Table 3 reports avg@32 for all 50+ hyperparameter-sweep runs, grouped by dominant mechanism and sorted by AVG within each group.
  • Uniform aggressive extrapolation collapses across all four benchmarks, while SG-OPD recovers the same extrapolation strength via sign-consistency gating.Figure 6 uses λ=1.8 for aggressive ExOPD and λhigh=1.8 for SG-OPD.

G Reproducibility

The reported results use a single training seed, although evaluation averages over 32 sampling seeds, and the authors defer formal multi-seed validation. Training requires substantial compute, while experimental design, code, and analysis were conducted by the authors despite limited language-model assistance.

  • Seed disclosure: ≤0.5% AVG variation across runs differing only in non-essential hyperparameters suggests the +1.98 improvement exceeds the sweep’s observed noise floor, but formal multi-seed study remains future work.Each Table 1 row uses a single training seed; avg@32 averages 32 evaluation sampling seeds and does not measure training-seed variance.
  • Compute footprint: Approximately 14 hours on 8 A100 80GB GPUs are required per T =100-step run, with the full 50+run sweep using roughly 7 000 A100-GPU-hours.Evaluation across four benchmarks at avg@32 takes approximately 40 minutes per checkpoint.
  • AI assistance: The authors used a large language model assistant for proofreading, writing-structure brainstorming, and converting summary tables to LATEX, while retaining responsibility for experimental design, code, and analysis.The assistance was limited to writing and formatting tasks rather than experimental execution or analysis.

H Failed and Alternative Designs

Several alternative designs failed to improve the final recipe: finer-grained gating and larger teacher-sampling budgets were ineffective, while multi-teacher distillation was unstable. Filtering wrong teacher trajectories matched gradient-level skipping, and full-vocabulary KL added insufficient benefit for its cost.

  • Two-level sign-consistency gate: 28.39: A four-level sign-consistency gate did not outperform the binary gate, likely because the verifiable reward is binary.The four levels distinguish a1a2 ∈ {++, +−, −+, −−}, whereas the binary gate uses {0, 1}.
  • Larger teacher-sampling ratio: 27.66–28.02: Doubling the teacher-sampling budget underperformed ρ=0.125 once the student reached ∼30% correctness.This result is consistent with PTS primarily helping during cold-start.
  • Skipping rather than filtering wrong teacher answers: Skipping wrong teacher trajectories at the gradient level was numerically identical to zeroing their loss contribution.This confirms that gradient mass on wrong teacher trajectories is the active variable.
  • Multi-teacher distillation: Multi-teacher distillation became unstable in the aref2 term when math and code teachers’ base distributions diverged on a token.The paper leaves a multi-teacher sign-consistency gate for future work.
  • Full-vocabulary KL: 1.7× slower: Full-vocabulary reverse-KL produced a ≤0.2 AVG improvement, insufficient to justify its compute cost.The reported results therefore use the token-level KL form, although the sign-consistency gate supports both forms.
Loading 2606.09304v1…