PaperlayerView exact revision on arXiv

Source-linked AI summary

Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillation

Kaiyuan Liu, Ziyuan Zhuang, Yang Bai, Bing Wang, Rongxiang Weng, Jieping Ye

arXiv:2605.13643v2cs.CL

TL;DR

In strong-to-weak OPD, a teacher-student advantage may persist even as feedback loses local discriminability. The paper uses a trajectory-specific margin change-point rule to stop dense supervision, raising five-benchmark average score from 36.8% with full OPD to 40.1% for a 1.7B student.

  • Problem

    In increasingly prevalent strong-to-weak settings, it remains unclear whether a teacher-student gap alone captures the local discriminability needed for dense OPD.

    Source: §1 · pp. 1–2

  • Method

    The paper measures teacher margins over a student's reachable candidates across sentence segments and halts dense supervision after a statistical change point.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    40.1% with the release rule versus 36.8% with full OPD for average score across five in-domain benchmarks (1.7B student); it also outperforms full OPD across student sizes and better preserves out-of-domain capabilities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Effective strong-to-weak OPD should assess whether teacher advantage remains locally teachable, not merely whether it exists.

    Source: §1 · p. 2

    The diagnostic release computation uses full-rollout analysis and does not inherently shorten generation, so training-efficiency gains are beyond this work's scope.

    Source: §4.1 · p. 8

Abstract

from arXiv · show

On-policy distillation (OPD) trains a student model on its own rollouts using dense feedback from a stronger teacher. Prior literature suggests that, provided teacher feedback is available, supervising the full sequence of response tokens should monotonically improve performance. However, we demonstrate that this assumption sometimes fails to hold in strong-to-weak OPD settings. While later segments of a generated trajectory may still exhibit a non-zero teacher-student advantage, they frequently lack the local contrast that makes dense feedback effective for prioritizing student learning. We term this failure mode local teachability collapse. The resulting principle is straightforward: supervision should concentrate on trajectory regions where the teacher's feedback remains discriminative, rather than uniformly covering the entire response. We operationalize this principle through a trajectory-specific release rule. This rule measures the teacher's margin over the student's top-$K$ candidate set, aggregates this margin across NLTK-tokenized sentence segments, and truncates dense OPD supervision upon detecting a BIC-style downward change point. Experimental results across strong-to-weak distillation tasks using the Qwen3 model family indicate that this release rule consistently outperforms standard full-trajectory OPD across five in-domain benchmarks at various student scales. Furthermore, compared to baseline distillation methods, our approach better preserves model capabilities on out-of-domain task. These results suggest that effective strong-to-weak OPD requires evaluating not only the availability of teacher guidance but also its local utility, ensuring that the generated feedback remains teachable.

1 Introduction

The paper identifies local teachability collapse in strong-to-weak on-policy distillation: teacher-student advantage can persist while local feedback loses contrast. It proposes a trajectory-specific rule to stop dense supervision when that contrast declines, improving in-domain results and preserving out-of-domain capabilities.

Introduction

  • OPD trains students on their own generated trajectories, but the assumption that supervising every response token is uniformly beneficial may fail in strong-to-weak distillation.Evaluating student-visited states can mitigate exposure bias associated with fixed offline teacher demonstrations [1].Source: §1 · p. 1
  • In strong-to-weak OPD, teacher feedback must remain locally discriminative to prioritize student learning; a persistent preference gap alone may not provide informative local guidance.The setting is increasingly relevant to on-device modeling and local deployments, where a highly capable teacher supervises a much smaller student.Source: §1 · pp. 1–2
  • Local teachability collapse is a sustained decline in teacher local contrast even as mean teacher-student advantage remains nonzero, leaving sampled-token feedback less discriminative.The paper links this concern to outcome-driven RL, where feedback that distinguishes alternatives is more useful for learning.Source: §1 · p. 2
  • The release rule measures the teacher’s top-1 versus top-2 margin over student-reachable candidates, aggregates it over NLTK-tokenized sentence segments, and masks dense OPD after a BIC-style downward change point.Before the change point, training retains the standard OPD advantage objective, isolating the effect of changing the supervision region.Source: §1 · p. 2
  • For a 1.7B student, the five-benchmark average rose from 36.8% with full OPD to 40.1%; the release rule also improved aggregate performance for 4B and 8B students and better preserved out-of-domain performance.A random-release control performed more weakly, supporting the use of trajectory-specific local-teachability signals.Source: §1 · p. 2

2 Related Work

Prior work addresses trajectory-level mismatch through data and alignment strategies, while OPD research tackles student-rollout supervision and its reliability. This paper positions local teachability collapse as a complementary temporal failure in strong-to-weak OPD with accessible teacher logits.

Off-policy distillation

  • Off-policy methods address trajectory-level mismatch through data selection or alignment, but reliable token-level teacher supervision remains open, motivating OPD when teacher logits are accessible [16] [30] [24].Teacher-trace SFT is sensitive to data provenance, distribution, and structure, can miss sequence-level distributions and suffer exposure bias [22] [36], while log-probability selection is confounded by response length [33].Source: §2 · pp. 2–3

On-policy distillation

  • Unlike Uni-OPD’s outcome-centric correction of sequence-level inconsistency [14], this work identifies local teachability collapse: dense supervision loses discriminative utility as strong-to-weak rollouts progress.Earlier OPD work addresses exposure bias, modifies objectives, studies sparse signals and tokenizer artifacts, or applies supervision only to prefixes [1] [38] [17] [12] [41].Source: §2 · p. 3
  • Multimodal OPD and self-distillation broaden teacher-signal sources [6] [20] [7] [9] [42] [28] [32] [11] [18], while this work focuses on white-box logits from a stronger external teacher.The motivating setting includes on-device modeling and local deployment, where the question is whether dense teacher feedback remains locally discriminative for a weak student’s rollout.Source: §2 · p. 3

3 Local Teachability Analysis

Teacher–student disagreement can persist late in a response even as token-level feedback loses discriminative value, motivating a trajectory-specific rule that stops dense OPD supervision after a detected teachability drop. The rule estimates teachability from teacher margins over the student’s reachable actions, aggregates them by sentence, and uses a downward BIC change point to select the supervised prefix.

Diagnosing local teachability collapse

  • For OPD-optimized rollouts, late mean advantage remains approximately At ≈−0.3 while normalized advantage dispersion declines, revealing local teachability collapse despite persistent disagreement.The analysis compares base-student and OPD-optimized rollouts across 500 training instances, partitioned into 20 normalized trajectory segments.Source: §3 · p. 4, Figure 1 · p. 4

Trajectory-specific release rule

  • The release signal measures the teacher’s nearest-competitor margin within the student’s top-K candidates, then aggregates it over NLTK-tokenized sentence segments.Unlike sampled-token advantage At, the margin tests whether the teacher can distinguish among actions available to the student at a given prefix; sentence aggregation reduces transient token-level artifacts.Source: §3 · p. 5
  • A profiled RSS-BIC comparison selects a single downward shift in sentence-level teachability; if that model does not improve on no change, the trajectory keeps full OPD supervision.The one-drop alternative requires a lower post-change mean than pre-change, and a release is accepted only when the best such model improves over the no-change model.Source: §3 · p. 6
  • After an accepted change point, the rule retains and rescales the prefix mask, changing where supervision applies without changing a trajectory’s total supervision weight.This focuses updates on the region where the teacher still has a clear preference among the student’s reachable actions and attenuates low-teachability suffixes.Source: §3 · p. 6
  • 24.0 mean profiled RSS-BIC gain accompanies 79% of samples accepting a significant downward release with gain greater than 6.Accepted release positions vary across trajectories, with a mean of 0.25 and median of 0.13, rather than forming a fixed prefix.Source: §3 · p. 6, Figure 3 · p. 7
  • The release rule is a practical heuristic, not a guarantee that every post-release token is detrimental.It aims to maintain standard OPD while sentence segments retain sufficient teacher contrast, then halt dense supervision after a substantial downward shift.Source: §3 · p. 7

4 Experiments

The experiments test whether dense OPD should stop when teacher feedback loses local teachability. The trajectory-specific release rule outperforms full-trajectory OPD without relying on a global prefix cutoff, and the reported checks support trajectory coupling and persistence at larger student scales.

4.1 Experimental Setup

  • The main setting uses Qwen3-1.7B-Non-Thinking as student and Qwen3-30B-A3B-Instruct-2507 as teacher, evaluated on five math benchmarks and GPQA.The method uses the teacher’s top-1/top-2 margin over the student’s top-K candidates, aggregates it across NLTK-tokenized sentence segments, and stops supervision after a downward profiled RSS-BIC change. This diagnostic requires full-rollout analysis and is not an efficiency claim comparable to FastOPD [41].Source: §4.1 · pp. 7–8

4.2 Main Results: Feedback Importance Varies Across Sequences

  • 40.1% AVG is reached by the trajectory-specific release rule, 3.3 points above full OPD and modestly above the best fixed-prefix row; GPQA moves in the same direction.Full OPD raises average math performance from 24.9% to 36.8%, while every fixed-prefix configuration also beats full OPD on AVG but requires a globally selected cutoff.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 Trajectory Coupling: More Than Retaining Fewer Tokens

  • Randomly permuting release points across samples lowers AVG from 40.1% to 38.0% and degrades GPQA, indicating that release points must remain coupled to each trajectory.Figure 3 shows sample-specific downward changes rather than a shared cutoff; Figure 7 offers a descriptive, four-checkpoint illustration that later diagnostic release positions generally align with stronger downstream performance.Source: §4.3 · p. 8

4.4 Sanity Checks: Support Size and Advantage Dispersion

  • Support-size results are close in aggregate, K = 4 gives the strongest AVG, and std(At) improves over full OPD but remains below the K = 4 top1–top2 margin.Advantage dispersion diagnoses a realized path, whereas the teacher margin measures state-local ranking; normalized teacher-margin curves also remain similar across support sizes.Source: §4.4 · pp. 8–9, Table 2 · p. 9

4.5 Scaling the Release Rule to Stronger Students

  • The average-level pattern persists with Qwen3-4B and Qwen3-8B students under the same teacher and DeepMath queries.FastOPD uses a 2048-token prefix in these experiments, selected for the best-balanced in-domain and out-of-domain performance in the main results.Source: Table 3 · p. 9, §4.5 · p. 9

5 Conclusion

In strong-to-weak on-policy distillation, a nonzero teacher–student advantage can coexist with declining local contrast, so dense supervision should fade when feedback loses local teachability. A trajectory-specific release rule halts supervision after a BIC-style drop in the teacher’s margin over the student’s reachable candidates and outperforms full-trajectory OPD across multiple settings.

5 Conclusion

  • The trajectory-specific release rule consistently outperforms standard full-trajectory OPD across multiple strong-to-weak distillation settings.It halts dense supervision following a BIC-style drop in the teacher’s margin over the student’s reachable candidates.Source: §5 · p. 9
  • Local teachability collapse occurs when a nonzero teacher–student advantage coexists with declining local contrast.The paper identifies this as a failure mode for dense supervision in strong-to-weak on-policy distillation.Source: §5 · p. 9
  • Effective dense OPD must assess whether the teacher’s advantage remains locally actionable, not merely whether an advantage exists.The proposed release rule makes supervision fade when the teacher’s feedback loses local teachability.Source: §5 · p. 9

A Technical Appendices and Supplementary Material · A.1 Release Rule Details

The release rule retains dense OPD supervision only over a trajectory prefix selected using local teacher–student discrimination. It combines student top-K candidates, sentence-level aggregation, and a BIC-style one-drop test, then rescales retained-prefix loss mass.

A.1 Release Rule Details

  • The rule uses the student’s local top-K support to define candidate actions at each response-token position.The algorithm ranks those candidates by teacher log-probability.Source: §A.1 · p. 13
  • For each token, it measures the teacher’s margin over the nearest competitor in the student’s top-K candidate set.The appendix describes this as a nearest-competitor teacher margin.Source: §A.1 · p. 13
  • Token-level advantages are aggregated across NLTK-tokenized sentence segments before selecting the release boundary.The algorithm iterates over sentence segments to calculate segment-level values.Source: §A.1 · p. 13
  • A profiled RSS-BIC one-drop test checks whether the segment values support a downward change point.The algorithm compares a no-change BIC with candidate one-drop fits and selects the lowest-BIC candidate when it improves on the baseline.Source: §A.1 · p. 13
  • After truncation, the retained-prefix advantages are rescaled to preserve loss mass.The appendix lists retained-prefix loss-mass rescaling as part of the experimental release rule.Source: §A.1 · p. 13
  • When a qualifying drop is found, dense supervision is kept for tokens before the boundary and removed afterward; otherwise, all valid response tokens are retained.The boundary follows the selected segment; if no candidate improves the baseline, the algorithm keeps the full valid-token mask.Source: §A.1 · p. 13

A.2 Theoretical Motivation for Local Teachability

The analysis motivates local teachability as action-dependent teacher contrast among the student’s plausible next tokens, rather than merely a nonzero teacher–student gap along a sampled trajectory. It presents the teacher-margin release rule and penalized downward-change criterion as useful operational signals, not guarantees of true change-point recovery or universal variance reduction.

A.2 Theoretical Motivation for Local Teachability

  • The analysis separates sampled-path advantage dispersion, which diagnoses lost token-specific variation, from action-wise variation within the student’s top-K candidates, which informs release.The release rule uses a state-local proxy at a fixed prefix rather than the trajectory-wise standard deviation of sampled-token advantages.Source: §A.2 · p. 13
  • Dense OPD is useful only when teacher feedback distinguishes among actions the student is likely to take; a nonzero sampled-token advantage alone does not ensure actionable supervision.When teacher preference over reachable alternatives becomes locally flat, the action-dependent component of the update shrinks and token-level feedback becomes less informative.Source: §A.2 · pp. 14–15
  • The nearest-competitor teacher margin within the student’s top-K set tests whether the teacher has a clear first choice, but it does not fully characterize local advantage variation.A sustained margin decrease can indicate that teacher feedback remains globally different from the student while becoming less locally actionable.Source: §A.2 · p. 14
  • Releasing a suffix can improve directional update SNR when that suffix contributes little aligned teacher signal relative to its variance.The rule is designed to detect this local-teachability-collapse regime through the teacher-margin curve.Source: §A.2 · p. 15
  • The BIC-style rule accepts a trajectory-specific release only when reduced residual error outweighs the one-drop model’s complexity penalty, avoiding releases based on small random fluctuations without guaranteeing true change-point recovery.The motivation is statistical discipline, not proof that BIC always identifies a true change point or that truncation is universally variance-reducing.Source: §A.2 · pp. 13, 15

A.3 Additional Diagnostic

The diagnostics show that teacher feedback can lose token-level discriminative value later in a rollout, even when feedback remains available. The decline appears robust across support sizes, and useful supervision boundaries depend on trajectory-level signals rather than a fixed token index alone.

A.3 Additional Diagnostic

  • As prefixes grow, teacher and student sampled-token log-probabilities align, advantage variability falls, and scalar feedback becomes less informative as a token-priority signal.This describes reduced local contrast, not necessarily a lack of teacher-student advantage.Source: §A.3 · p. 15
  • Across K ∈{2, 4, 8, 16, 32, 64}, normalized margin curves remain similar, while early segments have larger top1–top2 margins than late segments.This supports the interpretation that declining local teachability is not an artifact of the chosen support size.Source: Figure 5 · p. 16, §A.3 · p. 16
  • The BIC-style rule marks a lower-contrast post-change segment rather than proving suffix tokens useless; it is applied to all rollouts and falls back to full OPD when no downward change point is accepted.The fixed criterion uses no benchmark-specific cutoff or per-sample manual adjustment.Source: §A.3 · p. 16, Figure 6 · p. 16
  • Better fixed-prefix checkpoints tended to have later diagnostic release positions, suggesting useful cutoffs retain locally discriminative regions; with only four checkpoints, this is illustrative rather than a statistical correlation.Checkpoint performance was averaged across five downstream math benchmarks; a global prefix length alone does not explain the useful supervision boundary.Source: §A.3 · p. 16

A.4 Release-Point Case Studies

This section presents case studies of release-point behavior and an accompanying probability analysis. The cases address preserving a response’s core setup while excluding repetitive or noisy continuations from dense OPD supervision, and the analysis considers the probability that a sum of i.i.d. Exp(1) variables is at least its mean.

Case studies

  • The examples use release points to exclude repetitive or noisy terminal continuations that may receive counterproductive dense supervision, without truncating responses during evaluation.The primary reasoning or setup precedes the release point, though the highlighted regions are not guaranteed to be error-free.Source: §A.4 · p. 17
  • Figure 7 shows diagnostic release positions and fixed-prefix performance across four checkpoints, which the caption characterizes as descriptive rather than evidence of correlation.Source: Figure 7 · p. 17
  • In the limit example, the pre-release prefix gives the correct case split and exponential-decay observation, while the continuation repeats the argument and includes false starts.The case asks for the limit of n^3 x^n (1 − x)^4 over x ∈ [0, 1].Source: §A.4 · p. 17
  • After a correct reduction toward a central-limit-theorem solution, one continuation repeatedly debates Gamma symmetry and introduces erroneous intermediate statements before returning to the same answer.The released continuation illustrates noisy self-correction after the main solution route is already established.Source: §A.4 · p. 19

A.4 Release-Point Case Studies

  • T_n is the sum of n i.i.d. Exp(1) variables and has a Gamma(n,1) distribution.The passages also state that this distribution has mean n and variance n.Source: §A.4 · pp. 22, 24
  • The Gamma(n,1) distribution is not symmetric around n, so symmetry alone does not establish that P(T_n ≥ n) equals 1/2 for finite n.The passage contrasts the unequal densities at n − x and n + x.Source: §A.4 · p. 24

A.5 Hyperparameters

The default strong-to-weak OPD configuration follows prior work [38] [31], with an adjusted maximum inference output length to avoid truncated mathematical responses and biased evaluations.

A.5 Hyperparameters

  • The maximum inference output length was adjusted because the standard 16k-token limit truncated mathematical responses and biased evaluations.Table 4 presents the default training hyperparameters, but the supplied passages do not specify the adjusted length.Source: §A.5 · p. 25, Table 4 · p. 25

A.6 Evaluation

Evaluation uses benchmark-specific sampling budgets because of resource constraints, with larger per-problem sample counts for relatively fewer and harder benchmarks and calibrated counts for the rest.

A.6 Evaluation

  • Sampling budgets vary by benchmark because of resource constraints, with several hundred total sampled responses per benchmark used as an appropriate scale for most benchmarks.The authors base this scale on empirical experience.Source: §A.6 · p. 25
  • AIME25, HMMT25-Feb, and HMMT25-Nov receive 32 samples per problem because they contain relatively fewer and harder problems.Source: §A.6 · p. 25
  • The evaluation allocates 1 sample per MATH500 problem and 4 samples per Minerva and GPQA problem.Source: §A.6 · p. 25

A.7 Runtime Overhead … A.11 Reference Python Implementation

The supplementary sections report improved average code-domain performance over full OPD and the fixed-prefix baseline, describe runtime and scope limitations, and specify the dynamic-prefix algorithm.

A.7 Runtime Overhead

  • 14.28 hours for 100 steps were required by full-trajectory OPD, versus 13.97 hours for the release variant on eight H800 GPUs.Full OPD’s response length rose sharply in the initial 40 steps, while the release method’s output length increased more steadily.Source: §A.7 · p. 25, Figure 8 · p. 25

A.8 Code-Domain Results

  • The trajectory-specific release rule improves upon full OPD and slightly outperforms the fixed-prefix baseline in average performance across LiveCodeBench v6, MBPP, and HumanEval.The evaluation uses queries sampled from Eurus-RL-Code.Source: §A.8 · p. 26, Table 5 · p. 26

A.9 Future Work

  • Future work will extend the release rule beyond mathematical reasoning and code checks to broader instruction-following and mixed-domain settings..Source: §A.9 · p. 26
  • Experiments primarily cover the Qwen3 family, and validating the findings across broader model families remains future work.The paper identifies establishing local teachability collapse as its core contribution, while noting computational and resource constraints on broader evaluation.Source: §A.9 · p. 26
  • An online rollout-time detector could stop collecting low-value suffix supervision early, improving efficiency over post-hoc loss masking after full rollouts.This is proposed as a future direction.Source: §A.9 · p. 26

A.11 Reference Python Implementation

  • After truncating to the selected prefix, the algorithm rescales retained advantages to preserve per-sample loss mass.The implementation applies a prefix mask and scales by total loss-mask mass divided by retained loss-mask mass.Source: §A.11 · pp. 26, 28, 29
  • The reference algorithm computes teacher margins over the student’s top-K candidates, aggregates them by NLTK-style sentence segments, and scores segments using log1p of mean margin.The margin is the log-probability difference between the teacher’s two highest-probability candidates within the student’s top-K support.Source: §A.11 · p. 27
  • Profiled RSS-BIC selects a downward change point only when the one-drop model improves over the no-change model; otherwise, the rule keeps the full trajectory.The implementation compares a one-mean model with a two-mean model plus change point.Source: §A.11 · pp. 27–28

Related papers

Loading 2605.13643v2…