PaperlayerView exact revision on arXiv

Source-linked AI summary

Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation

Xincheng Wei, Yifan Ding, Yoshua Li, Yuquan Lu, Ziheng Li, Yi Lu, Dongsheng Ma, Rongxiang Weng, Xunliang Cai

arXiv:2609.39687v1cs.CLcs.LG

TL;DR

Standard OPSD relies on one fixed privileged-teacher setting, despite potentially useful corrections from nearby settings. Neighborhood OPSD selects complementary experts and routes their distributions at student states. Across three benchmarks, it improves Average@12 over OPSD on three model sizes.

  • Problem

    Standard OPSD uses the same unperturbed privileged teacher at every student-visited state, so useful supervision from nearby parameter settings may remain unused.

    Source: §1 · p. 1

  • Method

    Neighborhood OPSD greedily selects experts for filtered reference-prefix corrections beyond the pool’s current best, then routes a matching-token expert’s full distribution at each student state.

    Source: §1 · p. 2, §2.2 · p. 4

  • Results

    Average@12 improves over OPSD by 2.75, 1.67, and 1.94 points on Qwen3-1.7B, 4B, and 8B, respectively, across AIME 2024, AIME 2025, and HMMT February 2025.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Student-prefix continuations support using the selected pool beyond reference trajectories, and matched ablations support complementary selection and state-wise routing.

    Source: §1 · p. 2

    The selected pool remains fixed during training, although experts useful to the changing student may differ; periodically reselecting experts would add evaluation cost.

    Source: F Limitations and Future Work · p. 20

Abstract

from arXiv · show

On-policy self-distillation (OPSD) trains mathematical reasoning models using a privileged teacher that sees a reference solution and supervises student-sampled prefixes. Standard OPSD uses one fixed parameter setting at every state, but nearby settings may offer additional supervision. We find that local parameter perturbations reveal complementary reference-aligned corrections under the same reference context. Different experts supply these corrections at different reference positions. Their pool covers more such positions than the unperturbed privileged teacher. We introduce Neighborhood OPSD (N-OPSD) to turn these corrections into supervision at student-visited states. Offline, greedy selection builds a compact pool of frozen experts by rewarding filtered reference-token gains beyond the pool's current best at each position. The highest-peak expert need not provide the best training target. Online routing therefore separates the anchor direction from its level of support. MaxPeak selects the anchor token, and quantile selection chooses among experts whose top token matches it. The student learns from the chosen expert's full next-token distribution through the clipped forward-KL objective inherited from OPSD. We evaluate on AIME 2024, AIME 2025, and HMMT February 2025. Across three independent runs per method, Neighborhood OPSD improves the three-benchmark Average@12 over OPSD by 2.75, 1.67, and 1.94 points on Qwen3-1.7B, 4B, and 8B, respectively. Student-prefix continuations support using the pool beyond the reference trajectories used for selection. Matched ablations support filtered reference-token gains as a selection criterion. Accounting for overlap within the pool and routing by state further improve student accuracy. Inference uses only the distilled student.

1 Introduction

Standard OPSD uses one fixed privileged-teacher setting, although nearby settings can provide complementary reference-aligned corrections. Neighborhood OPSD selects and routes these experts as state-wise supervision, improving Average@12 over OPSD across three reasoning benchmarks.

1 Introduction

  • Standard OPSD uses the same unperturbed privileged teacher at every student-visited state, leaving useful supervision from nearby settings unexplored.The teacher sees the reference solution during training, while the student generates from the problem alone.Source: §1 · p. 1
  • Different local experts raise reference-token probability at different positions, and their pool covers more positions than the unperturbed teacher under the same filters.The corrections are measured against the frozen problem-only model on the same reference prefix.Source: §1 · p. 2
  • Neighborhood OPSD builds a compact pool of complementary experts offline, then routes an expert’s full next-token distribution at student-visited states.Because student prefixes can differ from reference prefixes, experts evaluate the student’s current prefix.Source: §1 · p. 2, Figure 2 · p. 3
  • MaxPeak chooses an anchor token, while quantile routing selects support from experts with that same top token; the chosen expert supplies the full distribution.This separates anchor direction from support level.Source: §1 · p. 2
  • Average@12 improves over OPSD by 2.75, 1.67, and 1.94 points on Qwen3-1.7B, 4B, and 8B, respectively, across three independent runs per method.Evaluation uses AIME 2024, AIME 2025, and HMMT February 2025.Source: §1 · p. 2

2 Method

Neighborhood OPSD selects complementary frozen experts using reference-prefix corrections, then routes their full next-token distributions at student states. Its training retains OPSD’s clipped forward-KL objective.

2 Method

  • Offline selection builds a complementary expert pool, while online routing chooses a supervision source at the student’s current prefix.The useful source can change as the rollout develops.Source: §2 · p. 3

2.1 Setup and Perturbation Experts

  • The student samples responses from the problem-only prompt, while the privileged teacher sees the reference solution and the same student prefix.Perturbation experts share the base checkpoint and privileged context, remain fixed during training, and are selected offline.Source: §2.1 · pp. 3–4

2.2 Complementary Expert Selection

  • Greedy selection rewards each expert for filtered reference-token corrections beyond the pool’s current best at each reference position.The score rewards newly covered positions and stronger corrections at already covered positions; student-confidence and selection clips filter candidate corrections.Source: §2.2 · p. 4

2.3 MaxPeak-Anchored Quantile Routing

  • MaxPeak chooses the token with the largest probability assigned by any expert, then quantile selection chooses support among experts sharing that top token.The selected expert supplies the full-vocabulary target; its level of support matters because large positive gaps can cause clipping.Source: §2.3 · pp. 4–5

2.4 Training Objective

  • Training applies the inherited OPSD pointwise clip to vocabulary-level forward-KL contributions at SCGate-retained positions and updates only the student.The fixed pool’s predictions and routed targets are recomputed on rollouts sampled from the current student.Source: §2.1 · p. 3, §2.4 · p. 5

3 Experiments

Experiments compare Neighborhood OPSD with OPSD across three Qwen3 scales and test expert selection, routing, pool size, cost, and distillation choices. Results support complementary expert selection and state-wise routing, with student-prefix continuations testing transfer beyond reference trajectories.

3.1 Experimental Setup

  • Experiments train Qwen3-1.7B, 4B, and 8B for 100 steps on 10,000 OpenThoughts problems and evaluate Average@12 on AIME 2024, AIME 2025, and HMMT February 2025.Average@12 uses 12 sampled responses per problem; the reported checkpoint is step 100.Source: §3.1 · p. 5

3.2 Main Results Across Model Scales

  • Neighborhood OPSD ranks first across all nine model-benchmark settings and exceeds OPSD’s three-benchmark Average@12 by 2.75, 1.67, and 1.94 points at 1.7B, 4B, and 8B.These results use three independent runs per method.Source: Table 1 · p. 6, §3.2 · p. 6

3.3 Complementary Expert Selection

  • Greedy marginal-gain PTA selection reaches 66.57 Average@12, beating independent PTA Top-K by 0.92 points and Quality Top-K by 1.57.Both PTA selectors use the same filtered per-token values; greedy sample coverage scores 65.37, 1.20 points below the proposed selector.Source: §3.3 · p. 6
  • At σ = .002, the selected pool covers 26.53% of held-out reference positions, versus 15.69% for M0, while SCGate and the selection clip further filter supervision.With M0, SCGate raises the three-run average from 64.63 to 64.91; Neighborhood OPSD reaches 66.57. Removing both SCGate stages gives 65.28, and reselection without the reference-side clip gives 64.26.Source: §3.3 · p. 7

3.4 Routing on Student Prefixes

  • On 300 student prefixes, Raw MaxPeak raises continuation accuracy from 83.33% to 86.67% and completion from 90.00% to 94.67% versus M0.Both use the same prefixes, privileged references, and 4096-token budget; continuation accuracy and downstream Average@12 peak at σ = .002, although reference coverage and anchor accuracy keep rising at larger radii.Source: §3.4 · p. 7, Table 3 · p. 7
  • At q = .75, MaxPeak-anchored quantile routing reaches 66.57 Average@12, beating Uniform averaging by 1.94 points, Token consensus by 1.57, and Random routing by 1.66.With the pool and anchor fixed, accuracy rises from q = 0 to .75 but drops 1.29 points at q = 1, showing the highest-peak expert need not supply the best target.Source: §3.4 · p. 8, Figure 3 · p. 8

3.5 Pool Size and Training Cost

  • K = 25 gives the highest tested three-benchmark average: 66.57 versus 64.54 at K = 1 and 64.35 at K = 50; it also reduces estimated step time by 25.63%.K = 25 retains 94.81% of K = 50’s selection-feasible coverage; its estimated step time is 1.423× OPSD+SCGate.Source: §3.5 · p. 8

3.6 Robustness to Distillation Choices

  • With the pool and router fixed, Neighborhood OPSD outperforms OPSD+SCGate under JSD and RKL, while clipped forward-KL has a three-run mean of 66.57.It retains an average advantage with Top-1-tail and Top-100-tail targets, including a 2.12-point gain over OPSD+SCGate for the binary Top-1-tail target; the full-vocabulary target scores 1.30 and 1.20 points higher than those compressed variants.Source: §3.6 · pp. 8–9, Figure 3 · p. 8

4 Related Work

Related work situates Neighborhood OPSD among on-policy privileged self-distillation, methods that adapt fixed-teacher supervision, and approaches using multiple teachers or perturbation experts. It keeps shared reference context while varying teacher parameters and selecting complementary local experts.

4.1 On-Policy and Privileged Self-Distillation

  • On-policy distillation trains on student-generated prefixes, while privileged self-distillation gives the teacher demonstrations or other context unavailable to the student; Neighborhood OPSD varies teacher parameters under shared reference context.Prior work also studies choosing privileged context per trajectory.Source: §4.1 · p. 9

4.2 Adapting Supervision from a Fixed Teacher

  • Fixed-teacher methods adapt targets, divergences, or loss weights, whereas Neighborhood OPSD expands the supervision source through complementary local experts.Its routed targets can also be used with divergence adaptation or loss weighting.Source: §4.2 · p. 9

4.3 Multiple Supervision Sources and Perturbation Experts

  • Unlike multi-teacher methods that select by instance, task, or specialist weighting, Neighborhood OPSD selects by marginal filtered PTA and routes one expert’s full distribution at each student state.Neural Thickets constructs local perturbation experts; RandOpt selects them by task performance and uses majority voting.Source: §4.3 · p. 9

5 Conclusion

The conclusion presents Neighborhood OPSD as a way to turn complementary reference-aligned corrections from a privileged teacher’s parameter neighborhood into state-wise supervision for one student. It reports improved three-benchmark Average@12 over OPSD across three Qwen3 scales.

5 Conclusion

  • Neighborhood OPSD improves three-benchmark Average@12 over OPSD by 2.75, 1.67, and 1.94 points on Qwen3-1.7B, 4B, and 8B.Inference uses only the distilled student.Source: §5 · p. 9
  • The selected pool achieves higher reference-side selection-feasible coverage than M0, and student-prefix continuations support using it beyond reference trajectories.Matched ablations support the roles of complementary expert selection and state-wise routing in downstream gains.Source: §5 · p. 9
  • The highest-peak expert need not provide the best training target, supporting separate choices of anchor direction and support level.Source: §5 · p. 9

A Method Details

The method filters reference-side selection gains, greedily builds a compact expert pool, and routes student states using expert predictions on the current prefix. An alternative token-consensus router instead averages full distributions from experts backing the consensus token.

A.1 Reference-Side Selection Clip

  • The reference-side selection clip gives no credit to candidate-position pairs whose reference-token contribution exceeds κsel, filtering sharp peaks the online clip may suppress.It is a selection proxy on a fixed reference prefix; passing it does not determine whether an online contribution will be clipped.Source: §A.1 · p. 11

A.2 Greedy Marginal Gain

  • Greedy selection repeatedly adds the remaining expert with the largest marginal gain, then retains the first K experts in the resulting insertion order.The coverage analysis motivates K = 25.Source: §A.2 · p. 11

A.3 MaxPeak Equivalence

  • MaxPeak selects the highest-peak expert and then its top token, maximizing the token’s support across experts on the current student prefix.Offline selection instead uses filtered reference-token PTA.Source: §A.3 · p. 12

B.1 Expert Construction and Selection

  • For Qwen3-8B, selection uses 501 candidates, a 1,000-question selection set, and the first 25 experts in greedy order; downstream benchmarks and the analysis set are excluded.The calibrated radius is σ = .002 for Qwen3-8B.Source: §B.1 · p. 12

B.2 Selector Definitions

  • The selector definitions contrast independent expert ranking by answer accuracy or filtered PTA with greedy selection that recomputes marginal gains as the pool grows.Greedy sample coverage adds experts for newly answered questions, while the PTA objective accounts for gains beyond the current pool.Source: §B.2 · pp. 12–13

B.3 Token Consensus Routing

  • Token consensus selects a top-1 token supported by the pool, gathers experts backing it, and averages their full vocabulary distributions with equal weights.A consensus token need not receive more than half of the votes; ties use the largest expert probability and then fixed expert order.Source: §B.3 · p. 13

B.4 Training Configuration

Training updates only the student while keeping experts fixed, using online SCGate and clipped forward KL. Baseline implementations use the frozen reference-conditioned teacher M0, with EOPD and PW-OPSD changing the objective or loss weighting.

B.4 Training Configuration

  • NEIGHBORHOOD OPSD uses full-parameter student optimization, forward KL, a full-vocabulary loss, and fixed experts; only the student is updated.Defaults include K = 25, q = .75, online SCGate threshold τ = .99, and κsel = κ = .06.Source: §B.4 · p. 13
  • Streaming expert passes avoid materializing all K distributions at once, reducing storage from O(KBS|V|) to O(KBS) routing metadata on CPU.Inference uses only the distilled student.Source: §B.4 · p. 13
  • Rollouts come from the problem-only student, while the privileged teacher evaluates the same completion with reference context; OPSD+SCGate applies the online gate to fixed M0 without offline selection.The SCGate ablation removes both gates and reselects the pool; the selection-clip ablation removes only the reference-side clip while retaining online OPSD clipping.Source: §B.4 · p. 13, §B.5 · p. 14

B.5 Baseline Implementations

  • EOPD tests entropy-aware distillation with the reference-conditioned M0 teacher, while PW-OPSD applies position-dependent loss weighting under the same rollout protocol.Both are baseline implementations using the privileged teacher.Source: §B.5 · p. 14

B.6 Evaluation and Statistics

Evaluation reports step-100 results and uses three independent runs for NEIGHBORHOOD OPSD, OPSD, EOPD, and PW-OPSD at each scale, with means and sample standard deviations.

B.6 Evaluation and Statistics

  • Three independent runs are reported for NEIGHBORHOOD OPSD, OPSD, EOPD, and PW-OPSD at each scale, using the step-100 checkpoint and mean ± sample standard deviation.Base, SFT, and GRPO results come from Zhao et al. (2026); other baselines are reruns, and remaining analysis variants are point estimates unless noted otherwise.Source: §B.6 · p. 14

C Complete Ablation Results

The ablations compare expert selection, routing, target distributions, and training settings, with additional analyses of perturbation radius, SCGate, and pool size. Reported results favor greedy marginal PTA selection, the tested σ = .002 setting, and K = 25 under the evaluated conditions.

C.1 Expert Selection

  • The expert-selection and routing tables define comparisons at σ = .002 and K = 25 on Qwen3-8B, while the selected-pool ablation fixes MaxPeak-anchored quantile routing and online SCGate.The supplied passages establish these comparison settings but do not state the table-cell outcomes.Source: §C.1 · p. 14, Table 4 · p. 14, §C.2 · p. 14

C.3 Divergence and Target Distribution

  • The target-distribution ablation compares full-vocabulary forward KL with JSD, RKL, and Top-N-tail variants, reporting Average@12 for Qwen3-8B as single runs.Top-N-tail compresses the top N tokens individually and combines the remaining probability mass into one category; clipping is applied after compression.Source: §C.3 · p. 15

C.4 SCGate and the Reference-Side Selection Clip

  • Removing the reference-side selection clip reduces Average@12 by 2.31 points, while removing both SCGate stages reduces it by approximately 1.3 points.The fixed-teacher SCGate comparison improves the average by 0.28 points.Source: §C.4 · p. 16

D.1 Perturbation Radius

  • The three-benchmark average reaches 66.57 at σ = .002, versus 65.83, 64.54, and 63.24 at σ = .001, .004, and .006, respectively.The .002 setting is also best on each benchmark, and both the continuation probe and downstream evaluation favor it among tested radii.Source: §D.1 · p. 16

D.2 SCGate Calibration and Sensitivity

  • At τ = .99, 65.63% of reference positions fall in the high-confidence group, where the problem-only model has 97.67% top-1 accuracy.This confidence audit motivates focusing supervision on lower-confidence positions.Source: §D.2 · p. 17

D.3 Expert Count

  • Greedy PTA selection reaches 66.57 Average@12, while the best tested K = 50 setting reaches 65.83 and has higher estimated step time.At K = 25, the pool retains 94.81% of K = 50 selection-feasible coverage; its estimated step time is 1.423× OPSD+SCGate and 25.63% below K = 50.Source: §D.3 · p. 17
  • At K = 50, q = .50 improves Average@12 by 1.48 points over q = .75, whereas q = .75 is the best tested quantile at K = 25.The best tested quantile therefore differs between the two pool sizes.Source: §D.3 · p. 17
  • The marginal share of selection-feasible coverage falls from 63.47% for the first expert to .39% at rank 25 and .15% at rank 50.Most available selection-feasible coverage is present by K = 25.Source: §D.3 · p. 17

D.4 Training-Step Timing

Training-step estimates combine rollout, student computation, and teacher costs; the reference-side audit finds higher pool coverage and MaxPeak accuracy than M0 at every tested radius.

D.4 Training-Step Timing

  • Teacher time is measured on cached student trajectories; Neighborhood OPSD sums two passes, first for routing metadata and then for target distributions.The unperturbed M0 teacher takes 2.0184 seconds for one forward pass.Source: §D.4 · p. 18
  • Estimated OPSD+SCGate step time is 162.6764 seconds; Neighborhood OPSD step time is its teacher time plus 160.670 seconds.The estimate uses the same rollout and student-computation times for both settings, and adds 0.012 seconds for routing for Neighborhood OPSD.Source: §D.4 · p. 18, Table 12 · p. 18, Table 13 · p. 18

E.1 Full Reference-Side Audit

  • At every tested radius, the selected pool has higher selection-feasible coverage than M0, and MaxPeak has higher anchor accuracy.Larger radii also increase JS divergence and MaxPeak peak probability.Source: §E.1 · p. 18

E.2 Complete Raw MaxPeak Radius Probe

The radius probe evaluates Raw MaxPeak on 300 student prefixes against M0, and finds its best continuation results at σ = .002; newly covered reference tokens include ordinary mathematical-solution components.

E.2 Complete Raw MaxPeak Radius Probe

  • Raw MaxPeak and M0 are compared on the same 300 student prefixes, with the same privileged references and up to 4096 new tokens.The prefixes comprise 150 completed wrong responses and 150 unfinished responses; unfinished generations remain in the denominator.Source: §E.2 · pp. 18–19
  • At σ = .002, Raw MaxPeak raises continuation accuracy by 3.33 points and completion by 4.67 points versus M0.At larger radii, the FKL proxy rises and continuation quality falls despite continued gains in reference-side anchor accuracy.Source: §E.2 · p. 19, Table 15 · p. 19

E.3 Frequent Newly Covered Reference Tokens

  • Newly covered reference tokens at σ = .002 include function words, LaTeX delimiters, numerals, punctuation, and reasoning verbs, not only self-reflective markers.The listed tokens are described as standard lexical and symbolic components of mathematical solutions.Source: §E.3 · p. 19, Table 16 · p. 20

F Limitations and Future Work

The experiments cover mathematical reasoning with the Qwen3 model family, and the fixed expert pool may become mismatched as the student changes.

F Limitations and Future Work

  • Experiments cover mathematical reasoning tasks and the Qwen3 model family; other model families and tasks remain outside the evaluated scope.The paper names code generation, multi-hop question answering, and tool use as settings for future evaluation.Source: F Limitations and Future Work · p. 20
  • Offline expert selection uses filtered PTA relative to the fixed problem-only distribution on reference prefixes, while the selected pool stays fixed during training.As the student changes, experts most useful to it may differ from those selected offline.Source: F Limitations and Future Work · p. 20
  • Periodic PTA recomputation and expert reselection could reduce pool mismatch, but repeated evaluation of all candidates would increase computational cost.The paper leaves efficient pool updates that balance cost and student-performance gains to future work.Source: F Limitations and Future Work · p. 20

Related papers

Loading 2609.39687v1…