PaperlayerView exact revision on arXiv

Source-linked AI summary

Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation

Xiang Chen, Futao Su, Kong Wang, Jiayi Chen, TanLin Li

arXiv:2610.02678v1cs.AI

TL;DR

On-policy distillation can require substantial teacher computation when every student-generated rollout receives teacher supervision. SR-OPD routes prompts and failed rollouts before teacher inference, using same-prompt student successes as references. Across six mathematical reasoning benchmarks, it uses 3.46–5.02% of Vanilla OPD’s teacher-input tokens while maintaining comparable reasoning performance.

  • Problem

    Querying the teacher for every rollout is costly, and uniform supervision is inefficient when its value varies across prompts, rollouts, and token positions.

    Source: §1 · p. 1

  • Method

    SR-OPD routes prompts with both successful and failed rollouts, then selects a failed rollout using its persistent departure from a same-prompt success and estimated teacher-input cost.

    Source: §1 · p. 2

  • Results

    Across three teacher–student pairs and six mathematical reasoning benchmarks, SR-OPD uses 3.46–5.02% of Vanilla OPD’s teacher-input tokens while maintaining comparable reasoning performance.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Controlled experiments under a matched teacher-input budget support both prompt routing and success-referenced rollout selection.

    Source: §6 · p. 9

    SR-OPD requires reliable verifier outcomes and mixed-outcome prompts; evaluation covers mathematical reasoning, three teacher–student pairs, and a limited range of rollout counts.

    Source: §5 · p. 9

Abstract

from arXiv · show

On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher, but providing such supervision for every rollout requires substantial teacher computation. We introduce Success-Referenced On-Policy Distillation (SR-OPD), which reduces this cost by selecting which prompts and rollouts receive teacher supervision. When the student produces both successful and failed rollouts for the same prompt, a successful rollout can serve as a natural reference for selecting failed rollouts. SR-OPD therefore focuses on such prompts and prioritizes failed rollouts whose hidden-state trajectories show sustained divergence from a successful reference, while accounting for estimated teacher-input cost. Across three teacher-student pairs and six mathematical reasoning benchmarks, SR-OPD uses only 3.46-5.02% of the teacher-input tokens required by Vanilla OPD in the one-pass setting while maintaining comparable reasoning performance. Under a controlled setting matched to 5% of Vanilla OPD's teacher-input budget, further experiments support both key design choices: focusing supervision on prompts with both successful and failed rollouts, and using successful rollouts to guide failure selection. These results indicate that a student's own successful behavior can serve as a useful reference for allocating teacher supervision under a fixed teacher-input budget.

1 INTRODUCTION

SR-OPD routes teacher supervision before teacher inference, using the student’s same-prompt successful rollouts as references for selecting failures. It addresses the cost of uniformly querying teachers for every rollout.

1 INTRODUCTION

  • Uniformly querying every rollout is costly because the value of teacher supervision varies across prompts, rollouts, and token positions.Prior token-level selection does not avoid the teacher-input cost of processing an entire queried rollout, while pre-query routing for standard teacher–student OPD remains underexplored.Source: §1 · p. 1
  • SR-OPD makes prompt- and rollout-level supervision decisions before teacher inference while leaving the underlying OPD procedure unchanged.It selects which prompts and failed rollouts receive teacher supervision.Source: §1 · pp. 1–2
  • Prompt routing retains prompts with both successful and failed rollouts, skipping all-success prompts without a failure target and all-failure prompts without a successful reference.The same-prompt success is used as a teacher-free reference.Source: §1 · p. 2, Figure 1 · p. 2
  • Rollout routing uses persistent-departure area (PDA) and estimated teacher-input cost to prioritize failed rollouts for teacher queries.The successful rollout serves only as a student-side reference and is not included in the teacher context.Source: §1 · p. 2
  • Across three teacher–student pairs and six mathematical reasoning benchmarks, SR-OPD uses 3.46–5.02% of Vanilla OPD’s teacher-input tokens while maintaining comparable reasoning performance.Under a matched 5% teacher-input budget, controlled experiments support both prompt routing and success-referenced rollout selection.Source: §1 · p. 2

2 RELATED WORK

Prior work explores teacher-guided supervision allocation, same-prompt outcome signals, and hidden-state signals for selecting failures. This section motivates using student-side information available before teacher inference to distinguish among failed rollouts.

Supervision signals in on-policy distillation

  • TA-OPD studies token-level signal teachability, while PG-OPD uses teacher–student compatibility over short prefixes to estimate rollout value.These approaches indicate that supervision allocation can use teacher or teacher–student information, which may itself require teacher computation.Source: §2 · p. 3

Same-prompt outcome signals

  • ReNIO reports stronger learning signals from incorrect than correct-only rollouts, and MOPD finds complementary information in successful and failed same-prompt rollouts.Outcome labels remain coarse because failed rollouts sharing a prompt can have substantially different reasoning processes.Source: §2 · p. 3

Hidden-state signals for failure selection

  • Prior studies use hidden states to identify representational differences between correct and incorrect reasoning for verification, candidate selection, or fine-grained credit assignment.These findings motivate representation geometry as a teacher-free signal for distinguishing failures beyond outcome labels alone.Source: §2 · p. 3

3 METHOD

SR-OPD uses student-side information to route teacher supervision in two stages before teacher inference. It retains prompts with successful and failed rollouts, then selects one failed rollout using its persistent representational departure from successful behavior and estimated teacher-input cost.

3 METHOD

  • For each prompt, SR-OPD first routes prompts and then compares failed rollouts with same-prompt successful behavior to select one for unchanged standard OPD supervision.Both routing decisions use student-side information and occur before teacher inference.Source: §3 · p. 3

3.1 ROUTING SETUP

  • For each training prompt, the student generates rollouts that a task verifier classifies as successful, failed, or invalid or unevaluable.Invalid or unevaluable rollouts are discarded; successful rollouts form the reference set and failed rollouts form the teacher-query candidate set.Source: §3.1 · pp. 3–4

3.2 PROMPT ROUTING

  • Prompt routing retains a prompt only when it has at least one successful and one failed rollout.All-success prompts lack a failed candidate, and all-failure prompts lack a same-prompt successful reference; skipped prompts are not assumed to have no learning value.Source: §3.2 · p. 4

3.3 SUCCESS-REFERENCED ROLLOUT ROUTING

  • For retained prompts, successful rollouts provide student-side references because the shared binary failure label cannot distinguish among failed candidates.The selector quantifies persistent representational departure and combines it with estimated teacher-input cost.Source: §3.3 · p. 4
  • Hidden-state trajectories are approximated at M positions on a normalized progress axis to align different rollout lengths while bounding routing computation.Mapped positions use relative progress and each rollout’s response-token length.Source: §3.3 · p. 4
  • PDA measures sustained representational departure from same-prompt successful behavior, with larger values for larger or more sustained departures.The score does not localize semantic errors or directly estimate a rollout’s true learning utility.Source: §3.3 · p. 5
  • The cost-aware score favors stronger PDA while discounting failed rollouts requiring more teacher-input tokens.Because the normalization is constant within a prompt, the score is a within-prompt supervision-efficiency measure, not a global budget-allocation objective.Source: §3.3 · p. 5

4 EXPERIMENTS

Across three teacher–student pairs, SR-OPD uses substantially fewer teacher-input tokens than Vanilla OPD while maintaining similar reasoning performance. Matched-budget studies isolate prompt and rollout routing, with component ablations and teacher-side diagnostics providing complementary mechanism evidence.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses three teacher–student pairs, six mathematical reasoning benchmarks, four rollouts per prompt, and three paired training seeds.Macro Avg is the unweighted mean over the six benchmarks.Source: §4.1 · p. 6

4.2 END-TO-END PERFORMANCE–COST TRADE-OFF

  • SR-OPD uses 3.64%, 3.46%, and 5.02% of Vanilla OPD’s teacher-input tokens on Qwen, Skywork, and Granite, respectively, while maintaining similar reasoning performance.Paired Macro Avg differences are +0.063, −0.089, and −0.075 points; all three 95% intervals include zero.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 CONTROLLED ROUTING STUDIES

  • At a matched 5% teacher-input budget, Mixed+Random exceeds Global+Random by +0.410, +0.467, and +0.338 Macro Avg points, with all three intervals above zero.Both arms use Random rollout selection; Figure 2(a) shows absolute Macro Avg means.Source: §4.3 · p. 7, Figure 2 · p. 8
  • With the mixed-outcome gate fixed, SR-OPD gains +0.616, +0.738, and +0.587 points over Mixed+Random on Qwen, Skywork, and Granite, respectively.Qwen and Skywork intervals exclude zero; all SR-OPD-minus-Static SR intervals include zero, so evidence is stronger versus Mixed+Random than for PDA’s incremental effect over Static SR.Source: §4.3 · p. 7, Figure 2 · p. 8

4.4 MECHANISM ANALYSIS

  • Component comparisons favor PDA over maximum departure and the full cost-aware score over its no-cost variant, but all twelve component means have intervals that include zero.The paired differences are +0.249, +0.287, and +0.192 points for PDA, and +0.191, +0.228, and +0.123 points for the full score; the authors treat these as directional evidence.Source: §4.4 · p. 7, Figure 2 · p. 8
  • Full SR score correlations with local update magnitude are 0.412, 0.271, and 0.258 on Qwen, Skywork, and Granite, respectively, indicating agreement with local post-query signals but not long-horizon utility.Figure 2(d) compares the pre-query score with three post-query teacher-side diagnostics.Source: §4.4 · p. 8, Figure 2 · p. 8

5 DISCUSSION AND LIMITATIONS

The discussion frames teacher allocation as two decisions: which prompts merit a query and which failure within a selected prompt receives supervision. The approach has boundaries in verifier outcomes, evaluation scope, and what its teacher-input savings establish.

Discussion

  • Teacher-supervision allocation separates prompt routing from choosing which failed rollout within a selected prompt receives supervision.The discussion treats stage-level experiments as the clearest evidence; component ablations and teacher-side diagnostics are complementary mechanism evidence.Source: §5 · p. 8
  • The study focuses on pre-query rollout selection; joint optimization with post-query token selection is proposed as future work to improve the performance–efficiency trade-off.The paper also proposes examining success-referenced selection in multi-rollout training such as GRPO.Source: §5 · p. 8

Limitations

  • Teacher-input-token savings do not establish equivalent reductions in total compute or wall-clock time, and matched-budget studies do not match prompt exposure or optimizer updates.With three paired training seeds, component effects remain directional rather than independently resolved.Source: §5 · p. 9
  • SR-OPD requires reliable verifier outcomes and mixed-outcome prompts; evaluation is limited to mathematical reasoning, three teacher–student pairs, and a limited range of rollout counts.Generalization remains open.Source: §5 · p. 9

6 CONCLUSION

SR-OPD routes teacher supervision before inference, using the student's successful behavior as a reference while leaving the OPD objective unchanged. The conclusion reports comparable reasoning performance with much lower teacher-input use, and notes verifier-related risk and supporting reproducibility materials.

6 CONCLUSION

  • SR-OPD makes prompt- and rollout-level routing decisions before teacher inference, using student success as a reference while leaving the standard OPD supervision objective unchanged.This reduces unnecessary teacher queries.Source: §6 · p. 9
  • 3.46–5.02% of Vanilla OPD’s teacher-input tokens sufficed across three teacher–student pairs and six mathematical reasoning benchmarks, with comparable reasoning performance in the one-pass setting.Matched-budget controlled experiments also supported prompt routing and success-referenced rollout selection.Source: §6 · p. 9

AI USE STATEMENT

  • The authors report using generative AI for literature retrieval, analysis-code development, and language editing, while retaining responsibility for design, analysis, verification, and manuscript content.Experimental results were obtained from saved outputs.Source: AI USE STATEMENT · p. 9

ETHICS STATEMENT

  • The work uses public mathematical-reasoning datasets and models, but systematic verifier errors may bias which rollouts receive supervision; verifier audits and label-corruption analyses assess this risk.Source: ETHICS STATEMENT · p. 9

REPRODUCIBILITY STATEMENT

  • The paper identifies Sections 3 and 4 as specifying the method and primary settings, and says supplementary materials provide experimental outputs, seed-level results, and analysis scripts for reproducing primary tables and figures.Source: REPRODUCIBILITY STATEMENT · p. 9

A SUPPORTING DETAILS, RESULTS, AND ANALYSIS

The appendix is organized to provide the method and implementation basis, experimental details, and analyses supporting the main paper.

A SUPPORTING DETAILS, RESULTS, AND ANALYSIS

A.1 SAMPLED-TOKEN OPD OBJECTIVE

The appendix describes the sampled-token OPD objective applied after routing and outlines how SR-OPD selects failed rollouts for teacher queries.

A.1 SAMPLED-TOKEN OPD OBJECTIVE

  • The unchanged sampled-token OPD objective is applied after routing, using student-visited decoding contexts and teacher token supervision.The token-level teacher–student log-probability difference is the rollout-generating student’s log probability minus the frozen teacher’s log probability for the sampled token.Source: §A.1 · pp. 11–12
  • The response-token mask retains valid response tokens, including terminal EOS, and excludes prompt tokens, padding, and invalid or unevaluable rollouts.The passage also introduces a detached token advantage and a one-epoch on-policy surrogate.Source: §A.1 · p. 11
  • The routing procedure discards unevaluable rollouts and makes no teacher query when valid rollouts lack either a success or a failure.Source: §A.1 · p. 12
  • For each failed rollout, the procedure compares successful references using trajectory alignment and persistent departure, then selects one failure by its cost-normalized score and deterministic tie-break.The teacher is queried only for the selected rollout.Source: §A.1 · p. 12

A.2 ROUTING-SCORE INTERPRETATION

PDA summarizes unrecovered alignment loss, and its cost-normalized score ranks failed rollouts within a prompt rather than measuring true learning utility.

A.2 ROUTING-SCORE INTERPRETATION

  • PDA records the magnitude and duration of unrecovered alignment loss, not a token-level semantic error location.Source: §A.2 · p. 12
  • Its cost-normalized score ranks failed rollouts within one prompt; the ranking does not directly measure learning utility or imply optimal allocation across prompts.Source: §A.2 · p. 12

A.3 COMPLETE COMPUTATION, IMPLEMENTATION, AND PROOFS

The appendix specifies how SR-OPD computes hidden-state trajectories and PDA, including how persistent departure captures both magnitude and duration. Cost normalization supports within-prompt ranking, not claims of global budget optimality or true learning utility.

A.3 COMPLETE COMPUTATION, IMPLEMENTATION, AND PROOFS

  • Student hidden states from the final normalization layer before the language-model head are sampled on a normalized response-position grid, with short-response positions allowed to map to the same token.All grid intervals still contribute to the computation, and empty or unevaluable generations are removed before prompt routing.Source: §A.3 · p. 12
  • PDA integrates the piecewise-linear trajectory departure, so sustained departures contribute more than temporary deviations that later recover.The appendix’s duration bound shows that departure magnitude over a set of positions contributes in proportion to that set’s measure.Source: §A.3 · p. 13
  • Multiplying PDA by Cmin/Cn cost-normalizes failed-rollout scores, preserving their within-prompt ranking while making no claim about global budget optimality or actual learning utility.The appendix also specifies deterministic tie-breaking by larger PDA, lower estimated cost, then smaller rollout ID; all-zero PDA reduces selection to least cost.Source: §A.3 · pp. 12–13

A.4 ADDITIONAL EXPERIMENTAL DETAILS, ACCOUNTING, AND STATISTICS

The appendix details the experimental accounting and isolates prompt routing, rollout selection, and selector components under controlled budgets. It also reports diagnostic, rollout-count, and verifier analyses with explicit limits on what their evidence establishes.

A.4 ADDITIONAL EXPERIMENTAL DETAILS, ACCOUNTING, AND STATISTICS

  • Native experiments use one training-data pass and 16 trajectory positions; teacher-input accounting counts serialized query inputs, while statistical intervals are pointwise, unadjusted summaries from three training seeds.Generation, verification, routing, and optimization are excluded from teacher-input accounting; controlled runs target 5% of Vanilla OPD use and may slightly exceed the target on the final query.Source: §A.4 · p. 13

A.5 CONTROLLED PROMPT ROUTING

  • At the matched 5% budget, all three paired Mixed+Random minus Global+Random Macro Avg effects are positive, with pointwise intervals above zero.The comparison changes the prompt gate while holding rollout selection to Random.Source: Table A2 · p. 14, §A.5 · p. 14

A.6 CONTROLLED ROLLOUT SELECTORS

  • The four controlled failed-rollout selectors differ in their pre-query ranking signal, ranging from no signal for Random to same-prompt persistent departure for SR-OPD.Adapted TLR uses length and entropy; StaticSR uses mean representation distance with the same reference and cost factor as SR-OPD.Source: §A.6 · p. 14

A.7 CONTROLLED ROLLOUT ROUTING

  • All three StaticSR differences are positive, but their intervals include zero; the stage-level comparison with Mixed+Random is stronger than this incremental comparison.Table A3 reports controlled rollout-routing results at the matched 5% teacher-input budget.Source: §A.7 · p. 15, Table A3 · p. 15

A.8 MECHANISM ABLATIONS

  • All twelve paired component-ablation means are positive, but every pointwise interval includes zero, so the evidence is directional.The tested contrasts vary reference source, persistent departure versus maximum departure, and inclusion of the teacher-input cost term.Source: §A.8 · p. 15, Table A5 · p. 16

A.9 TEACHER-SIDE DIAGNOSTICS

  • Teacher-side diagnostics use 200 failed items per pair, and their positive proxy associations do not establish long-horizon learning utility.The disjoint validation subset comes from the same DAPO-Math-17k dataset, so it is not an independent generalization set.Source: §A.9 · pp. 15–16, Table A6 · p. 16

A.10 ROLLOUT-COUNT SENSITIVITY

  • In a separate 3% cap sensitivity run, realized teacher-input ratios remain close to the nominal cap for both K = 8 and K = 16.This experiment does not include the primary K = 4, 5% configuration.Source: §A.10 · p. 17, Table A7 · p. 17, Table A8 · p. 17

A.11 VERIFIER AUDIT AND ROBUSTNESS

  • The verifier analysis audits 400 items per teacher–student pair and evaluates routing under 5% and 10% random label corruption; the supplied results describe the measures, not their values.Table A9 reports audit error rates with Wilson intervals and Macro Avg under label corruption.Source: §A.11 · p. 17, Table A9 · p. 18

Related papers

Loading 2610.02678v1…