PaperlayerView exact revision on arXiv

Source-linked AI summary

SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillation

Zikun Qu, Min Zhang, Mingze Kong, Zhiwei Shang, Yikun Ban, Shuang Qiu, Zhongxiang Dai

arXiv:2608.04419v1cs.LGcs.AI

TL;DR

Standard OPD may underrepresent plausible alternatives, while teacher probabilities do not reliably identify which candidates lead to successful student continuations. SPOT selectively probes uncertain, mismatched positions and calibrates teacher-based targets with verifier-scored outcomes, achieving the highest macro Pass@8 across evaluated settings while maintaining competitive average accuracy.

  • Problem

    Standard reverse-KL OPD can under-cover plausible alternatives, while teacher entropy and local probabilities do not identify useful probing positions or downstream-successful candidates.

  • Method

    SPOT uses acquisition–exploration–exploitation to allocate sparse probes using teacher uncertainty, top-k mass, and student mismatch, then outcome-calibrates teacher-anchored targets with verifier-scored continuations.

  • Results

    SPOT achieves the highest macro Pass@8 in every evaluated setting and the highest or second-highest macro Avg@8 across student scales, training sets, benchmarks, and model families.

  • Takeaways & Limitations

    The results indicate stronger multi-sample solution coverage while maintaining competitive average accuracy across the evaluated settings.

Abstract

from arXiv · show

On-policy distillation (OPD) provides dense teacher supervision on student-generated trajectories, but standard reverse-KL training can assign insufficient probability to other plausible continuations. Teacher entropy alone does not reveal whether uncertainty is concentrated among a few plausible next tokens or dispersed over a long probability tail, nor whether the student already represents those candidates well. Moreover, local teacher probabilities may not predict downstream success. We introduce Sparse Probing and Outcome-calibrated Targets OPD (SPOT), which addresses two coupled decisions, where to probe and what to distill, through an acquisition--exploration--exploitation procedure. During acquisition, a position-level score combines normalized teacher entropy, the probability mass captured by a small top-$k$ candidate set, and student--teacher mismatch to allocate a limited probing budget. During exploration, SPOT evaluates teacher-proposed candidates through verifier-scored student continuations. During exploitation, these outcomes produce a closed-form, KL-regularized target that favors candidates with better downstream outcomes while remaining anchored to the teacher distribution. Extensive experiments across multiple student models and reasoning benchmarks demonstrate the effectiveness of SPOT in improving reasoning performance while balancing solution quality and coverage.

1 INTRODUCTION

SPOT frames on-policy distillation as two coupled decisions: where to probe and what to distill. It uses acquisition, exploration, and exploitation to select mismatched uncertain positions and construct outcome-calibrated targets from verifier-scored continuations.

  • 1 INTRODUCTION: Standard reverse-KL OPD favors the teacher’s dominant continuation and may underweight other plausible continuations, limiting solution coverage.EOPD partially addresses this with teacher-entropy-triggered top-k forward-KL augmentation.
  • 1 INTRODUCTION: SPOT allocates probing to positions where multiple teacher alternatives have meaningful probability but are poorly represented by the student.This targets uncertainty concentrated among plausible candidates rather than indiscriminate probing.
  • 1 INTRODUCTION: Teacher local probabilities need not predict downstream success, motivating outcome evaluation of candidate branches under the current student policy.A lower-probability alternative can succeed when a higher-probability continuation fails.
  • 1 INTRODUCTION: SPOT separates position selection from target construction through acquisition, exploration, and exploitation, combining teacher uncertainty, top-k mass, mismatch, and verifier outcomes.Its target is derived by reward-tilting the teacher distribution while retaining the teacher as a proposal prior.
  • 1 INTRODUCTION: SPOT achieves the highest macro Pass@8 in all three evaluated settings and the highest or second-highest macro Avg@8 across six mathematical reasoning benchmarks.The results indicate stronger multi-sample solution coverage while maintaining competitive average accuracy.

2 PRELIMINARIES

OPD queries the teacher on student-generated prefixes and trains with reverse-KL supervision, providing dense feedback but potentially under-covering plausible alternatives. EOPD adds entropy-triggered coverage, while SPOT revisits both where to intervene and what to distill.

  • On-Policy Distillation: OPD samples student trajectories, queries the teacher on each visited prefix, and applies reverse-KL loss for dense token-level supervision.In practice, the objective uses tokens sampled by a frozen behavior policy and PPO-style clipping.
  • On-Policy Distillation: Because reverse KL is mode-seeking, OPD can under-cover plausible alternatives when the teacher is uncertain.Teacher queries on student-visited prefixes reduce statedistribution mismatch while retaining dense feedback.
  • Entropy-Aware On-Policy Distillation: EOPD augments OPD with mode-covering supervision at positions of high teacher entropy using an entropy-based intervention criterion.Its formulation uses teacher top-k candidates with truncated, renormalized targets while retaining the student’s full-vocabulary probabilities.
  • Motivation: SPOT revisits both acquisition and distillation because EOPD relies on a scalar entropy criterion and an uncalibrated teacher prior.SPOT distinguishes probability mass from relative shape through top-k-renormalized quantities.

3 METHODOLOGY

SPOT uses an acquisition–exploration–exploitation procedure to decide where to probe and what to distill. It prioritizes positions with compact, plausible teacher alternatives that the student mismatches, evaluates candidates through verifier-scored continuations, and converts outcomes into KL-regularized targets anchored to the teacher.

  • Acquisition–Exploration–Exploitation: SPOT’s three stages allocate evidence by selecting positions, evaluating teacher candidates with student continuations, and constructing outcome-calibrated distillation targets.The procedure addresses the coupled decisions of where to probe and what to distill.
  • Position Acquisition: Where to Probe: The acquisition score favors positions where multiple plausible candidates capture most teacher mass and the student underweights or differently ranks them.This combines teacher entropy, top-ks mass concentration, and student–teacher mismatch, rather than relying on entropy alone.
  • Training Objective: SPOT adds local outcome-calibrated supervision at verifier-supported positions, while the auxiliary branch term vanishes when B+ = ∅ and standard OPD remains.The coefficient β controls the auxiliary loss strength relative to trajectory-wide OPD.
  • Sparse Probing and Outcome Calibration: What to Distill: At selected positions, SPOT probes kp teacher candidates by sampling Np frozen-student continuations and estimating each candidate’s downstream value with a verifier.Probing is retained only when at least one candidate has positive estimated continuation value, with overhead bounded by M × kp × Np continuations per trajectory.
  • Outcome-Calibrated Targets: Outcome-calibrated targets maximize expected downstream value within a KL trust region around the renormalized teacher prior instead of collapsing onto the empirically best candidate.The closed-form target adds an outcome-grounded continuation-value bonus to teacher log-probabilities; γ controls how strongly higher-value candidates are favored.

4 EXPERIMENTS

Across student scales, datasets, benchmarks, and architectures, SPOT improves multi-sample reasoning coverage while preserving or improving average accuracy. Ablations attribute these gains to its acquisition score and verifier-guided outcome calibration, with performance depending on the branch-supervision weight.

  • Main results: SPOT achieves the best macro Pass@8 across all three student scales and improves macro Avg@8 and Pass@8 over OPD and EOPD.Against OPD, gains are 0.47–1.48 points in macro Avg@8 and 4.55–5.28 points in macro Pass@8; against EOPD, gains are 0.29–0.68 and 2.49–3.19 points, respectively.
  • Main results: SPOT’s larger Pass@8 gains alongside preserved or improved Avg@8 indicate broader viable-solution coverage without diluting individual-attempt accuracy.Avg@8 measures per-sample correctness, whereas Pass@8 measures whether eight samples include at least one correct solution.
  • Acquisition ablation: The full acquisition score performs best, reaching 21.51/41.60 macro Avg@8/Pass@8, while omitting top-k mass or student–teacher mismatch reduces performance.The ablation compares entropy, top-k mass, combined mass-undercoverage, and JS shape-mismatch components under identical training and evaluation settings.
  • Verifier-guidance ablation: Verifier-guided calibration improves the macro Avg@8/Pass@8 results over using the uncalibrated teacher proposal prior at probed positions.The ablation removes reward tilting and positive-reward gating while retaining full acquisition and branch distillation; the supplied passage reports a 3.21-point improvement.
  • Coverage across sampling budgets: SPOT ranks first or ties from k = 8 onward and retains 12.50–16.67-point gains over OPD at k = 64 on the competition benchmarks.Methods are close at k = 4, while EOPD leads on AIME25 at that budget.
  • Sensitivity analysis: β = 0.1 leads the three competition benchmarks and ties β = 0.5 on MATH500, while the competition mean Pass@8 is lowest at β = 1.0.The coefficient β controls the tradeoff between trajectory-wide OPD and outcome-calibrated branch supervision.

5 RELATED WORK

Related work frames SPOT within knowledge distillation and on-policy distillation for reasoning models. Knowledge distillation transfers teacher capabilities through predictive-distribution matching or sequence imitation, while OPD uses student-generated rollouts with dense token-level feedback.

  • Knowledge Distillation: Knowledge distillation transfers capabilities from high-capacity teachers to compact students by matching predictive distributions or imitating teacher-generated sequences.The cited paradigm includes Hinton et al. (2015) and Kim & Rush (2016).
  • Knowledge Distillation: Recent distillation methods use alternative divergence objectives, contrastive learning, and selective supervision based on sample difficulty or token importance.The passage attributes these developments to Ko et al. (2024; 2025), He et al. (2025), and Guo et al. (2025).
  • On-Policy Distillation for Reasoning Models: On-policy distillation trains students on self-generated rollouts with dense token-level teacher feedback, mitigating exposure bias and improving credit assignment over outcome-only RL.The passage describes OPD as providing finer credit assignment than outcome-only reinforcement learning.
  • On-Policy Distillation for Reasoning Models: Recent OPD work broadens teacher access and transfer scope through reward extrapolation, black-box teachers, and multi-teacher distillation.These directions are associated with Yang et al. (2026b) and Ye et al. (2026a) in the supplied passage.

6 CONCLUSION

SPOT addresses reverse-KL OPD’s under-coverage of plausible alternatives by allocating sparse probing where teacher uncertainty and student–teacher mismatch indicate value, then distilling verifier-informed outcomes through teacher-anchored KL-regularized targets.

  • Conclusion: SPOT combines normalized teacher entropy, top-k probability mass, and student–teacher mismatch to allocate a sparse probing budget.This addresses the limitation that teacher entropy alone cannot identify where probing is most useful.
  • Conclusion: Verifier-scored student continuation values produce outcome-calibrated targets that remain anchored to the teacher proposal prior.The targets are constructed with KL regularization to balance outcome information and teacher guidance.

A DERIVATION AND PROPERTIES OF OUTCOME-CALIBRATED TARGETS · A.1 CLOSED-FORM DERIVATION AND TRUST-REGION EQUIVALENCE

The appendix formalizes outcome-calibrated targets on a finite candidate set with positive teacher prior probabilities. It shows that KL-regularized value optimization and a KL trust-region formulation yield the same unique Gibbs-form solution, with γ controlling the induced radius.

  • A DERIVATION AND PROPERTIES OF OUTCOME-CALIBRATED TARGETS: The derivation fixes a retained position and treats V as the probed branch-value estimate over a finite candidate set S.The teacher prior satisfies p(v) > 0 for every v ∈ S, and Δ(S) denotes the probability simplex.
  • A.1 CLOSED-FORM DERIVATION AND TRUST-REGION EQUIVALENCE: For γ > 0, the KL-regularized objective rewards high estimated downstream value while anchoring the target to the teacher prior.Proposition A.1 states that this objective has a unique optimizer.
  • A.1 CLOSED-FORM DERIVATION AND TRUST-REGION EQUIVALENCE: The Gibbs variational identity shows that the regularized objective is maximized exactly when the candidate distribution equals ργ.This follows because DKL(ρ∥ργ) = 0 if and only if ρ = ργ.
  • A.1 CLOSED-FORM DERIVATION AND TRUST-REGION EQUIVALENCE: The regularized and trust-region formulations describe the same solution path, with radius ϵ(γ) = DKL(ργ∥p).Proposition A.2 states that ργ is the unique solution of the corresponding trust-region problem for every γ > 0.
  • A.1 CLOSED-FORM DERIVATION AND TRUST-REGION EQUIVALENCE: When the KL constraint is active, the KKT multiplier η gives γ = 1/η and produces ρ(v) ∝ p(v) exp(V(v)/η).Thus γ is the reciprocal of the KL multiplier, while a shared γ induces a position-dependent trust-region radius ϵt(γ).
  • A.1 CLOSED-FORM DERIVATION AND TRUST-REGION EQUIVALENCE: A feasible distribution with larger expected value would contradict regularized optimality, establishing uniqueness of the trust-region optimizer.The proof uses the fact that a feasible distribution has no larger KL divergence from p than ργ.

A.2 GEOMETRY OF THE CALIBRATION PATH · A.3 ESTIMATED-VALUE IMPROVEMENT UNDER TEACHER ANCHORING

The calibration path increases estimated downstream value as it moves farther from the teacher prior, with diminishing returns under a relaxed KL budget. Finite-temperature anchoring preserves teacher support, while the improvement guarantee applies only to the probed value estimate, not unknown population performance.

  • A.2 GEOMETRY OF THE CALIBRATION PATH: For nonconstant V, increasing γ strictly raises estimated branch value and divergence from the teacher prior.This establishes γ as a monotone value–prior tradeoff parameter.
  • A.2 GEOMETRY OF THE CALIBRATION PATH: The induced radius increases from 0 to −log P⋆, and γ corresponds one-to-one with the radius on the nondegenerate path.Here P⋆ denotes the teacher probability mass on maximum-value candidates.
  • A.2 GEOMETRY OF THE CALIBRATION PATH: When multiple candidates share the maximum value, the large-γ limit preserves their relative teacher probabilities instead of selecting arbitrarily.The calibration therefore retains teacher-prior distinctions among tied optima.
  • A.2 GEOMETRY OF THE CALIBRATION PATH: Relaxing the teacher-centered KL budget improves optimal estimated branch value with strictly diminishing marginal returns.The curvature result formalizes that additional KL budget yields progressively smaller value gains.
  • A.2 GEOMETRY OF THE CALIBRATION PATH: At every finite temperature, calibration preserves the teacher’s top-kp support.Finite γ changes relative probabilities through exponential tilting without removing teacher-supported candidates.
  • A.3 ESTIMATED-VALUE IMPROVEMENT UNDER TEACHER ANCHORING: The improvement–deviation bounds quantify both the estimated-value benefit of exponential tilting and its conservatism under teacher anchoring.The target is optimized relative to the teacher prior, which supplies the anchoring comparator.
  • A.3 ESTIMATED-VALUE IMPROVEMENT UNDER TEACHER ANCHORING: The guarantee concerns the probed estimate V = ˆVt and does not itself establish improvement in the unknown population branch value.This limits the formal claim to the value quantity used during probing.

A.4 PREFERENCE CORRECTION AND THE BINARY-VERIFIER CASE

SPOT changes teacher preferences only when outcome evidence overcomes prior log-odds, with γ setting the reversal threshold. In the binary-verifier case, calibration reallocates probability only between successful and unsuccessful branches while preserving teacher-relative probabilities within each group.

  • Preference correction: γ sets the evidence threshold for reversing a teacher ranking, while equal candidate values preserve teacher-relative odds.Adding the same constant to all branch values leaves the calibrated target unchanged.
  • Binary-verifier case: With binary outcomes, one probe rollout per candidate yields branch-value estimates in {0, 1}, separating successful branches S+ from unsuccessful branches.The binary-verifier odds update uses prior successful mass P+ and calibrated successful mass Q+.
  • Binary-verifier case: The calibrated successful-to-unsuccessful odds equal e^γ P+/(1 − P+) relative to the prior odds P+/(1 − P+).This follows the closed-form relation Q+/(1 − Q+) = e^γ P+/(1 − P+).
  • Binary-verifier case: Calibration spends all KL deviation on reallocating mass between outcome groups and introduces no within-group distortion.Within successful and unsuccessful groups, calibrated probabilities preserve teacher-relative probabilities; D_KL(ρ_γ∥p) equals the Bernoulli KL between Q+ and P+.
  • Binary-verifier case: The B+ gate removes positions with P+ = 0, while P+ = 1 leaves all candidate values equal and gives ρ_γ = p.Only the informative mixed case reallocates aggregate prior odds between successful and unsuccessful branches.

B IMPLEMENTATION DETAILS · C MORE ABLATION EXPERIMENT RESULTS · C.1 OUT-OF-DOMAIN GENERALIZATION

The paper specifies reproducible off-policy, on-policy, evaluation, and SPOT-specific procedures, then reports a structured out-of-domain transfer profile in which SPOT improves GPQA-Diamond and AlpacaEval while nearly matching EOPD on MMLU-Pro.

  • B IMPLEMENTATION DETAILS: Off-policy KD uses DistillKit with one teacher response sampled per training problem.Table 4 summarizes the off-policy configuration.
  • B IMPLEMENTATION DETAILS: On-policy OPD, EOPD, GRPO, and SPOT use verl with asynchronous SGLang rollouts on 4 × A800 GPUs, FSDP2, and bfloat16.Table 5 summarizes the principal training hyperparameters.
  • B IMPLEMENTATION DETAILS: All methods use AdamW, 0.01 weight decay, gradient clipping at 1.0, and four mini-batch updates, while distillation methods omit learning-rate warmup.EOPD adds a unit-weight top-16 forward-KL loss when teacher entropy is at least 0.8; GRPO uses KL and entropy coefficients of 10^-3 and 0.
  • B IMPLEMENTATION DETAILS: SPOT-specific hyperparameters cover acquisition, probing, gating, and loss settings, which are defined in Algorithm 1 and Section 3.2.Table 7 summarizes these SPOT-specific hyperparameters.
  • B IMPLEMENTATION DETAILS: Evaluation uses vLLM in bfloat16 with seed 42, native chat templates, disabled thinking, boxed-answer extraction, and Math-Verify equivalence checking.Missing or unparsable answers are counted as incorrect.
  • C.1 OUT-OF-DOMAIN GENERALIZATION: SPOT improves GPQA-Diamond over the strongest baseline by 2.21 points in Avg@8 and 11.62 points in Pass@8, while leading AlpacaEval WR by 1.40 points.The benchmarks diagnose transfer across deliberative reasoning, broad knowledge, and instruction following.
  • C.1 OUT-OF-DOMAIN GENERALIZATION: On MMLU-Pro, SPOT ranks second at 42.26, trailing EOPD by 0.64 points while exceeding GRPO and OPD by 0.40 and 1.03 points, respectively.This result indicates near-best broad subject-matter reasoning despite the structured transfer profile.

C.2 CONSISTENCY ACROSS MODEL FAMILIES

On Llama-3.2-3B-Instruct students with Llama-3.1-8B-Instruct teachers, SPOT improves macro Avg@8 and Pass@8 over OPD, with especially consistent gains in Pass@8. This pattern mirrors Qwen results across model families, capacities, checkpoints, and training-data regimes.

  • Evaluation setup: SPOT is evaluated within the Llama family using a Llama-3.2-3B-Instruct student trained on MATH and a Llama-3.1-8B-Instruct teacher.All methods retain native optimization protocols and use eight samples under identical decoding.
  • Within-family comparison: 17.53 vs. 12.57: SPOT raises macro Avg@8 from OPD’s 12.57 to 17.53, while macro Pass@8 increases from 29.65 to 34.05.These are within-family Llama comparisons across four benchmarks.
  • Within-family comparison: SPOT’s macro Avg@8 is essentially tied with GRPO at 17.53 vs. 17.45, while its macro Pass@8 exceeds EOPD by 2.54 points.The advantage is asymmetric across the two metrics.
  • Within-family comparison: SPOT is best or tied-best in Pass@8 on all four Llama benchmarks, although GRPO is stronger in AMC23 Avg@8 and OPD ties.The benchmark-level pattern matches the macro-level asymmetry.
  • Cross-family interpretation: Across Qwen and Llama settings, SPOT leads macro Pass@8 in every reported setting while macro Avg@8 is best or nearly tied with the strongest baseline.The recurrence spans Base and Instruct checkpoints, two model families, multiple capacities, and two training-data regimes.
Loading 2608.04419v1…