PaperlayerView exact revision on arXiv

Source-linked AI summary

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

Yi Ding, Ruqi Zhang

arXiv:2608.31046v1cs.LGcs.CL

TL;DR

OPD’s teacher scores student-generated, off-policy trajectories, leaving the reliability and source of its gains unclear. The paper analyzes this supervision and finds that low-probability-token suppression with entropy-adaptive negative advantages enables OPSA, which improves reasoning performance without external supervision. Its scope is limited by evaluation on relatively small models and possible weakness when output distributions are already overly sharp.

  • Problem

    OPD provides dense token-level supervision, but teachers score student-generated trajectories that are off-policy for them, leaving supervision reliability and the source of student improvement unclear.

  • Method

    The paper analyzes OPD’s noise, effective tokens, and learning signals, then introduces OPSA with entropy-adaptive negative advantages and no external supervision.

  • Results

    OPSA generalizes across model families and tasks, improving Qwen3-1.7B Avg@32 by 263%–307% relative to the base model across three benchmarks and more than doubling Pass@32 on each.

  • Takeaways & Limitations

    Much of OPD’s improvement can be reproduced without teacher supervision, while entropy-adaptive negative advantages further support policy self-improvement.

  • Takeaways & Limitations

    Experiments use models up to 9B parameters, so OPSA’s scaling to larger or mixture-of-experts models remains unclear; benefits may also be limited for already-sharp policies.

Abstract

from arXiv · show

On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base \texttt{Qwen3-1.7B}, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.

1 INTRODUCTION

The paper questions whether OPD’s gains come from reliable teacher knowledge transfer, finding that teacher-free suppression of low-probability tokens can reproduce them and motivate OPSA.

  • Motivation: RLVR provides sparse response-level rewards, whereas OPD supplies dense token-level advantages from a teacher on student-sampled trajectories.OPD requires shared vocabularies and white-box teacher logits.
  • Findings: Teacher supervision in OPD is noisy, yet standard OPD, noisy-only training, and noisy-filtered training converge to comparable accuracy.This makes the source of OPD’s improvement unclear.
  • OPSA: OPSA is a supervision-free token-level RL method that assigns entropy-adaptive negative advantages to suppress tail tokens and redistribute probability mass among head tokens.It sharpens low-entropy distributions while preserving diversity at high-entropy fork tokens.
  • Findings: OPD improvement concentrates on low-log-probability tokens, while fixed negative advantages can match teacher-provided advantages.These findings suggest suppression, rather than teacher distillation, accounts for much of OPD’s gains.
  • Evaluation: 263%–307% relative Avg@32 improvements over the base model and more than doubled Pass@32 across three benchmarks demonstrate OPSA’s reported generalization.The result is reported for Qwen3-1.7B across the three benchmarks.

2 TEACHER SUPERVISION IN OPD IS HIGHLY NOISY, BUT STUDENTS IMPROVE REGARDLESS

OPD computes teacher-based token advantages on trajectories sampled by the student, creating an off-policy supervision concern. The analysis finds substantial noise, but student performance remains comparable across noise-filtering choices.

  • OPD setup: OPD minimizes reverse KL on student-sampled prefixes and assigns positive or negative token credits according to teacher preferences.The teacher’s scores are applied to trajectories sampled from the student policy.
  • Noise definition: Teacher supervision is noisy when answer-token advantage signs disagree with verifier-determined correctness.Incorrect answers with positive advantages and correct answers with negative advantages are both counted as noisy.
  • Noise results: 30.6% overall noise occurs with the 4B teacher, rising to 34.7% with the 30B-A3B teacher and 50.6% with the 235B-A22B teacher.For the 4B teacher, 20.4% of correct trajectories receive negative answer-token advantages and 40.8% of incorrect trajectories receive positive ones.
  • Noise results: Greater teacher capability is associated with supervision that becomes increasingly negative and less sensitive to answer correctness.The paper attributes this to increasing distributional mismatch between student and teacher policies.
  • Training comparison: Standard OPD and variants trained exclusively with or without noisy trajectories converge to comparable performance after similar numbers of gradient steps.This result suggests OPD gains do not depend straightforwardly on removing noisy supervision.

3 WHERE DOES STUDENT IMPROVEMENT COME FROM?

The analysis locates OPD’s effective learning in low-log-probability student tokens and negative advantages, motivating teacher-free training with fixed or entropy-adaptive suppression signals.

  • Which Tokens?: Most high-log-probability tokens provide negligible gradients, while OPD gains arise primarily from a small fraction of low-log-probability tokens.The analysis separates token selection from the type of learning signal.
  • Which Signals?: Negative advantages are the effective majority signal, and replacing all OPD advantages with a fixed negative value yields comparable performance.This motivates testing teacher-free advantage assignment.
  • Gradient analysis: Gradients vanish when advantage magnitude is small or when the sampled token’s student probability approaches one.Tokens in either regime have negligible impact on training.
  • Token-level evidence: 29.2% of tokens have exactly zero advantage, while 51.7% have advantage magnitude below 10^-4.Near-zero advantages are concentrated among tokens assigned high log probability by the student.
  • Token-level evidence: Restricting training to top-log-probability tokens produces no noticeable AIME24 improvement, even with original or random advantages.This supports their weak effective learning signal.
  • Implication: These findings motivate suppressing low-probability student-sampled tokens without a teacher.The paper presents this as the basis for its subsequent method.
  • Fixed advantages: Low-probability-token training with a fixed negative advantage steadily improves AIME24 avg@4, while matching full-token OPD’s comparable performance.The response length also increases gradually and eventually stabilizes.

4 METHODOLOGY

OPSA converts findings about low-log-probability tokens and entropy-dependent negative signals into a teacher-free training recipe. Its updates suppress sampled tail tokens while preserving or redistributing probability among alternatives according to position entropy.

  • 4 METHODOLOGY: OPSA is motivated by evidence that negative signals on low-log-probability tokens drive improvement, while entropy determines their appropriate magnitude.The method replaces teacher supervision with entropy-adaptive negative advantages.
  • 4 METHODOLOGY: δ = 1 assigns larger-magnitude negative advantages to higher-entropy tokens, whereas δ = −1 reverses this relationship and δ = 0 recovers fixed negative advantages.The relative entropy score r_i is computed over the lowest-20%-logp positions within each response.
  • 4.1 ENTROPY DETERMINES THE AMOUNT OF NEGATIVE SIGNALS: 50.0% avg@4 on AIME24 is reached with δ = 1, compared with 35.13% for standard OPD.The positively correlated variant substantially outperforms the negatively correlated variant, showing the benefit of stronger negative advantages at high-entropy tokens.
  • 4.3 WHY DOES OPSA WORK?: At high-entropy positions, OPSA suppresses sampled tail tokens and redistributes mass among head tokens, while at low-entropy positions it avoids tail sampling and preserves confident predictions.These updates steer reasoning away from unlikely branches without collapsing diversity at high-entropy fork positions.
  • 4.3 WHY DOES OPSA WORK?: High-entropy positions are associated with reflective tokens such as “wait” and “but,” which often signal reflection or self-correction.OPSA’s reshaped distribution increases the probability of these reflective reasoning behaviors.

5 EXPERIMENTS

Experiments evaluate OPSA across model families, mathematical reasoning, code generation, and general question answering, comparing it with several on-policy baselines. OPSA consistently improves performance, generalizes beyond its training domain, and preserves diversity while benefiting from a moderate token-selection ratio.

  • 5.1 EXPERIMENTAL SETUP: Experiments use Qwen3 and Qwen3.5 models trained on DAPO-17k questions without labels or ground-truth answers, evaluated on mathematical, code-generation, and general Q&A benchmarks.The mathematical benchmarks are AIME24, AIME25, and HMMT25; out-of-domain tasks include MBPP+ and GPQA-Diamond.
  • 5.1 EXPERIMENTAL SETUP: OPSA is compared with GRPO, TTRL, OPD, and OPSD as representative on-policy training methods.The comparison covers differences in supervision signals and training setup.
  • 5.2 MAIN RESULTS: 263%–307% relative Avg@32 gains are obtained by OPSA over Qwen3 base models across the three mathematical benchmarks.For Qwen3-1.7B, the relative improvements are 263% on AIME24 and AIME25 and 307% on HMMT25.
  • 5.2 MAIN RESULTS: 11.04 points in Avg@32 and 8.89 points in Pass@32 separate OPSA from the best baseline averaged across the three benchmarks.OPSA consistently outperforms all compared baselines while requiring no external supervision.
  • 5.2 MAIN RESULTS: OPSA improves both MBPP+ and GPQA-Diamond across different models, demonstrating generalization beyond the mathematical training domain.With thinking disabled during training, OPSA can match base-model performance with thinking enabled; enabling thinking at inference yields further improvements.
  • 5.3 ANALYSIS: Masking fork tokens largely eliminates OPSA’s increases in response length and accuracy, linking these gains to updates at reflective fork positions.These positions are where probability redistribution among competing head tokens supports long-form reasoning.
  • 5.3.3 ABLATION STUDY OF THE FRACTION OF TRAINED TOKENS: Training on the lowest 10% of log-probability tokens performs substantially worse, whereas the 20%, 30%, and 40% settings all raise Avg@4 above 45.The results indicate that OPSA is not highly sensitive to the exact token-selection ratio and that 20% is sufficient for substantial gains.

6 RELATED WORKS

Related work places OPSA among on-policy distillation, self-distillation, and label-free policy-improvement methods. The supplied passages also include an ablation figure concerning OPSA’s training-token ratio.

  • 6 RELATED WORKS: On-policy distillation methods provide token-level supervision on student-generated trajectories, while related self-distillation methods replace or modify the external teacher.The related-work discussion includes GKD, MiniLLM, K1-based OPD, and OPSD.
  • 6 RELATED WORKS: Label-free policy-improvement methods derive training signals from the model or its rollouts without external supervision.Examples include self-rewarding methods, TTRL, and EMPO.

7 CONCLUSION

The paper argues that OPD’s gains largely arise from suppressing low-probability student tokens rather than requiring teacher supervision. OPSA uses entropy-adaptive negative advantages to enable self-improvement while reshaping token probabilities and preserving exploration at reasoning forks.

  • 7 CONCLUSION: OPD improvement is largely reproduced by fixed negative advantages on low-probability tokens, without teacher supervision.The analysis identifies low-log-probability student-sampled tokens as the main locus of effective learning.
  • 7 CONCLUSION: OPSA suppresses tail-token probabilities and redistributes probability mass among competing head tokens.This adaptively sharpens predictions at low-entropy positions while preserving diversity at high-entropy reasoning forks.
  • 7 CONCLUSION: OPSA provides external-supervision-free policy self-improvement using entropy-adaptive negative advantages.The method is motivated by the finding that policy uncertainty can provide a fine-grained learning signal without rewards, reference answers, or teacher supervision.
  • 7 CONCLUSION: OPSA outperforms OPD without teacher supervision while exhibiting similar training dynamics, including response-length growth.This comparison motivates reexamining whether OPD gains reflect teacher imitation, student-distribution reshaping, or both.

A LIMITATIONS AND FUTURE DIRECTIONS

The experiments are limited to relatively small models, and OPSA’s scaling and usefulness for already-sharp policies remain uncertain. The method may also have limited ability to expand the underlying exploration frontier.

  • A LIMITATIONS AND FUTURE DIRECTIONS: Experiments cover models up to 9B parameters, leaving OPSA’s effectiveness on larger models and mixture-of-experts architectures unclear.The scope boundary is attributed to limited computational resources.
  • A LIMITATIONS AND FUTURE DIRECTIONS: OPSA may offer limited benefits for heavily post-trained models whose output distributions are already overly sharp and have very low entropy.The method primarily redistributes existing probability mass.
  • A LIMITATIONS AND FUTURE DIRECTIONS: OPSA may not substantially expand the policy’s exploration frontier, as reflected by relatively modest improvements in thinking-mode Pass@k.The paper suggests combining OPSA with other reinforcement learning methods to improve exploration.

B.1 TRAINING CONFIGURATION

The paper provides a dedicated table of OPSA training hyperparameters and specifies rollout decoding choices for training alongside checkpoint selection based on Avg@4.

  • B.1 TRAINING CONFIGURATION: Table 4 provides the training hyperparameters used for OPSA.The detailed configuration is referenced as the paper’s training setup.
  • B.1 TRAINING CONFIGURATION: Training rollouts use the default decoding parameters from the slime repository.This setting applies during rollout generation rather than evaluation.
  • B.1 TRAINING CONFIGURATION: The best checkpoint for each method is selected using Avg@4 performance.Evaluation decoding parameters follow the Qwen3 model card, while checkpoint selection follows prior work.

B.2 EVALUATION CONFIGURATION

The evaluation details specify dedicated hyperparameter tables, 32-response sampling, and reporting with Avg@32 and Pass@32. Additional analyses compare overhead, reflective-fork filtering, and response diversity using token-level 4-gram Jaccard distance.

  • B.2 EVALUATION CONFIGURATION: Table 5 provides the evaluation hyperparameters, while Table 6 compares OPSA’s training and inference overhead with baselines.The overhead comparison concerns both training and inference costs.
  • B.2 EVALUATION CONFIGURATION: Evaluation generates 32 responses per prompt with thinking mode disabled and reports Avg@32 and Pass@32.Inference uses SGLang and Qwen3 model-card decoding parameters.
  • B.2 EVALUATION CONFIGURATION: OPSA and OPD train faster than GRPO because they avoid extensive rollout sampling for response-group construction.OPSA is additionally more efficient than OPD because it does not deploy or run forward passes through a larger teacher.
  • B.3 MORE DETAILS: Reflective-fork analysis excludes low-log-probability positions when a top-5 candidate belongs to the predefined reflective word set.The reflective word set includes terms such as “wait,” “however,” “alternatively,” and “check.”
  • B.3 MORE DETAILS: Response diversity is measured with pairwise Jaccard distance between sets of token-level 4-grams retained from response prefixes.Prefix lengths are L ∈ {512, 1024, 2048, 4096, Full}; larger DJ(L) indicates greater response diversity.
  • B.3 MORE DETAILS: AIME24 diversity averages distances across all pairs among R = 32 sampled responses and then macro-averages across P = 30 problems.The analysis also reports the difference between OPSA and the base model.

C.1 COMPARISON TO NEGATIVE SAMPLE REINFORCEMENT

OPSA is compared with negative-sample reinforcement methods and evaluated under matched inference budgets. It achieves higher Pass@32 while allocating uncertainty differently across token positions.

  • OPSA achieves higher pass@32 than GRPO and NSR on AIME24, AIME25, and HMMT25.
  • OPSA exhibits substantially lower training entropy than GRPO and NSR while maintaining higher pass@32.Aggregate entropy alone is therefore not a reliable indicator of exploration ability; uncertainty allocation across token positions matters more.
  • OPSA’s exploration preserves balanced probability among plausible head tokens at high-entropy fork positions.It suppresses tail tokens while maintaining diversity where multiple continuations remain plausible.
  • A token-budget-matched control tests whether OPSA’s gains are explained merely by generating longer responses.The control modifies GRPO and OPD responses and applies a minimum generation-length constraint to approximately match OPSA’s average response length.

C.4 OPSA AS A COLD-STARTING FOR GRPO TRAINING

The paper tests whether OPSA can provide a cold start for subsequent GRPO training. An OPSA-initialized Qwen3-4B model is further trained with GRPO on DAPO-17k.

  • Figure 12 reports the resulting training dynamics for GRPO started from the OPSA cold-starting 4B model.
  • OPSA checkpoints are used to initialize Qwen3-4B before additional GRPO training on DAPO-17k.
  • The cold-start experiment investigates whether OPSA can enable additional performance gains during subsequent reinforcement learning.
Loading 2608.31046v1…