SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting

Binbin Zheng; Xing Ma; Yiheng Liang; Jingqing Ruan; Xiaoliang Fu; Kepeng Lin; Benchang Zhu; Ke Zeng; Xunliang Cai

Source-linked AI summary

SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting

Binbin Zheng, Xing Ma, Yiheng Liang, Jingqing Ruan, Xiaoliang Fu, Kepeng Lin, Benchang Zhu, Ke Zeng, Xunliang Cai

arXiv:2604.10688v2Apr 12, 2026cs.LGcs.AIcs.CL

TL;DR

Sparse outcome rewards make token-level credit assignment difficult, and uniform OPD supervision overlooks rollout-level signal quality. SCOPE routes correct and incorrect trajectories into complementary perplexity-weighted objectives, achieving average relative improvements of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines on six mathematical reasoning benchmarks.

Problem
OPD provides dense teacher supervision but assumes it is uniformly reliable, while uniform treatment can mishandle corrective guidance and learning utility across rollout types.
Method
SCOPE routes rollouts by correctness, applies teacher-perplexity-weighted KL distillation to incorrect trajectories and student-perplexity-weighted MLE to correct trajectories, and normalizes weights by group.
Results
11.42% average relative improvement in Avg@32 and 7.30% in Pass@32 are reported over competitive baselines across six mathematical reasoning benchmarks.
Takeaways & Limitations
SCOPE provides signal-quality-aware supervision that reinforces boundary capabilities on valid paths while distilling informative corrections on flawed ones.
Takeaways & Limitations
SCOPE depends on automatically verifiable outcomes and is evaluated mainly in domains such as mathematical reasoning and code generation; subjective-feedback tasks may require additional verification mechanisms.

Abstract

from arXiv · show

On-policy reinforcement learning has become the dominant paradigm for reasoning alignment in large language models, yet its sparse, outcome-level rewards make token-level credit assignment notoriously difficult. On-Policy Distillation (OPD) alleviates this by introducing dense, token-level KL supervision from a teacher model, but typically applies this supervision uniformly across all rollouts, ignoring fundamental differences in signal quality. We propose Signal-Calibrated On-Policy Distillation Enhancement (SCOPE), a dual-path adaptive training framework that routes on-policy rollouts by correctness into two complementary supervision paths. For incorrect trajectories, SCOPE performs teacher-perplexity-weighted KL distillation to prioritize instances where the teacher demonstrates genuine corrective capability, while down-weighting unreliable guidance. For correct trajectories, it applies student-perplexity-weighted MLE to concentrate reinforcement on low-confidence samples at the capability boundary rather than over-reinforcing already mastered ones. Both paths employ a group-level normalization to adaptively calibrate weight distributions, accounting for the intrinsic difficulty variance across prompts. Extensive experiments on six reasoning benchmarks show that SCOPE achieves an average relative improvement of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines, demonstrating its consistent effectiveness.

1 Introduction

SCOPE addresses signal-quality heterogeneity in OPD by routing correct and incorrect rollouts into complementary, adaptively weighted supervision paths. It reports improved Avg@32 and Pass@32 performance over competitive baselines on mathematical benchmarks.

Motivation: Sparse outcome-level rewards make token-level credit assignment difficult, while OPD adds dense teacher KL supervision but treats rollout signals uniformly.This uniform-reliability assumption overlooks differences between corrective guidance on failed trajectories and learning utility on successful ones.
Signal calibration: Teacher and student perplexity provide complementary indicators of supervision quality for incorrect and correct trajectories, respectively.The framework prioritizes reliable corrective guidance while emphasizing correct samples near the capability boundary.
Framework: SCOPE routes rollouts by correctness, using teacher-perplexity-weighted KL distillation for incorrect trajectories and student-perplexity-weighted MLE for correct trajectories.Both paths use group-level normalization to calibrate weights within prompts of varying difficulty.
Results: 11.42% average relative improvement in Avg@32 and 7.30% in Pass@32 are reported over competitive baselines on mathematical benchmarks.The experiments cover six reasoning benchmarks and include extended experiments for broader applicability.

2 Preliminary Analysis

Preliminary analyses identify two weaknesses of uniform on-policy optimization: reinforcing successful trajectories can reduce reasoning diversity, while teacher correction can become inefficient on flawed prefixes. These findings motivate complementary weighting for correct and incorrect rollouts.

2.1 Diversity Degradation: Uniformly reinforcing correct trajectories amplifies dominant reasoning paths and marginalizes valid low-probability alternatives, while dense teacher signals can suppress diverse student explorations.The combined effect is described as severe mode collapse.
2.1 Diversity Degradation: Pass@1 improves while Pass@32 falls from 93.7% to 84.9% under PSR on Qwen2.5-7B, illustrating the Pass@k paradox.The passage presents this as evidence that uniform optimization can sharpen a policy at the expense of diversity.
2.1 Diversity Degradation: Pass@1 increases while Pass@32 drops from 76.5% to 75.0% when OPD is applied to all generated trajectories of DeepSeek-R1-Distill-Qwen-1.5B.This mirrors the diversity trade-off observed with PSR.
2.1 Diversity Degradation: Uniform optimization over-reinforces dominant correct solutions and extinguishes rare unconventional paths, motivating greater weight for under-explored correct trajectories.The proposed weighting target is preservation of reasoning diversity.
2.2 Rectification Inefficiency: Teacher correction is inefficient when flawed student prefixes degrade the context and make the teacher’s guidance noisy.The analysis frames this as a bottleneck for correcting incorrect on-policy trajectories.
2.2 Rectification Inefficiency: Low-PPL prefixes outperform high-PPL prefixes in teacher recovery across truncation levels by up to +19.4%.At an 80% truncation ratio, even the best-performing group declines to approximately 35%, supporting down-weighting high teacher-PPL samples.

3 Methodology

SCOPE branches on verified rollout correctness and applies distinct surrogate objectives with group-relative perplexity weighting. Correct trajectories receive student-guided MLE reinforcement, while incorrect trajectories receive selectively weighted teacher KL correction, followed by a unified normalized objective.

3 Methodology: SCOPE routes on-policy rollouts by outcome and uses perplexity-calibrated weighting to filter misleading teacher noise.The method consists of outcome-driven branching, dual-path weighting, and an overall objective.
3.1 Outcome-Driven Group Branching: For each prompt, the student generates N responses, and a verifier assigns each response a binary reward used to partition correct and incorrect trajectory sets.The correct set contains responses with R_i = 1, while the incorrect set contains responses with R_i = 0.
3.1 Outcome-Driven Group Branching: The on-policy formulation accounts for behavior-policy distribution shift with a token-level importance-sampling ratio before optimizing the partitioned subsets.Separate surrogate objectives are then defined for correct and incorrect trajectories.
3.1 Outcome-Driven Group Branching: Correct trajectories are optimized by maximizing their likelihood, reinforcing self-generated valid reasoning rather than relying on teacher guidance.This branch uses the model’s intrinsic successful attempts as the supervisory target.
3.1 Outcome-Driven Group Branching: Incorrect trajectories use teacher-policy guidance through an on-policy distillation objective based on token-level log-ratios.The teacher branch is intended to provide external corrective supervision where failed trajectories lack inherent targets.
3.2 Dual-Path Adaptive Weighting: DPAW operates within each prompt’s candidate response group and applies distinct perplexity-based weighting to correct and incorrect trajectories.The framework is shown as replacing standard OPD’s uniform supervision with branching, weighting, and a unified objective.
3.2 Dual-Path Adaptive Weighting: Sequence perplexity is computed from length-normalized sequence log-probability to quantify trajectory uncertainty.The weighting mechanism uses this sequence-level quantity for group-relative calibration.
3.2 Dual-Path Adaptive Weighting: Correct but high-perplexity student trajectories receive amplified supervision to emphasize low-confidence alternative reasoning paths.The student-guided weight uses a group-relative softmax over length-normalized negative log-probabilities.

4 Experiment

SCOPE is evaluated against reinforcement-learning and distillation baselines across mathematical and code-generation benchmarks, with analyses of training dynamics, Pass@k behavior, and its adaptive-weighting ablation. It consistently improves reasoning performance and preserves exploration diversity.

Experimental Setup: Experiments use two student–teacher configurations trained on DeepMath and compare SCOPE with GRPO, KD, and OPD.The evaluation covers six mathematical benchmarks and reports Avg@32 for stability and Pass@32 for upper-bound capability.
Main Results: SCOPE achieves the best Avg@32 performance under the primary mathematical-reasoning configuration, improving over standard OPD by an average relative +5.54%.The gains include +10.69% on Olympiad and +6.59% on AMC23.
Training Dynamics: SCOPE sustains policy entropy and achieves superior performance and sample efficiency compared with GRPO and OPD during training.GRPO shows continuous entropy decay, whereas OPD and SCOPE maintain healthier policy entropy; OPD later plateaus.
Pass@k Performance: SCOPE’s Pass@k continues improving through k = 32 across AIME24, AIME25, and AMC23, unlike baselines with restricted diversity scaling.The method amplifies unconventional valid trajectories near the student’s capability boundary.
Ablation Study: Removing DPAW drops AIME25 Pass@32 from 50.9% to 45.7%, while removing student-guided weighting lowers AIME24 Pass@32 from 77.9% to 74.1%.The ablation supports complementary roles for student-guided exploration and teacher-guided noise mitigation.
Extended Experiments: On code-generation benchmarks, SCOPE improves Avg@32 over OPD by an average +4.69% and raises average Pass@32 from 59.2% to 60.3%.It also exceeds GRPO by +4.21% on average and remains effective beyond mathematical reasoning.

5 Related Work

The paper situates SCOPE between verified-reward reinforcement learning and knowledge distillation. It targets sparse terminal supervision and distribution problems in existing reinforcement-learning and distillation paradigms.

Reinforcement Learning with Verified Rewards: RLVR uses deterministic outcome verifiers to provide unambiguous signals, but standard GRPO still relies on sparse scalar rewards at the terminal step.This leaves long reasoning trajectories with limited token-level supervision.
Knowledge Distillation: Off-policy KD transfers teacher capabilities from static trajectories but suffers from exposure bias and distribution mismatch, motivating on-policy alternatives.OPD addresses the static-data issue through teacher supervision on student-sampled trajectories.

6 Conclusion

SCOPE introduces signal-quality-aware on-policy distillation through correctness-based routing and complementary perplexity-weighted supervision. Across mathematical and code-generation tasks, it reports improvements over competitive baselines and broader applicability.

Framework: SCOPE routes incorrect trajectories to teacher-perplexity-weighted KL distillation and correct trajectories to student-perplexity-weighted MLE.Group-level normalization calibrates weights across prompts with varying difficulty.
Results: Across six mathematical benchmarks, SCOPE achieves average relative improvements of 11.42% in Avg@32 and 7.30% in Pass@32 over competitive baselines.Additional code-generation results support effectiveness across further tasks.

Limitation

The study is limited by its reliance on automatically verifiable outcomes and by computational constraints on model, architecture, and domain coverage.

Scope: SCOPE requires automatically verifiable outcome signals, restricting current experiments to mathematical reasoning and code generation.Subjective or preference-driven tasks may require additional reward models or more sophisticated verification.
Evaluation Boundary: Computational resources constrain evaluation on larger foundation models, MoE architectures, and more diverse verifiable domains.The authors identify these settings as directions for further experiments.

Ethical Considerations

The study addresses ethical responsibilities by using public data and licensed resources, while emphasizing transparency and responsible use because model outputs may remain unreliable in high-stakes settings.

Experiments use publicly available datasets and benchmarks without private, sensitive, or personally identifiable information.
The authors use datasets, benchmarks, and pretrained models under their original licenses and terms of use.
Released resources will follow corresponding licensing requirements and support transparency through reported configurations and evaluation details.
Model outputs may remain unreliable in certain cases and should not be the sole basis for high-stakes decisions.

A Theoretical Motivation and Derivation

SCOPE motivates adaptive supervision by showing that uniform distillation can amplify unreliable teacher signals on flawed prefixes and over-reinforce frequent correct paths. Its dual weighting and group-level normalization target reliable corrections, underexplored valid reasoning, and stable update scales.

Incorrect trajectories: Uniform distillation on flawed prefixes can amplify noisy teacher signals because the teacher evaluates behavior-policy trajectories rather than the full teacher distribution.The detached token-level log-ratio acts as an advantage-like signal in the surrogate formulation.
Incorrect trajectories: Flawed prefixes can reduce teacher confidence, making the distillation signal reflect low-confidence evaluations instead of meaningful corrective directions.Low teacher likelihood increases the negative log-probability term used in the detached signal.
Incorrect trajectories: High teacher perplexity can enlarge the stochastic update scale and introduce noisy gradients on incorrect trajectories.The second-moment bound is controlled by the magnitude of the detached distillation signal.
Incorrect trajectories: SCOPE emphasizes incorrect trajectories with high teacher likelihood and suppresses high-perplexity trajectories to reduce likely noisy teacher influence.Trajectory-level teacher perplexity summarizes the regime in which the teacher assigns low average likelihood to the sampled path.
Correct trajectories: Uniform reinforcement of successful trajectories can amplify dominant reasoning modes because frequent paths are sampled more often than rare but valid paths.SCOPE assigns larger weights to correct trajectories with higher student perplexity, partially offsetting this sampling-frequency bias.
Correct trajectories: Length normalization is retained to avoid systematically favoring shorter or longer trajectories.
Group-level normalization: Group-level softmax normalization makes each supervision branch a convex combination of trajectory gradients rather than an unbounded rescaling by raw perplexity.It controls update scale while retaining adaptive weighting, introducing signal-calibrated bias rather than an unbiased estimator.

B.2 Evaluation Benchmarks

The evaluation covers mathematical reasoning and code generation across benchmarks spanning varied sources, difficulty levels, and reasoning patterns. Code experiments use TACO for training and LiveCodeBench and HumanEval for evaluation.

Mathematical reasoning: Mathematical reasoning is evaluated on benchmarks covering different problem sources, difficulty levels, and reasoning patterns.The suite ranges from high-school competition problems to olympiad-style tasks.
Code generation: Code generation training uses the TACO dataset, which contains 25,202 programming problems from diverse competitive programming platforms.The sources include Codeforces, AtCoder, Aizu Online Judge, and GeeksforGeeks.
Code generation: Code generation evaluation uses LiveCodeBench from 2024.08 to 2025.02 and HumanEval benchmarks.

C.1 Preliminary Experiment

The preliminary experiments examine teacher perplexity, error recovery, temperature sensitivity, computational overhead, and representative high-perplexity failures. They characterize how perplexity stratification relates to teacher recovery and how weighting temperature affects optimization.

Setup: 2,000 DeepMath problems are sampled, with four student-generated reasoning trajectories per problem.Trajectories use DeepSeek-R1-Distill-Qwen-1.5B with temperature τ = 0.6, top-k = 20, top-p = 0.95, and maximum response length 32,768 tokens.
Perplexity stratification: Incorrect trajectories are scored by teacher perplexity over response tokens and divided into four equal-sized quartile buckets.Table 6 reports teacher perplexity and negative log-likelihood statistics for the buckets.
Error recovery: Teacher error recovery is measured after truncating incorrect trajectories at ratios r ∈ {0.2, 0.4, 0.6, 0.8}, using four completions per prefix.Table 7 reports mean accuracy and the Q1–Q4 spread for each truncation level.
Temperature sensitivity: τ = 1.0 consistently gives the best performance across AIME24, AIME25, and AMC23 compared with τ = 0.5 and τ = 2.0.Lower temperature over-sharpens weights, while higher temperature weakens adaptive weighting toward uniform OPD.
Computational overhead: SCOPE incurs additional time overhead primarily from teacher queries because the evaluated architecture synchronously separates rollout generation and teacher log-probability acquisition.The weight-calculation overhead itself is minimal, and asynchronous execution is expected to improve efficiency toward GRPO.
Qualitative error analysis: High-perplexity errors include structural collapse and logical hallucinations, such as numerical overflow, infinite loops, flawed premises, and self-contradictions.The examples are drawn from errors with PPL ≥1.80.
Qualitative error analysis: One representative trajectory computes P ≈16.888 correctly before an erroneous re-approximation produces the final answer 17 instead of 16.