PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Reasoning Efficiently via Relaxed On-Policy Distillation

Jongwoo Ko, Sara Abdali, Young Jin Kim, Tianyi Chen, Pashmina Cameron

arXiv:2603.11137v1cs.LGcs.CL

TL;DR

Reasoning transfer to capacity-constrained models remains difficult, while on-policy distillation lacks reasoning-centric optimization advances. REOPOLD relaxes strict imitation through reward clipping, token-level dynamic sampling, and exploration-to-refinement training; it reports state-of-the-art results across reasoning tasks and up to 3.32× inference efficiency in Pass@K on Geometry3K and MathVerse.

  • Problem

    Reasoning transfer to small models is difficult, and on-policy distillation has yet to benefit from reasoning-centric optimization advances.

    Source: §1 · pp. 1–2

  • Method

    REOPOLD treats on-policy distillation as policy optimization and relaxes strict imitation through reward clipping, token-level dynamic sampling, and exploration-to-refinement training.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    Across mathematical, visual, and tool-use reasoning tasks, REOPOLD reports state-of-the-art performance, with up to 3.32× inference efficiency in terms of Pass@K on Geometry3K and MathVerse.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    The findings support REOPOLD as an efficient approach to reasoning transfer for small models across mathematical, visual, and tool-use tasks.

    Source: §1 · p. 2, §6 · p. 10

    On-policy distillation faces an exploration-alignment trade-off: higher sampling temperatures introduce tokens farther from the teacher distribution and exacerbate reward variance.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

On-policy distillation is pivotal for transferring reasoning capabilities to capacity-constrained models, yet remains prone to instability and negative transfer. We show that on-policy distillation can be interpreted, both theoretically and empirically, as a form of policy optimization, where the teacher-student log-likelihood ratio acts as a token reward. From this insight, we introduce REOPOLD (Relaxed On-Policy Distillation) a framework that stabilizes optimization by relaxing the strict imitation constraints of standard on-policy distillation. Specifically, REOPOLD temperately and selectively leverages rewards from the teacher through mixture-based reward clipping, entropy-based token-level dynamic sampling, and a unified exploration-to-refinement training strategy. Empirically, REOPOLD surpasses its baselines with superior sample efficiency during training and enhanced test-time scaling at inference, across mathematical, visual, and agentic tool-use reasoning tasks. Specifically, REOPOLD outperforms recent RL approaches achieving 6.7~12x greater sample efficiency and enables a 7B student to match a 32B teacher in visual reasoning with a ~3.32x inference speedup.

1. Introduction

REOPOLD targets instability and negative transfer in on-policy distillation by treating teacher–student likelihood ratios as rewards and regulating how teacher signals shape student updates. The paper presents this analysis and method as a route to more efficient, scalable reasoning transfer for small models.

Introduction

  • On-policy distillation transfers a capable teacher’s guidance through student-generated trajectories and has outperformed RL and supervised fine-tuning on reasoning tasks, but lacks the reasoning-centric optimization advances seen in RL.The cited background includes Yang et al. (2025), Pati˜no et al. (2025), Lu et al. (2025), and Xiao et al. (2026); the introduction contrasts this with RL advances such as GRPO (Shao et al., 2024).Source: §1 · pp. 1–2
  • Vanilla on-policy distillation can cause negative transfer and rapid entropy collapse, while reliance on similarly sized teachers limits its practical utility.The introduction notes that prior work restricts teachers to matching student sizes for stability, such as an 8B teacher for an 8B student rather than a 32B teacher.Source: §1 · p. 2
  • The analysis casts on-policy distillation as policy optimization with the teacher–student log-likelihood ratio as a fixed reward, and identifies heavy-tailed negative rewards and signal inefficiency as instability drivers.Stop-gradient makes the objective equivalent to standard policy gradient and acts as a control variate to mitigate variance.Source: §1 · p. 2
  • REOPOLD combines reward clipping, token-level dynamic sampling, and multi-stage training to temper and selectively apply teacher signals, filtering harmful signals and stabilizing optimization.The framework softens aggressive updates while regulating the learning signal.Source: §1 · p. 2
  • REOPOLD achieves superior training sample efficiency and test-time scaling for small models, with state-of-the-art results across mathematical, visual, and tool-use reasoning tasks.The introduction reports that it significantly outperforms standard baselines.Source: §1 · p. 2

2. Background and Related Work

The background contrasts reward-based policy optimization in reasoning-focused RL with on-policy distillation, which trains on student-generated trajectories. It presents on-policy distillation as a way to reduce train-test discrepancy and reports strong reasoning performance with lower computational cost.

2.1. Reinforcement Learning for Reasoning Models

  • Reasoning RL samples groups of responses from a previous policy and updates policy parameters using token-level importance ratios and estimated advantages.The described formulation omits some clipping operations for brevity (Schulman et al., 2017).Source: §2.1 · p. 2
  • Reasoning RL commonly uses final-answer correctness rewards, while refinements target bias, update constraints, clipping, normalization, KL regularization, and adaptive sampling.Examples include GRPO (Shao et al., 2024), Dr. GRPO (Liu et al., 2025c), and DAPO (Yu et al., 2025).Source: §2.1 · pp. 2–3
  • Stop-gradient in vanilla RKL yields a markedly lower, more stable gradient norm and superior validation performance, despite similar training-loss trends.The figure caption connects this result to the theoretical equivalence in Remark 3.1 and favors treating the log-likelihood ratio as a fixed reward signal.Source: Figure 3 · p. 3

2.2. On-Policy Distillation for Reasoning Models

  • On-policy distillation trains on student-sampled trajectories to mitigate exposure bias and minimizes reverse KL, equivalent to maximizing expected teacher-to-student log-likelihood ratio.Off-policy distillation such as SFT instead learns from static teacher-generated outputs; prior work reports that on-policy distillation improves generation quality (Agarwal et al., 2024).Source: §2.2 · p. 3
  • On-policy distillation can surpass RL in reasoning performance while using 10× fewer GPU hours, as reported for Qwen3 and subsequent studies.The cited studies include Yang et al. (2025), Lu et al. (2025), and Pati˜no et al. (2025).Source: §2.2 · p. 3

3. Analysis of On-Policy Distillation

On-policy distillation can be understood as policy-gradient optimization with the teacher-student token log-likelihood ratio as reward. Stop-gradient stabilization does not remove finite-sample challenges: heavy-tailed and near-zero rewards, plus an exploration-alignment trade-off, can undermine stability and efficiency.

3.1. Theoretical Equivalence to RL and Strong Baseline

  • Stop-gradient treats the θ-dependent reward as fixed, preserving expected gradients while suppressing finite-sample variance and stabilizing training with better validation performance.The omitted reward-gradient term has zero expectation but nonzero variance; removing it reduces gradient norms.Source: §3.1 · pp. 3–4
  • On-policy distillation is formally equivalent to an on-policy policy-gradient method using the teacher-student token log-likelihood ratio as reward.Unlike full-vocabulary discrepancy calculations, it operates on sampled tokens, avoiding the memory cost of storing full distributions.Source: §3.1 · p. 3

3.2. Optimization Challenges in On-policy Distillation

  • When students sample tokens the teacher assigns negligible probability, log-likelihood ratio rewards approach −∞, producing heavy-tailed gradients and destructive updates that shift the student from its original distribution.Figure 4 illustrates the heavy negative-reward tail associated with this instability.Source: §3.2 · p. 4, Figure 4 · p. 4
  • Near-zero rewards on most aligned tokens provide negligible learning signals while consuming computation, diluting effective sample size and reducing efficiency; prompt-level filtering is ineffective for token-level distillation.Figure 4 depicts the near-zero reward concentration.Source: §3.2 · p. 4, Figure 4 · p. 4
  • Rapid entropy decline narrows outputs and risks premature convergence, while higher sampling temperatures increase divergence from the teacher and reward variance, creating an exploration-alignment trade-off.The entropy decline is observed during training in Figure 6.Source: §3.2 · p. 4

4. REOPOLD: Relaxed On-Policy Distillation for Compact Reasoning Models

REOPOLD uses a unified, dynamic on-policy distillation objective that adapts learning signals across training stages through mixture-based reward clipping, entropy-guided token selection, and exploration-to-refinement training. Together, these mechanisms target unstable penalties, uninformative tokens, and entropy collapse.

4. REOPOLD: Relaxed On-Policy Distillation for Compact Reasoning Models

  • REOPOLD formulates a unified, dynamic objective that adapts the learning signal across training stages through reward clipping, token-level sampling, and multi-stage training.The training procedure combines reward clipping, entropy-based dynamic sampling, and an exploration-to-refinement strategy.Source: §4 · p. 4

4.1. Reward Clipping via Mixture-Based Regularization

  • Mixture-based clipping truncates heavy-tailed negative rewards at a theoretically derived floor, selectively targeting outliers while preserving RKL’s mode-seeking nature and reducing sensitivity to hyperparameters.Unlike Skew RKL, the method clips outliers rather than globally altering the objective.Source: §4.1 · pp. 4–5

4.2. Entropy-Guided Token-Level Dynamic Sampling

  • Entropy-guided token sampling focuses updates on high-uncertainty tokens, filtering near-zero-reward noise and improving convergence speed and best accuracy.The bottom 60th entropy percentile has rewards concentrated around zero, while high-entropy tokens often carry more meaningful learning signals.Source: §4.2 · pp. 5–6

4.3. Exploration-to-Refinement Multi-Stage Training

  • Two-phase training first suppresses strongly negative rewards to preserve diverse modes, then applies entropy-based masking to refine the policy on uncertain, divergent tokens.During exploration, REOPOLD disables token-level dynamic sampling for dense supervision; refinement reintroduces negative feedback on high-entropy tokens.Source: §4.3 · p. 6

5. Experimental Results

Across mathematical, visual, and agentic visual tool-use tasks, REOPOLD improves on-policy distillation’s sample efficiency, benchmark performance, and stability. Its visual reasoning results also show stronger test-time scaling, while analyses examine teacher choice, longer training, and component contributions.

5.1. Extension: Math Reasoning

  • REOPOLD matches ProRL in 600 versus 2,000 steps (>6.7× efficiency), beats DeepScaleR-1.5B-Preview and DeepMath-1.5B by 300 steps (>12×), and outperforms vanilla RKL in 150 versus 300 steps.Efficiency is normalized by total training samples to account for batch sizes.Source: §5.1 · p. 6
  • With the 7B student, vanilla RKL drops sharply and fails to improve AIME-24 beyond the base model, while REOPOLD prevents degradation and improves continuously across benchmarks.The source attributes RKL’s instability to the student’s already solidified reasoning capabilities.Source: §5.1 · p. 7, Figure 7 · p. 7

5.2. Main Results: Visual Reasoning

  • REOPOLD beats GRPO and RKL across visual reasoning and perception benchmarks for 3B and 7B models, surpasses NoisyRollout and PAPO, and reaches up to 3.32× Pass@K inference efficiency.The inference efficiency is reported for visual test-time scaling; Maj@K results also maintain a better scaling trajectory than RKL.Source: §5.2 · p. 8, Table 2 · p. 8

5.3. Analysis: Visual Reasoning

  • REOPOLD consistently beats RKL across all six visual benchmarks with either teacher, gaining an average 1.34% accuracy with the 7B teacher and 0.93% with the 32B teacher.The tested teachers are Qwen2.5-VL-7B-Instruct and Qwen2.5-VL-32B-Instruct, finetuned on Geometry3K and MMK12, respectively.Source: §5.3 · p. 8, Table 3 · p. 9
  • At 300 training steps on Geometry3K and MMK12, REOPOLD outperforms RKL and reaches average accuracy of 54.36% with 3B and 58.83% with 7B models.Extending training improves both the baseline and REOPOLD, while REOPOLD shows better scalability.Source: §5.3 · p. 8, Table 4 · p. 9, Table 5 · p. 9
  • In component tests, stop-gradient, reward clipping, token-level dynamic sampling, and multi-stage training add improvements, with multi-stage training yielding the best overall performance.A stricter token-selection threshold (β = 0.2) outperforms β = 0.5; filtering low-entropy tokens mitigates gradient dilution.Source: §5.3 · pp. 8–9, Table 6 · p. 9, Figure 8 · p. 9

5.4. Extension: Agentic Reasoning with Visual Tool-Use

  • On agentic visual tool-use tasks, REOPOLD beats RKL and GRPO overall, though GRPO is slightly more accurate on the Pixel test split; REOPOLD also shows better sample efficiency.It surpasses GRPO even with the complex reward designs proposed by Su et al. (2025), and can be applied directly without intricate reward engineering.Source: §5.4 · p. 10, Table 7 · p. 10

6. Conclusion

The conclusion presents REOPOLD as a response to optimization instability in on-policy distillation, replacing rigid imitation with flexible, stabilized training. It reports superior reasoning performance across mathematical, visual, and agentic tasks, and argues that relaxing strict imitation is essential for scaling compact models.

6. Conclusion

  • REOPOLD yields superior performance across mathematical, visual, and agentic reasoning compared to recent RL algorithms.The conclusion describes the framework as resolving vanilla distillation’s fundamental instability.Source: §6 · p. 10
  • On-policy distillation is theoretically equivalent to policy optimization and inherits fundamental optimization instabilities that standard RL solutions cannot effectively resolve.Source: §6 · p. 10
  • REOPOLD replaces rigid imitation with stabilized training through mixture-based reward clipping, token-level dynamic sampling, and a unified exploration-to-refinement strategy.Source: §6 · p. 10
  • At 50% training, REOPOLD outperforms all baselines.Figure 9 reports average score by training step.Source: Figure 9 · p. 10

A. Mathematical Derivations · A.1. RKL

The RKL derivation relies on regularity conditions that make its gradients and likelihood ratios well-defined. It shows that optimizing RKL is equivalent to policy-gradient optimization with a teacher-derived advantage weighted by an importance-sampling ratio.

A.1. RKL

  • The derivation begins by specifying regularity conditions needed to ensure its validity.Source: §A.1 · p. 15
  • The derivation assumes π_θ is continuously differentiable and its gradient is bounded by an integrable function, permitting interchange of the gradient and expectation.Source: §A.1 · p. 15
  • It assumes absolute continuity, π_θ ≪ π_θold, so any observation assigned zero probability by π_θold also has zero probability under π_θ.Source: §A.1 · p. 15
  • This support-coverage condition ensures the importance-sampling ratio ρ_t(θ) and log-likelihood ratio R_t(θ) are well-defined and finite almost everywhere.Source: §A.1 · p. 15
  • The derivation adopts techniques similar to PPO or GRPO to obtain the RKL objective and its gradient.Source: §A.1 · p. 15
  • RKL optimization is mathematically equivalent to maximizing a standard policy-gradient objective, with R_t(θ) as the advantage and ρ_t(θ) as its importance-sampling weight.Source: §A.1 · p. 15

A.2. Derivation of Clipping Threshold · B. Additional Related Work

The appendix derives a convex-mixture upper bound for the teacher–student log-ratio and compares clipping with alternative stabilizers. It also reviews reasoning models, policy optimization, and on-policy distillation for reasoning.

A.2. Derivation of Clipping Threshold

  • Jensen’s inequality yields a convex-mixture upper bound on the teacher–student log-likelihood ratio, motivating the clipping threshold.The derivation applies for λ ∈ [0, 1).Source: §A.2 · p. 16
  • ρ(θ) clipping affects fewer than 0.2% of samples after initial steps, making its stabilizing effect minimal in on-policy distillation.The passage contrasts this with the heavy negative tail of the reward distribution, where rare extreme values can dominate gradients.Source: Figure 10 · p. 16, §A.2 · p. 16
  • REOPOLD remains robust across λ, and its lowest accuracy exceeds Skew RKL’s peak, while Skew RKL drops significantly at λ = 0.7.The method applies log λ/(1−λ) as a clipping threshold rather than modifying the global objective, selectively clipping heavy-tailed outliers.Source: Figure 11 · p. 16, §A.2 · p. 16

B. Additional Related Work

C. Detailed Experimental Setup

The section reports REOPOLD’s extended inference trade-off and specifies separate training and evaluation configurations for mathematical, visual, and agentic visual tool-use reasoning.

Math reasoning

  • Math reasoning uses Verl-based on-policy distillation for 300 iterations, with 8 responses per prompt, 128/32 global/mini-batches, and 4 updates per rollout step.Rollouts use a maximum response length of 8192 and temperature 1.0; training uses AdamW at 1 × 10^-5, taking about 200 and 312 GPU hours for 1.5B and 7B models. Evaluation uses length 32768, temperature 0.6, top-p 0.95, and LUFFY data (Yan et al., 2025).Source: C. Detailed Experimental Setup · p. 18

Visual reasoning

  • Visual reasoning uses Verl-based on-policy distillation for 60 iterations, with 12 responses per prompt, batch size 128, mini-batch size 64, and 2 updates per step.Training uses response length 2048, temperature 1.0, and AdamW at 5 × 10^-6, taking about 20 and 24 GPU hours for 3B and 7B models; evaluation uses length 8192, temperature 0.6, top-p 0.95, and NoisyRollout data (Liu et al., 2025b).Source: C. Detailed Experimental Setup · p. 18

Agentic reasoning with visual tool-use

  • Agentic visual tool-use training uses VerlTool for 40 iterations, with 8 trajectories per prompt, batch size 128, mini-batch size 64, and 2 updates per step.Rollouts use maximum round 2, response length 8192, and temperature 1.0; AdamW at 1 × 10^-6 training takes about 120 GPU hours on one 8× H100 node. Evaluation follows Su et al. (2025) with maximum round 5, length 8192, temperature and top-p 1.0, using PixelReasoner-RL training data and PixelReasoner-provided InfoVQA, TallyQA, and VStar datasets.Source: C. Detailed Experimental Setup · p. 18

Extended inference results

  • REOPOLD consistently offers a better accuracy–latency trade-off than RKL, while the 7B student matches or beats the 32B teacher at lower latency.The comparison covers Pass@K and Maj@K as sample budget K increases, up to 64 for Geometry3K and 16 for MathVerse.Source: Figure 12 · p. 19

D. Additional Analyses and Discussions

The analyses show REOPOLD’s test-time gains extend to consensus robustness, teacher overhead is moderate relative to student generation, and full-vocabulary or Top-5 variants offer no meaningful improvement.

Extended test-time scaling results

  • On Geometry3K and MathVerse, both 3B and 7B REOPOLD models maintain a consistent lead in Maj@K as well as Pass@K, indicating stronger consensus robustness.The authors interpret this as increasing the probability of correct reasoning chains rather than relying on diverse lucky guesses.Source: D. Additional Analyses and Discussions · p. 19

Training time analysis

  • Teacher cost is 8%–22% per step; at 8192 tokens student generation takes 77.3% and teacher 8.2%, while the teacher share rises to 21.8% at 2048 tokens.The teacher’s relative cost is inversely proportional to generation length, and student generation remains the primary bottleneck in both task settings.Source: Figure 13 · p. 19, D. Additional Analyses and Discussions · p. 19

Comparison to full vocabulary distillation

  • GKD’s full-vocabulary distillation OOMs on 150K-vocabulary storage; Top-5 was less efficient than sampled-token methods, and neither variant meaningfully improved REOPOLD.The OOM occurs because GKD must store student and teacher values across the 150K vocabulary.Source: Table 9 · p. 19, D. Additional Analyses and Discussions · p. 19

E. Qualitative Evaluation

Qualitative examples show REOPOLD preserving visual grounding on Hallusion Bench and self-correcting its reasoning on MathVerse, where Vanilla RKL hallucinates or gets trapped in circular logic.

E. Qualitative Evaluation

  • On Hallusion Bench, REOPOLD accurately identifies the chart peak, while the RKL-trained baseline hallucinates a flat trend despite visual evidence.Figure 14 contrasts the baseline’s flat-trend hallucination with REOPOLD’s correct identification of the peak.Source: Figure 14 · p. 20
  • On MathVerse, REOPOLD self-corrects an initial 96◦ answer to 84◦, whereas Vanilla RKL loops at Step 5 and hallucinates the final calculation at Step 8.REOPOLD explicitly triggers re-evaluation after deriving the incorrect value.Source: Figure 15 · p. 21

Related papers

Loading 2603.11137v1…