PaperlayerView exact revision on arXiv

Source-linked AI summary

On-policy Distillation with Verifiable Reward

Wenze Lin, Jiale Zhao, Xitai Jiang, Songde Rao, Yining Li, Shenzhi Wang, Bingxiang He, Gao Huang

arXiv:2608.24696v1cs.LGcs.AI

TL;DR

RLVR is sparse while OPD ignores trajectory correctness, and existing combinations introduce hyperparameters or heuristic trade-offs. OPDVR uses a ReLU-gated reformulation of sampled-token OPD to align distillation rewards with verified correctness, consistently outperforming standard OPD across six reasoning benchmarks.

  • Problem

    RLVR provides sparse task-level feedback, whereas OPD provides dense guidance without aligning its objective to trajectory correctness; existing integrations add hyperparameters and heuristic trade-offs.

  • Method

    OPDVR reformulates sampled-token OPD as an RLVR method and applies a ReLU gate that aligns reward signs with trajectory correctness while preserving teacher distributional guidance.

  • Results

    OPDVR consistently outperforms standard OPD across six reasoning benchmarks, with gains of 2.7 points on AIME24 and 2.1 points on AIME25 in the same-architecture setting.

  • Takeaways & Limitations

    The reformulation makes OPDVR readily integrable with policy-gradient algorithms, including REINFORCE, GRPO, and DAPO.

  • Takeaways & Limitations

    Sampled-token OPD’s reward sign is determined by the teacher-student probability ratio rather than trajectory correctness.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at https://github.com/LeapLabTHU/OPDVR.

1. Introduction

RLVR offers task-level correctness but sparse feedback, while OPD supplies dense token-level guidance without trajectory correctness. OPDVR combines their strengths through a hyperparameter-free ReLU-gated sampled-token objective.

  • RLVR provides explicit outcome rewards but offers little supervision for intermediate reasoning steps, making credit assignment challenging.
  • OPD provides dense token-level guidance, complementing RLVR’s sparse task-level correctness supervision.
  • Existing combinations explicitly or selectively apply OPD and RLVR, often introducing extra hyperparameters and heuristic trade-offs.
  • OPDVR applies a simple ReLU gate to sampled-token OPD, combining OPD and RLVR without introducing hyperparameters.
  • The method reformulates sampled-token OPD rewards using trajectory correctness and teacher-student distribution discrepancies, then aligns reward signs with correctness.

2. Related Work

RLVR supplies verifiable task-level rewards, whereas OPD supplies dense token-level teacher guidance but remains distributional. Prior combinations integrate them through weighting, switching, or related mechanisms.

  • RLVR uses rule-based verifiers to provide objective signals such as answer correctness or code compilation.
  • OPD distills a teacher’s output distribution into the student and offers fine-grained guidance at every generation step.
  • OPD’s purely distributional objective drives the student to mimic the teacher’s output distribution.
  • Prior methods combine OPD and RLVR by weighting objectives, switching by advantage sign, aligning outcome and process signals, or weighting GRPO advantages with probability ratios.

3. Preliminaries

RLVR optimizes policies with verifier-based trajectory rewards, while GRPO stabilizes policy optimization through group-relative advantages. OPD instead minimizes reverse KL divergence from a teacher, with sampled-token OPD offering a cheaper approximation.

  • 3.1. Reinforcement Learning with Verifiable Rewards (RLVR): RLVR uses verifier rewards such as +1 for correct and −1 for incorrect trajectories in REINFORCE, while other methods use +1 and 0.
  • 3.1. Reinforcement Learning with Verifiable Rewards (RLVR): RLVR maximizes expected trajectory reward and estimates its gradient using the REINFORCE log-derivative trick with sampled responses.
  • 3.1. Reinforcement Learning with Verifiable Rewards (RLVR): GRPO normalizes response rewards against a sampled group’s mean and standard deviation to reduce variance.
  • 3.1. Reinforcement Learning with Verifiable Rewards (RLVR): GRPO typically uses verifier rewards in {0, 1}, because group-relative normalization centers advantages around zero.
  • 3.2. On-policy Distillation: OPD minimizes reverse KL divergence from a teacher; full-vocabulary computation is expensive, motivating Top-k and sampled-token approximations.

4. Method

Sampled-token OPD can be viewed as RL with a teacher-student log-ratio reward, but its reward sign may conflict with trajectory correctness. OPDVR fixes this using ReLU gating, and the same framework extends to group-relative advantages.

  • 4.1. Revisiting Sampled-Token OPD from an RLVR Perspective: Sampled-token OPD approximates the reverse KL gradient with one sampled token and treats the teacher-student log-ratio as an implicit reward.
  • 4.1. Revisiting Sampled-Token OPD from an RLVR Perspective: The sampled-token OPD gradient has the same form as the RLVR gradient, with the log-ratio serving as the reward coefficient.
  • 4.1. Revisiting Sampled-Token OPD from an RLVR Perspective: Because its sign depends only on the teacher-student probability ratio, sampled-token OPD can assign signs inconsistent with trajectory correctness.
  • 4.2. OPDVR: A Simple Gated Mechanism: OPDVR applies a ReLU gate to sampled-token rewards, enforcing non-negative rewards for correct trajectories and non-positive rewards for incorrect trajectories.
  • 4.3. Interpreting the ReLU Gating Mechanism as a Conditional Mask: The gate masks updates conflicting with verifier signals while retaining teacher-controlled reward magnitude through the probability ratio.
  • 4.4. Group Relative Policy Distillation (GRPD): GRPD replaces binary correctness with group-relative advantages that are positive above the group average and negative below it.

5. Experiments

Experiments evaluate OPDVR across same- and cross-architecture distillation settings, six reasoning benchmarks, group-relative advantages, gating ablations, and training dynamics. OPDVR consistently improves over OPD and related baselines, while the gating mechanism remains stable across settings.

  • Main Experimental Settings: OPDVR is evaluated in same-architecture and cross-architecture distillation settings on six reasoning benchmarks.The benchmarks are AIME24, AIME25, AMC, MATH500, Minerva, and OlympiadBench.
  • Main Results: OPDVR consistently outperforms sampled-token OPD and top-64 OPD across all six benchmarks in both distillation settings.In the same-architecture setting, it gains 2.7 points on AIME24 and 2.1 points on AIME25 over sampled-token OPD; in the cross-architecture setting, gains include 5.5 points on AMC and 1.7 points on MATH500.
  • Group Relative Policy Distillation: GRPD outperforms both GRPO and OPD across all six benchmarks, with gains of 6.5 points on AIME24 and 10.9 points on AIME25 over GRPO.GRPD also surpasses OPD on five of six benchmarks, including a 2.8-point improvement on AIME24.
  • Ablation Study: Inverse-Gated Experiment: Reversing the gate produces the ordering OPDVR > OPD > Inverse-Gated in both training curves and final benchmarks.The inverse-gated variant falls below vanilla OPD on all six benchmarks, although it still improves over the initial model.
  • Training Dynamics: Entropy and response-length dynamics vary by teacher–student pair, whereas the zero-gated token ratio remains stable across settings.The ratio stays around 0.48–0.50 for the 4B student and 0.40–0.44 for the 1.7B student, meaning roughly half of sampled tokens are masked.

6. Conclusion

The paper reframes sampled-token OPD through an RLVR lens and introduces OPDVR, which gates distillation signals by trajectory correctness. This combines dense teacher guidance with verifier-aligned updates and consistently outperforms standard OPD across mathematical reasoning benchmarks.

  • Sampled-token OPD can reward or penalize tokens according to teacher-student probability ratios rather than trajectory correctness.
  • OPDVR adds a simple ReLU gate that enforces RLVR-compliant reward signs without hyperparameters or heuristic trade-offs.
  • OPDVR preserves teacher guidance while aligning distillation updates with task-level correctness.
  • OPDVR consistently outperforms standard OPD across mathematical reasoning benchmarks.
  • OPDVR becomes compatible with policy-gradient algorithms including REINFORCE, GRPO, and DAPO.

A. Theoretical Analysis: Why OPDVR Improves over Sampled-Token OPD

This section formalizes why OPDVR improves sampled-token OPD by analyzing OPD’s objective, token-level state and action definitions, and trajectory-level verifier rewards.

  • The analysis compares OPDVR against the sampled-token OPD objective.
  • At generation step t, the state is the question together with the preceding output, and the action is the sampled token.
  • The trajectory-level verifier reward is binary, with R ∈ {+1, −1}.

A.1. Directional Alignment with the Verifier Gradient

OPDVR’s gated update is always aligned with the verifier gradient, whereas sampled-token OPD can oppose it when the implicit token reward conflicts with trajectory correctness.

  • The sampled-token update direction is the policy-gradient direction of the sampled token, scaled by its implicit OPD reward.
  • OPDVR scales the sampled-token policy gradient by R ReLU(Rr_t), zeroing updates whose direction conflicts with the verifier signal.
  • OPDVR’s update has a nonnegative projection onto the RLVR gradient for every nonzero token gradient.
  • Sampled-token OPD’s projection equals r_tR∥u_t∥^2 and becomes negative whenever r_t and R have opposite signs.
  • The conflicting condition corresponds to correct trajectories where the student is more confident, or incorrect trajectories where the teacher is more confident.

A.2. Decomposition of the OPD Gradient

OPDVR can be understood as sampled-token OPD with its verifier-opposing gradient component removed.

  • The OPD gradient decomposes into an OPDVR component and a verifier-conflicting component.
  • The conflict term contains OPD updates whose implicit reward and verifier reward have opposite signs.
  • When the signs agree, OPDVR equals OPD and the conflict term vanishes; when they disagree, OPDVR removes the conflicting update.
  • Thus OPDVR is precisely OPD with the harmful verifier-opposing component removed.

A.3. A Simplified Token-Level Analysis: OPDVR Can Strictly Outperform the Teacher

Under a suboptimal teacher and an initially stronger student, standard sampled-token OPD converges toward the teacher, whereas OPDVR preserves the student's stronger policy and strictly outperforms both baselines.

  • Assumptions: With teacher probability p < 1/2 and initial student probability q0 > p, the analysis assumes the student initially exceeds the teacher.Here, a1 is the correct token and receives verifier reward +1, while a2 is incorrect and receives −1.
  • Sampled-token OPD: Standard sampled-token OPD minimizes reverse KL divergence and has its global optimum at the teacher policy, q = p.Thus, OPD moves the student toward the teacher even when the student initially performs better.
  • OPDVR: Under OPDVR, the expected gradient vanishes under these assumptions, so the policy remains at its initially stronger value q0.The result follows from the gated updates for the correct and incorrect tokens.
  • Conclusion: OPDVR strictly outperforms both standard sampled-token OPD and the teacher policy in this simplified analysis.The comparison follows because OPD converges to the teacher while OPDVR preserves the initially stronger student policy.

B. Hyperparameters

The paper reports its experimental hyperparameter settings in Table 5 and states that models use the Verl framework on NVIDIA GeForce RTX 5090 GPUs.

  • Hyperparameter settings: Detailed hyperparameter configurations for the experiments are provided in Table 5.The table is identified as containing hyperparameter settings.
  • Training framework: All models are trained using the Verl framework with the specified settings.The passage attributes the training framework to Sheng et al. (2025).
  • Hardware: All experiments are conducted on NVIDIA GeForce RTX 5090 GPUs.This specifies the hardware used across the experiments.
Loading 2608.24696v1…